Learn how to automate complex workflows with Argo Workflows

Introduction

Creating and managing complex workflows is a top priority for modern software development and operations teams. To achieve this, you need a robust and scalable tool that can automate tasks with precision and reliability. In this comprehensive course, we'll take you on a journey from the fundamentals to advanced techniques, culminating in a real-world example: automating an Employee Onboarding Process. By the end of this course, you'll be equipped with the skills and knowledge to create Argo workflows.

Argo Workflows

Argo Workflows is an open-source container-native workflow orchestrator for Kubernetes. It allows you to define, schedule, and execute workflows directly on Kubernetes clusters using YAML.

Argo Workflows offer you scalability and flexibility by leveraging Kubernetes scaling capabilities, and it's used in various fields, the image below, taken from it's UI shows some of it's use cases:

Core Concepts

• Workflow: A sequence of tasks to accomplish a goal. The workflows are Kubernetes resources, so you can use kubectl to manage them.
• Steps/DAG: Defines task dependencies as steps or a directed acyclic graph.
• Template: Reusable components to simplify workflow definitions. There are several types of templates (Container, Resource, DAG, Suspend...). To implement a workflow we should use one or more templates, these templates execute works by running Kubernetes pod. In this guide, we're going to take a deep-dive into the two most common template types: Container and DAG.

Getting Started with Argo Workflows

If you want to get your hands dirty while reading this guide, you should first install and set up Argo Workflows. Please visit this link for installation instructions. Otherwise, you can use the Killercoda or the demo environment provided in the official Argo Workflows documentation at https://argo-workflows.readthedocs.io

Writing Your First Workflow

Let's create our first workflow, open a new file and paste the following code. It's a simple workflow that will prints "Hello, Argo!"

1apiVersion: argoproj.io/v1alpha1  
2kind: Workflow  
3metadata:  
4  generateName: hello-world-  
5spec:  
6  entrypoint: main  
7  templates:  
8    - name: main  
9      container:  
10        image: alpine:latest  
11        command: [echo]  
12        args: ["Hello, Argo!"]
13

Argo Workflows is implemented using K8s Custom Resource Definition. So, the workflows are defined in YAML format. Every workflow must contain one or more templates. Here, we have only one template. In the workflow's spec, we define the templates, but also the entrypoint, which is the first template that must be executed if we have multiple templates defined in our workflow. The template has the type 'container', which requires an OCI image to be executed; in the example, we used 'alpine:latest'.

To execute the workflow we have to submit it to our kubernetes cluster. The following command use Argo CLI to submit our workflow.

1argo submit hello-world.yaml --watch

1$ argo get -n argo @latest
2Name:                hello-world-k2wm8
3Namespace:           argo
4ServiceAccount:      argo
5Status:              Succeeded
6Conditions:          
7 PodRunning          False
8 Completed           True
9Created:             Wed Dec 18 19:22:56 +0000 (1 minute ago)
10Started:             Wed Dec 18 19:22:56 +0000 (1 minute ago)
11Finished:            Wed Dec 18 19:23:06 +0000 (1 minute ago)
12Duration:            10 seconds
13Progress:            1/1
14ResourcesDuration:   0s*(1 cpu),5s*(100Mi memory)
15
16STEP                  TEMPLATE     PODNAME            DURATION  MESSAGE
17 ✔ hello-world-k2wm8  hello-world  hello-world-k2wm8  6s 
18

Everything you can accomplish with the CLI can also be achieved through the UI, which provides a visual representation of workflow progress. This enables you to easily debug and troubleshoot failed workflows. Additionally, you can submit workflows directly from the UI, as illustrated in the following image.

Submit an Argo workflow using the UI

Parameters and Variables

Now that we've covered the basics of workflows, let's dive deeper into customizing the work done by workflows using inputs and parameters. This allows you to dynamically pass parameters to workflows, making them more flexible and adaptable to your specific needs. Here's an example:

1apiVersion: argoproj.io/v1alpha1  
2kind: Workflow  
3metadata:  
4  generateName: dynamic-params-  
5spec:  
6  entrypoint: echo  
7  arguments:  
8    parameters:  
9      - name: input  
10        value: "default-value"  
11  templates:  
12    - name: echo  
13      container:  
14        image: alpine:latest  
15        command: [echo]  
16        args: ["{{inputs.parameters.input}}"]

In this example we have added a parameter with a default value, when you submit the workflow the input will be sent to the template. Using the CLI execute this command to submit the workflow using our custom input value

1argo submit dynamic-params.yaml -p input="Custom Value"

In addition to customizing your workflows with parameters, Argo Workflows offers a wide range of features that enhance its capabilities, including:

• Artifact Management: There are built-in support for git, HTTP, GCS, and S3 artifacts.
• Event-Driven Workflows: Trigger workflows based on external events like webhooks or Kubernetes events.

CI/CD Integration

Argo Workflows seamlessly integrates with popular CI/CD solutions, making it easy to incorporate workflow automation into your existing development pipeline. For instance, this GitHub Action enables the automated testing of workflows: https://github.com/marketplace/actions/submit-argo-workflows-from-github

Real-World Example: Automating an Employee Onboarding Process

After learning how to create workflows, let's apply what we have learned to implement a real-world example. We are going to automate the process of employee onboarding. Employee onboarding involves multiple tasks, such as creating accounts, assigning tools, and sending documentation. Argo Workflows can automate this process, reducing manual effort and errors.

To implement our workflows we are going to use these tasks:

• Send a welcome email.
• Create accounts for internal tools.
• Generate contracts.
• Schedule training sessions.

To simplify our workflow, we will utilize the Alpine image and write a message about the process to execute. Please replace the sample image and command with your own OCI image and process.

1apiVersion: argoproj.io/v1alpha1  
2kind: Workflow  
3metadata:  
4  generateName: employee-onboarding-  
5spec:  
6  entrypoint: onboarding
7  arguments:  
8    parameters:  
9      - name: name  
10        value: "default-name"
11      - name: email  
12        value: "default-email"
13  templates:  
14    - name: onboarding  
15      dag:  
16        tasks:  
17          - name: send-welcome-email  
18            template: email  
19
20          - name: provision-accounts  
21            template: accounts  
22            dependencies: [send-welcome-email]  
23
24          - name: generate-contracts  
25            template: contracts  
26            dependencies: [send-welcome-email]  
27
28          - name: schedule-training  
29            template: training  
30            dependencies: [provision-accounts, generate-contracts]  
31
32    - name: email  
33      container:  
34        image: alpine:latest  
35        command: [echo]  
36        args: ["Welcome email sent to {{workflow.parameters.email}}"]  
37
38    - name: accounts  
39      container:  
40        image: alpine:latest  
41        command: [echo]  
42        args: ["Accounts provisioned for {{workflow.parameters.email}}"]  
43
44    - name: contracts  
45      container:  
46        image: alpine:latest  
47        command: [echo]  
48        args: ["Contract generated for {{workflow.parameters.name}}"]  
49
50    - name: training  
51      container:  
52        image: alpine:latest  
53        command: [echo]  
54        args: ["Training scheduled for {{workflow.parameters.name}}"]
55

As seen before submit your workflows given the required parameters like this:

1argo submit onboarding.yaml -p email=guru@workflows.guru -p name="Workflows Guru"

Best Practices and Optimization

It's essential to use best practices that ensure efficient and effective workflow management. For instance, you have to design your workflows in a modular fashion, breaking them down into reusable templates can significantly improve maintainability and scalability. Additionally, implementing a retry policy is crucial to ensure that workflows can recover from failures and continue to work smoothly.

Conclusion

In this guide, we explored the fundamentals of Argo Workflows and demonstrated its practical application through a real-world example of automating employee onboarding. Argo's versatility makes it an ideal solution for a wide range of use cases, including CI/CD, data pipelines, and more. If you're interested in learning more, we encourage you to continue exploring Argo Events and Argo CD.