Automating Your Development Pipeline: A Step-by-Step Guide to Github Workflows

Introduction

GitHub Workflows allow developers to automate their software development lifecycle from building and testing to deploying applications.

In this course, we'll dive into the fundamentals of GitHub Workflows, explore real-world use cases, and walk you through step-by-step examples to automate tasks like testing, building, and deploying applications. By the end, you'll have the skills to create robust, automated workflows that save time, reduce errors, and enhance collaboration across your team.

What is a Github Workflow?

GitHub Workflows are powerful automation tools integrated directly into GitHub Actions, allowing you to create custom software development lifecycle processes directly in your repository. They help you automate tasks like:

• Continuous Integration (CI)
• Continuous Deployment (CD)
• Code testing
• Building and packaging
• Deploying to cloud services
• Running security scans

GitHub Workflows are defined in YAML files and are executed when certain events occur in your repository. They enable you to:

• Automate repetitive tasks like testing and deployment.
• Create CI/CD pipelines.
• Trigger actions based on events like pull requests, pushes, or scheduled times.

Basic Workflow Structure

A workflow is process defined in a YAML file located in the .github/workflows directory of your repository. Each workflow consists of:

• Events: Triggers for your workflow (e.g., push, pull_request, schedule).
• Jobs: Units of work to perform in the workflow, running in parallel or sequentially.
• Steps: Individual tasks within a job, such as running scripts or commands.
• Actions: Reusable code blocks performing specific functions, often contributed by the community.

The following YAML show an example of a basic workflow

1
2name: My First Workflow
3on: [push]  # Trigger on push to any branch
4
5jobs:
6  build:
7    runs-on: ubuntu-latest
8    steps:
9      - uses: actions/checkout@v3
10      - name: Run a one-line script
11        run: echo Hello, world!
12

Build a CI/CD workflow using Github

In this section, we will build a basic CI pipeline for a Python project and then extend it to create a pipeline to deploy a Node.js application to a Linux VM.

Basic CI Workflow

Let's create a pipeline that will ensures that every change to the main branch is tested and linted automatically.

1
2name: Python Application CI
3
4on:
5  push:
6    branches: [ main ]
7  pull_request:
8    branches: [ main ]
9
10jobs:
11  build:
12    runs-on: ubuntu-latest
13
14    steps:
15    - uses: actions/checkout@v3
16    
17    - name: Set up Python
18      uses: actions/setup-python@v3
19      with:
20        python-version: '3.9'
21    
22    - name: Install dependencies
23      run: |
24        python -m pip install --upgrade pip
25        pip install -r requirements.txt
26    
27    - name: Run tests
28      run: |
29        pytest tests/
30    
31    - name: Lint with flake8
32      run: |
33        pip install flake8 
34        flake8 .
35

This pipleline run on two type of events: when we push to the main branch and when we create a pull_request targeting the main branch.

The pipeline defines a single job called build, which runs on the latest version of Ubuntu (ubuntu-latest). The jobs contains five steps:

• Checkout Code: The actions/checkout@v3 action is used to clone the repository's code into the workflow's environment. This is a necessary first step to access the application code
• Set Up Python: The actions/setup-python@v3 action is used to install Python. In this case, Python version 3.9 is specified.
• Install Dependencies: Dependencies listed in the requirements.txt file are installed using pip install -r requirements.txt.
• Run Tests: The pytest framework is used to run tests located in the tests/ directory. This ensures the code is functioning as expected.
• Lint with flake8: The flake8 tool is installed and used to lint the codebase. This checks for style and syntax issues, ensuring the code adheres to best practices.

Deploy a Node.js App

In this section we will create a github workflow that will deploy a Node.js application a linux VM. The workflow will:

• Run tests
• Lint the code
• Deploy to our Linux VM
• Send a notification to an external API

Before using this pipeline, you should first create the required secrets on your github secrets. (You should create SSH_PRIVATE_KEY, VM_USER, VM_HOST, APP_DIR)

1
2name: Deploy Node.js app to Linux VM
3
4on:
5  push:
6    branches: [ main ]
7
8jobs:
9  build-and-deploy:
10    runs-on: ubuntu-latest
11
12    steps:
13    - uses: actions/checkout@v3
14    
15    - name: Use Node.js
16      uses: actions/setup-node@v3
17      with:
18        node-version: '16'
19    
20    - name: Install Dependencies
21      run: npm ci
22    
23    - name: Run Tests
24      run: npm test
25
26    - name: Lint code
27      run: npm run lint # Add a linting step if applicable
28
29    - name: Deploy Node.js App to Linux VM
30      env:
31        SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
32        VM_USER: ${{ secrets.VM_USER }}
33        VM_HOST: ${{ secrets.VM_HOST }}
34        APP_DIR: ${{ secrets.APP_DIR }} # Directory on the VM
35      run: |
36        set -e # Exit on error
37        mkdir -p ~/.ssh
38        chmod 700 ~/.ssh
39        echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
40        chmod 600 ~/.ssh/id_rsa
41
42        # Add VM's SSH fingerprint to known_hosts
43        ssh-keyscan -H $VM_HOST >> ~/.ssh/known_hosts
44
45        # Copy application files to the VM
46        scp -r . $VM_USER@$VM_HOST:$APP_DIR
47
48        # SSH into the VM and deploy the app
49        ssh $VM_USER@$VM_HOST << 'EOF'
50        set -e # Exit on error
51        cd $APP_DIR
52        npm ci --production # Install production dependencies
53        pm2 start app.js --name <My node app name>
54        echo "Deployment successful!"
55        EOF
56    
57    # This is just an example assuming that you have an HHTP API to call
58    - name: Notify Monitoring Service
59      env:
60        MONITORING_API_KEY: ${{ secrets.MONITORING_API_KEY }},
61      run: |
62        curl -X POST https://monitoring-service.com/api/deploy
63             -H "Authorization: Bearer $MONITORING_API_KEY"
64             -H "Content-Type: application/json"
65             -d '{
66               "repository": ${{ github.repository }},
67               "commit": ${{ github.sha }},
68               "status": "deployed"
69             }'
70

Common Workflow Triggers

Triggers are a fundamental component of GitHub Workflows, enabling automated workflows to be initiated in response to specific events or actions. These events can include push requests, pull requests, or other repository activities, allowing developers to create customized workflows that respond to changing conditions. Below is a list of triggers that you can use in your github workflows.

• push: Runs on code push
• pull_request: Runs on pull request creation/update
• schedule: Runs on a set schedule
• workflow_dispatch: Manual trigger
• repository_dispatch: Triggered by external events

Optimizing Gihtub Workflows

Optimizing GitHub Workflows is crucial for maximizing development productivity, as it enables teams to reduce manual errors. By fine-tuning their workflows, developers can automate repetitive tasks, improve code quality, and accelerate deployment times.

To implement effective CI/CD piplelines here is a list of best practice that you must apply to your Github Actions:

• Use Secrets: Never hardcode sensitive information
• Keep Jobs Focused: Each job should have a single responsibility
• Optimize Caching: Use caching to speed up workflow runs
• Use Matrix Builds: Test across multiple environments
• Monitor and Log: Ensure comprehensive logging

Conclusion

GitHub Workflows provide a powerful and a flexible way to automate your software development processes. By understanding the basic concepts and gradually building more complex workflows, you can significantly improve your development efficiency and code quality.