CICD Patterns with GitHub Actions and Docker
CICD is a shorthand to refer to continuous integration and continuous delivery/deployment. CICD includes automated building, testing and deployment of applications. It connects development and operation activities and teams responsible for these stages in application life cycles.
CICD often builds on version control (i.e. GitHub, GitLab, Bitbucket), and hosted CICD services (GitHub actions, GitLab pipelines, Bitbucket pipelines, and many others like Travis, CircleCI, etc.). The workflow looks like this:
- Commit your changes to the git remote
- Changes on the remote trigger the CICD pipeline
- If the steps in the pipeline are successful a new delivery or deployment is initiated
The difference between continuous delivery and continuous deployment is that delivery usually means that all the artifacts are delivered that are required for the deployment, but deployment happens in a separate step. Continuous deployment means that a new version is deployed without manual intervention. Think like your Shiny app is included in a Docker image (delivered) vs. the new image is deployed as a running container.
However, when it comes to Docker-based delivery and deployments, there are multiple ways how CICD can be set up. In this post, I review some of the common patterns I have seen in Docker-related CICD workflows.
Why should you care?
Why is this whole CICD thing important? On one side, it just makes your life easier by automating some of the steps that would otherwise require you to run tests, copy files, push/pull images, etc. Making sure that no human error occurred along the way. By removing the human element and relying on great tooling, your chances of making silly mistakes are hugely eliminated.
On the other side, platform-as-a-service (PaaS) offerings often include some form of integration with major hosted git provides, such as GitHub and GitLab (this latter can be self-hosted too). Therefore, it is a good idea to take advantage of these integrations to reap the benefits of automation.
Non-git-based delivery and deployment
By non-git based I mean that the source code management and the delivery/deployment ideas of your activities are independent. You use git to manage your codebase (versioning). From the same code base, you make sure that the right artifacts(files, images) are built, tested, delivered, and deployed.
Local image build
In this most basic scenario, you version your codebase. You build Docker images locally. You push the images to a Docker registry. When you deploy the image, you trigger the pull yourself (via command line or user interface) which then initiates a new deployment.
This is not a very common pattern in production. It is most likely that you start with something like this when you are just experimenting.
An example would be how to deploy Docker container images to virtual machines or Kubernetes clusters using OpenFaaS. In this case, the faas-cli is used to build images locally, push these into a registry, then instruct the servers to pull the updated images and deploy them.
PaaS-based image build
In this scenario, the version control is still independent of the deployment process in the sense that GitHub won't instruct your PaaS provider about what to do. It is your responsibility. Once you push a new version of your app, PaaS takes care of the build and deployment processes.
An example of this scenario is deploying apps to Heroku from the command line interface (CLI). The twist here is Heroku CLI sets up a new remote on Heroku's servers. So in a way, git instructs the PaaS about the deployment. But you also keep your other remote (GitHub, etc.) to version your code. When you push a new git commit (the event), the Heroku remote will kick off the Docker build process and the deployment of the new version for you.
Enter GitOps
GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD, and applies them to infrastructure automation. – GitLab Docs
This means that the remote git platform is responsible for all downstream events. Secure communication between different service provides (GitHub and your PaaS) is set up using access tokens (revocable "secrets").
The advantage of this setup is that a lot more steps can be automated within the same workflow, like testing etc. This will make the process more robust.
Git-triggered and PaaS-based image build
This setup is very similar to the previous, PaaS-based image build. The difference is that the build process is triggered by the code hosting platform, GitHub for example. Pushing the new commit, often to a certain pre-configured branch, will result in a webhook alerting the PaaS provider. The PaaS servers then launch the deployment process.
GitHub will only deliver the news to the PaaS, but no GitHub actions are involved here. Examples for this scenario include the GitHub integration with the DigitalOcean App Platform. Setup is usually super simple, but the ease of setup is at the expense of full control over how exactly the process is taking place. If you need more control then see the next option.
Git-triggered and GitHub-based image build
The last scenario circles back to the 1st option, but instead of everything happening locally, the whole build/deployment process is happening on remote servers. The git push event kicks off the GitHub action (or similar) CICD process. The Docker image build happens inside the GitHub actions runners.
This setup and the required GitHub actions steps give you the most control of how you want things built and delivered. For example, you can specify if you want to cache the Docker image layers. This is also ideal to make sure new deployment only happens if all steps are successful. E.g. testing your code might find errors in unit tests. You don't want to release a new version until the issues are dealt with.
Here are a few examples:
- Heroku deployment with GitGub actions
- DigitalOcean app platform with image caching example
- CICD with the Fly.io platform with a Shiny example workflow
- Functions and microservices with OpenFaaS and GitHub actions
When there is a command-line tool to interact with the PaaS, you can most likely set up a GitHub actions workflow with that.
Conclusions
CICD practices are important for a robust and automated developer experience. Knowing that the workflows are in order gives you peace of mind. Deploying containerized applications adds a few extra steps to your workflows. As a result, there are multiple ways of approaching it.
Some workflows are simpler, others are more complex. You can start simple and add complexity as your needs evolve. Try easy integrations when prototyping, add more serious testing and GitHub actions as your application matures. Once you get the hang of it, CICD becomes second nature. Tooling is getting better every day.