Setup DevOps CI CD pipeline by using docker containers and Image promotion 

If you write code and then check it into the version control, build the code and deploy on your dev server. Then deploy the same build artifacts onto your test server and post testing deploy on staging and on production. You might also use a orchestration tool for continuous delivery and use tools across your software engineering lifecycle. But you still might be living in a bubble. Why? Because the software engineering world has moved onto Everything as Code.

The Gap

So you might question here. I have my source code , test scripts, and even pipeline as code for my continuous integration pipeline. So where exactly is the gap?

The big gap might just be infrastructure as code. Usually as a development team, we feel we don’t really completely own the activities related to infrastructure setup. Hence we de-prioritize the task of automation of environments or infrastructure and move them towards the end of our activity list. Similarly we might de-prioritize the infrastructure automation if we are Operations team. We might argue here that writing deployment scripts should be the responsibility of the development team. So this area is at times left uncatered for long unless there is a separate devops team.

But is it enough to write the deployment scripts and run them across the environments?

Well, this might sound like a reasonable solution if you are still a devops newbie, as this does seem to provide consistency across environments. However, this may not turn out to be a good solution in the real world. Why?

First of all, we have built our code and tested on a particular machine. There might be some dependencies or pre requisites for running the code which might not have been installed on all the machines or VMs. So unless, we are planning to re-install the complete software on the machine and not just related to the application, we might face such dependencies like “code is working on dev but not on test server”.

The Solution

The simple solution in this case is to build docker images. Docker images can be saved as code and can also be propagated from one system to another. Hence, we can easily propagate the image from dev environment to QA, to staging and then to production while avoiding lot of issues due to system dependencies in the process.

Image Promotion Pipeline

Hence there is a need for a sound architectural approach here. A recommended approach would be to build once and promote docker images from dev environment to the subsequent environments rather than promoting the deployment scripts or even dockerfiles.

This is because the latest version of docker image might change in the docker hub, the docker images might have got updated at the backend while still showing the same version. The other APIs on which our application has dependency might get revised. What this implies is that as one moves from dev to subsequent environments by building docker image individually for each environment, the docker image might get modified even though it has been built from the same dockerfile. Hence, this is where the image promotion pipeline helps.

There are multiple tools available which allow for this kind of image promotion pipelines without spending the effort to build the capability inhouse. For example, the tools available in the binary repository management space like sonatype nexus or jfrog artifactory to name just a couple.

Benefit

This architecture approach will help in many ways:

1. The issue of docker image getting modified without intent will get prevented.

2. “The bug cannot be reproduced in my environment” type of typical issues will get drastically reduced.

3. Testing on staging will be reduced as new environment related issues will be prevented.

4. Reduced chances of issues in production during new deployments.

5. This is more of a necessity for microservices deployment. Deploying microservices directly on virtual machines might prove to be expensive.

6. If you are working with microservices, you can use the same hardware to test and deploy multiple microservices in parallel and hence improve productivity and time to market.

Conclusion

Now coming back to the question “Do I need to jump on the docker container bandwagon to improve my CI CD pipeline? Well, I feel the question should have already got answered by now but just in case if you are still looking for an answer: Better sooner than later”