Why does my development project need a devops pipeline?

So the question is, why should you even bother to create a continuous delivery pipeline.

I will explain by narrating a real incident that happened while working on a large project with multiple applications integrated to provide end to end functionalities.

I was working on this project a few years ago. The launch date for a major functionality was close when the customer asked us to accommodate changes due to new rules and regulations that had been declared for the industry. Besides there were unexpected organization wide changes.

So the next iteration saw the team working on code changes for the functionality that was getting ready to be launched. Due to ad-hoc changes to the code, lot of code got broken and bugs got introduced.

Then there was another challenge, in order to incorporate even a minor feedback suggested by the customer, we had to follow the complete engineering process.

The engineering process included development/ code changes, code review, unit testing, functional testing, regression testing and testing on staging servers before it could be made ready to be shown to customer. So a change requiring just 1 to 2 hrs of development time would usually be available for customer feedback on staging after 2 to 3 days.

This was not acceptable to the customer as it delayed the feedback cycle and chances of achieving our goal to launch the functionality by end of the iteration. The customer also went ahead to narrate how another vendor team of theirs was able to make the changes almost immediately and finalize the functionality with customer feedback in the same meeting, so why couldn’t we.

So we questioned ourselves how we could deliver with the agility required by the customer without compromising on the quality? 

After analysis, we got the following breakup on the cause of delays and came up with recommendations to address each one of them:

1. Multiple functional testing cycles

Unit testing done was bare minimum. It was restricted to the new fix. There were no sanity unit test cases which ensured that the new fix has not broken an existing functionality. So additional issues got raised after deploying the fix.

Recommendation: Automated unit test cases to be written so that all related test cases could also be run with the new change.

2. Performance issues

There were new issues related to response time, application crash due to memory leaks etc as Coding guidelines were not being followed properly in some deliveries and manual reviews sometimes missed the code optimization. 

Recommendation: Static analysis/ code review tool to be used to maintain code consistency in code quality across releases.

3. Time taken by Regression testing

Regression testing took the maximum time before functionality could be deployed to staging. Due to complex integrations between applications, all major integration test cases had to be run which were run manually.

Recommendation: Functional testing to be automated. Regression testing suite to include a subset of automated test cases. 

4. Sanity testing on staging

On staging server, which was a replica of the production server and had almost real time data required a major subset of regression cases to be run.

Recommendation: Again an automated test suite could complete the job quickly. 

5. Inter-team coordination and prioritization

Once the developers release a functionality to QA team, they move on to next development task. In the meantime, when QA logs a bug, developers juggle their attention to release a new functionality and to resolve the bug. Most of the time, they would not even have checked the new bugs logged in their name as they were too involved in working on the new functionality. Similarly after fixing the bug, some developers take their own sweet time to mark it as fixed in the tool. These issues then get highlighted during daily stand ups. However, many hours get wasted before everyone is updated on the status. 

Recommendation: What if the developer would get to know just a few minutes after checking in the code, that there are functional issues with the code, wouldn’t it be a big time saver! This can be achieved with a continuous delivery pipeline.

6. Bug cannot be reproduced

This is a legendary response to a decent no. of defects raised by QA that it was working fine on development server. Well this the most common issue where most of the tine gets spent in analyzing the actual problem. The problem could be related to deployment, configurations or data or any other reason which is more often than not related to manual mode of configuration and deployment.

Recommendation: We came up with a step wise recommendation here. First step is to follow a infrastructure as a code mechanism to configure server and to make automated deployments. As a second step, it was recommended to create environments like dev, test and staging servers using containers and destroy them once the task for which they were created got completed. This should be achieved with infrastructure as code so that same code can be followed to configure infrastructure and make deployments across environments. If you are still on long away from containers, then at least have an automated code based deployment on all the environments which can be used for production as well.

The Next Step

As you go through the recommendations, you would realize that the resolution of all these concerns lies in automating the process to improve productivity and quality.

Well once you implement the above recommendations and you would have automated unit tests, static analysis, automated functional tests, a build triggered on checking code in source code version control, infrastructure as a code, automated deployment, and an automated pipeline to connect all these. And Voila you have built a devops pipeline!!