How do your business and product teams engage in innovation and experimentation quickly? By having the luxury of making changes fast while not worrying if some new feature will break or will not be successful. We can unlock this agility by eliminating potential points of waste in our development and delivery processes.

The goal of a good Continuous Integration / Continuous Delivery (CI/CD) process should be to reliably deploy ‘good’ ideas to production quickly (least amount of time possible between code/feature check-in and production deployment). If an idea is ‘good’ or not, can only be truly known once in production, therefore the need to try changes quickly.

Every CI/CD process contains these four high level phases – Source, Build, Test and Production. Continuous Integration covers the first two, while Continuous Delivery covers all the phases and creates a deployable artifact, albeit with a manual control gate before production. Continuous Deployment automates that final gate as well.

Here are some practical guidelines, best practices and learnings we have seen while trying to move our CI/CD processes to the next level. I have organized them under five categories, as follows.

    1. CI/CD Workflow Infrastructure
      • Automate provisioning of your build/deploy infrastructure and resources – using CFN, Terraform etc.
      • Have a good Test/Deployment Management tool/service – like Jenkins / AWS CodePipeline – to execute and visualize the various stages. This will help provide clarity to the ‘tribal knowledge’ as it has the following benefits
        • Models the pipeline through a visual workflow, spreads the knowledge and simplifies understanding of your deployment processes end-to-end
        • Smoothens the operation, by providing ways to incorporate “stop the line” capabilities, manual approval steps between stages etc.
      • Optimize for low spend i.e. pay as little as you can
        • Use spot instances for building, or
        • Leverage pay as you go managed service like AWS CodeBuild, that scales with your needs
    2. Build Phase
      • Centrally manage app configuration files
        • Move your ‘’ or similar application config file from your deployment package to a central repository like S3
      • Manage your secrets by moving them out of config files and placing them in a encrypted secret store. Ensure the store has good permissions security and also highly available
      • Pin external dependencies to specific versions – i.e. not using the “Latest” tag. This will help provide predictability for your builds
      • Package only runtime requirements for production, for example, not build tools, just the application and what’s required to run the executable
      • Optimize for speed
        • layer caching or extracting a base image
        • co-locate build with artifact repo
    3. Testing
      • Get good at managing the multiple environments before the prod environment. You might want to consider a separate environment each for Beta, Integration, Security/Load Testing etc. Simplify this with automation and incorporation into the CI/CD pipeline stages
      • Automate testing in a Beta or Test environments
        • Canary testing – initially some simple URL based testing and move on to advanced acceptance testing
        • Load testing using a home-grown service or something like BlazeMeter
        • Security testing in all phases – Build, Image creation and hardening, Infrastructure creation, Integration and finally Regular operations’ phases. For more details see my blog on InformationWeek about security in DevOps.
    4. Deployment
      • Get creative in your deployments. Choose an option that fits your business needs and yes, definitely automate
        • In-place
          • Doubling the size of your cluster – if your app/cost permit it
          • Rolling – if we don’t want to grow above 100% ever, but can shrink during deployment
        • Canary – a subset of requests goes to the new revision – helps monitor metrics before full deployment
        • Blue/Green
          • with DNS Swap – building parallel infrastructure
          • target group swap using Host header routing
        • Allow updates of individual components. Microservices, anyone?
        • Know if your application works against the same database schema when you release new app code. You would think this is obvious but this is an easy miss for beginning micro-service deployments
        • Consider possibly automating application deployment into a separate region – this can act as your Disaster Recovery as well
        • Boost your confidence levels by integrating automated acceptance testing against any new environments or a canary before cutover
    5. Monitoring & Operations
      • Health checks help prevent many a failed deploys. Use them to your advantage
      • ensure solid notifications for your CI/CD process – for example, CI/CD process changes like code commits, status changes for builds, tests and deployments with corresponding results
      • try to
      • automate your operations wherever possible
      • have governance for compliance with the company’s policies around security, industry regulations etc. – using a tool/service like
        • AWS Config that allows you to assess, audit and evaluate your resources using rules, or
        • service like SunGard AS’s managed offering that provides among other things
          • best practice access management using temporary credentials/roles and
          • intuitive rules based configuration approach for better cloud footprint management
          • learn from your logs and metrics by applying machine learning
          • automate your operations

Achieving a self-healing, fully automated, secure and quality focused CI/CD state requires a diligent approach. The patterns and practices listed in this article around advanced aspects within automated building, testing, deployment, monitoring and operations will help guide you on next steps.

Just as a popular meme goes – “Stay curious my friends” – I hope this gives you something to think about on your DevOps maturity journey.

Kiran Chitturi is a technology leader on the Sungard AS emerging tech team. With 18+ years of experience, Chitturi was instrumental in Capital One’s cloud transformation journey and in setting up AIG’s next-generation incident response automation architecture. He has a master’s in Computer Science

Comments are closed.