Need Frequent and High Quality Releases? Automate and Monitor

This is a guest blog post from Krishnan Badrinarayanan from ZeroTurnaround, creator of JRebel & LiveRebel, the productivity tools that make Java more productive for development teams.


Users and customers today have rapidly changing needs, expect 24/7 connectivity and reliability, and access services over smartphones, tablets and PCs. As software product vendors — irrespective of whether we are in development and operations – we need to push updates frequently to satiate our users’ needs. Or else, we risk losing them to competitors or alternatives. Nothing blows more than unhappy customers or declining revenues.

Responding to User Demands

What do we do to stay ahead? We release updates faster — welcome to the devops software race! As a software team, we step up to the plate and:

  • draw out product roadmaps with frequent releases
  • organize around small teams and amp up agile practices
  • an automate where possible (artifacts, tests, configuration, etc.)
  • deploy away in a hurry and take a deep breath!

Speaking to the last point, the unfortunate truth is that frequent change increases the likelihood of production errors and instability. You can release updates only so fast without ensuring that you haven’t made any mistakes. Also, with frequent deployments often come some application downtime and disrupted users. Sounds like no fun, right?

Our question:

How do we release high quality software updates frequently without disrupting or degrading end user experience?

To start with, we run faster! But we can only do so for a while before we wear ourselves out.

What’s holding us back from increasing release velocity?
Here are two blockers (I’m sure there are others!) that hold us back from delivering awesome updates frequently while ensuring quality end user experience.

  • Tedious release processes: Developers have a multitude of collaboration tools, Kanban boards, build tools and agile practices, and can build features quickly. But once it comes to releasing these changes through QA, staging and production, the process is more manual, ad hoc and slow.
  • Unreliability in production: Applications that flawlessly run in development sometimes fall short or fail in production. Production environments interact with third party services and are subject to slow internet connections, regional ISP’s, firewalls and so on.

Let’s start by addressing them in order to become DevOps mavens that deliver only the highest quality of software.


Now that we’ve broadly identified two blockers that stand between us and DevOps utopia, let’s address them.

Collaborate on a Singular Goal
It’s no surprise that Development teams and Operations teams think about things differently, but have you ever considered that what makes both teams work best are actually diametrically opposed? Devs push for change and ops drive for stability.

What we need is a culture where development and operations teams unite, to draw from each other’s experiences, perspectives, concerns, and continuously improve processes to build and deliver resilient products rapidly, into the hands of delighted customers. This requires discipline, a sense of mutual respect between individuals, teams and even departments, focus on the big picture, and a strong commitment towards delivering on business goals.

Streamline and Automate Release Processes
Developers enjoy streamlined processes with their collaboration and build tools to produce releases effectively. However, the folks that release software through QA, staging and to end users do not share the same good fortune. Quite often, they get artifacts, scripts and resources from different members of the team, and run them in sequence in order to deploy and test the application. If one of the scripts happen to be erroneous or of the wrong version, the deployment will fail. This lack of coordination and automation greatly slows down the release process.

Here’s a plan for you.

Step 1: Critique Your Processes
Take a step back and map your portfolio of applications, and their release processes.

  • Processes: Where do the dependencies lie? Are there any commonalities or redundant steps in the processes? Can they be consolidated?
  • Bottlenecks: Where do the bottlenecks lie and what’s causing them? How expensive is each bottleneck in terms of idle resources downstream, delayed time to market, quality, and other key metrics based on your situation? Can they be ironed-out by adding more resources or refining processes?

Start with small opportunities that can bring big improvements.

Step 2: Automate
Once you have an optimized release process – not just in terms of tasks but also in terms of assembling and grouping release deliverables in a more consistent manner, look to automate repetitive and error-prone processes. Time and effort saved here can be put towards works on projects that innovate and add value to your users.

Depending on how far along your are on your project or product lifecycle, you can attempt to automate your optimized release process. Check out this report titled “Release Management for Enterprises” that discusses the issues and solutions to release management. It then goes on to building an automated release pipeline that can enable you to practice continuous delivery. What you will need is:

  • a source code repository like Github
  • a continuous integration tool like Jenkins or Bamboo to generate builds
  • automated unit, integration and acceptance tests for these builds
  • an artifact repository to store these builds
  • LiveRebel to automate deployments – code, db and config – with zero downtime


The image above lays out what you can achieve when you safely automate your release pipeline with all the right checks and balances.

Now you’re ready to increase release velocity as needed, to the maximum.

Once Released, Monitor for Quality
By automating each of the steps in the release pipeline, you have already set yourself up for increasing release velocity while ensuring high quality.

However, no amount of preparation can ensure predicted results in production. Users tend to use software in the most unique ways that lead to undesirable outcomes. Network traffic, third party services and local ISPs or browsers can do just as much damage to user experience. The internet is rife with accounts of websites going down, causing brand damage and revenue. These outages are significantly noticeable and very public.

What About the Subtle Changes in Quality?
Watch out for these. They will alter your end user experience costing you customer acquisition and brand loyalty. It is death by a million pinpricks. Nobody, ourselves included, enjoy using a slow application or a site that throws JavaScript or 404 errors. User wait time tolerance for videos is 2 seconds, after which they will abandon the site. This number is even lower for other types of sites!

Enter New Relic, a performance monitoring tool that can help identify prime violators that degrade application performance and behavior, and diagnose them.

Look for Symptoms
Monitor your application for performance and behavior in production. Make this an ongoing activity during peak and off-peak hours alike. A keen insight into your application and infrastructure health can reveal snags that would otherwise go unnoticed. You really want to look for symptoms that compromise end user experience, such as:

  • how are my servers and infrastructure performing at current capacity?
  • how is one location faring compared to others?
  • what parts of my application – internal or third party – impact performance?
Diagnose Issues and Make Decisions

Use New Relic to diagnose why these symptoms occur. Are they because of poor code, network conditions, server capacity or third party services? What do the logs and diagnostics say? What conclusions would you make based on the evidence? What can you package up for your development team so that they can investigate further?

The goal is to clearly understand the root cause of issues that impact your end user experience and make decisions on what can be optimized to ensure that the symptoms don’t occur again.

Aim for Continuous Improvement
Especially in the case of rapid release cycles (or continuous delivery), constant monitoring is a necessity. With every iteration, identify symptoms diagnose them and present action items that  should be taken into consideration during the next development iteration.

Rapid Release Cycle

The goal is continuous improvement. To push farther to improve application quality and user experience with each release. This of course comes with the added benefit of catching issues early and holding parties accountable for services they provide.

Next Steps – Tool Up for Continuous Delivery With a Feedback Loop
Build your own automated release pipeline to practice continuous delivery safely and put a strong feedback loop in place to diagnose issues before your end users see them. Most of the tools required are either open source (Jenkins, Artifactory, Arquillian, Selenium) or can be acquired for free (LiveRebel, New Relic, GitHub).

If you have any questions or comments, send me a tweet at @LiveRebel. Enjoy!

My Bio:'

View posts by .

Interested in writing for New Relic Blog? Send us a pitch!