Customer reliability engineering

I was blown away when I spoke with Dave Rensin during my visit to the Google campus. He is Google's Director of Customer Reliability Engineering, and his views on customer support are world-class.

Dave's goal is to drive customer anxiety to zero; to remove the things that would cause customers to want to leave.

To do this, Dave's team has several principles they follow:

1. Make sure your customer understands they are not alone. They need to feel a sense that "we are in this together" and you will stay with them until the problem is solved.
   
2. Ensure customers never feel like they are talking in a vacuum. Never let the customer feel like you know more than you are telling them. Tell your customer all the details (without sensitive info, of course).
   
3. Create a shared fate with your customers. Arbitraging issues with money (or credit) is not good enough. Dave's team reviews their customers' production systems and provide guidance on how to make it up to Google's standards. If the customer meets those standards, they will then identify when issues are caused by their customer's system, and proactively reach out with possible solutions (utilizing a shared dashboard both his team and the customer can see) -- and they do that for $0. This drives mutual accountability; truly being "in it together."

Dave goes into some details of the above in the video below. More details can be found in Dave's Google Blog post.

Do DevOps

DevOps is not a buzzword; it is the way quality software gets deployed fast.

In order for software teams to truly embrace DevOps, they must have an inherent continuous improvement culture which embraces ruthless amounts of automation. Many of my examples below will be Java-specific, but this can apply to all types of software languages.

The deployment pipeline
Your deployment pipeline is critical to enabling speed, so I will expand a bit more here. Some questions to ask yourself: How often do you deploy code to production? How long are your builds? How long does it take to do a production deployment? How often do we have bugs in production? Staging/UAT? Dev? The answers may vary based on many factors, but odds are, you can improve dramatically in all areas.

• Continuous integration. Enabling a distributed group of developers to integrate their local code into a shared development environment as efficiently as possible is the key first step. Generally a build server (like Jenkins or Bamboo) can help to enable this. Most importantly, though, are the automated tests which run on the code before moving it to development. These can be things like PMD or SonarQube which check for best practice violations, standards, or bugs. Similarly, unit, integration, and security tests can and should be run here. The key is code is not allowed to move to development until all tests are passed. We strive for quality, production-ready code even in development.
  
  
• Peer code reviews.* This is probably the only manual step of the deployment process. Having an additional pair of (usually senior- or architect-level) eyes helps to drive team standards, code re-use, scalability, security, and efficiency. Some teams may find it hard to incorporate this critical step, but it must become part of the process.
  
  
• Automated testing. Automated tests can occur at each stage, either with each build (depending on speed), or some regular rhythm (like nightly). These tests can be regression, smoke tests, integration, or performance tests. Visibility of the results are key, as test failures must be addressed promptly. Regular testing also helps to ensure tests stay current. As the test suite grows to have a comfortable percentage of coverage, code can move faster to production with less manual testing.
  
  
• Auto-build, auto-deploy. The build servers mentioned above can automate the process of building and deploying code to each environment. Moving to production may require additional steps due to segregation of duties and change controls. As a result, I recommend making everything standard changes -- this way a change control ticket can be opened automatically by the deployment process rather than requiring manual change controls to be approved. In the lower environments, builds and deploys can be scheduled automatically or occur automatically once new code is committed.
  
  
• Same artifact in each environment. Consistency is key in ensuring quality. Using the same artifact (or Docker image if you use Docker) throughout each environment minimizes variability.
  
  
• Visibility. It is important that with all of the above it is easily accessible and visible to all stakeholders -- from the project managers to the developers. Broken builds, for example, need to be remediated fast as they prevent code from moving for the other developers.
  
  
• Forward and back. Getting to production quickly is important, but it is also imperative to have a way to revert deployments fast. Your pipeline should support this.

Configuration management
Configuring and managing environments in a streamlined and automated fashion enables speed and consistency. Configuration management tools like Puppet or Chef enable centralized management of multiple servers at once. This is key to being able to quickly spin up or down new environments as needed, patch, or ensure the same settings are applied to each without individually tending to each.

These tools can also be used to push software to desktops. This is useful for a team of developers looking to ensure everyone has the same version and configuration of tools on their machines at all times. It also helps with installing those tools as it can literally be a simple double click and go get a cup of coffee.


Containers & container orchestration
Step aside VM's, containers are the new thing. Docker containers wrap your software in a complete filesystem. It is more lightweight than a VM, and enable speed through ensuring standardization of the environment. Their small size means you can have several containers inside one VM. The key point being that containers enable true application portability, as they abstract the underlying infrastructure from the app itself.

As your environment grows with more and more containers, orchestration tools like Kubernetes become important to help manage them all from a central place.


Situational awareness
It is key for the team to know the health of the system at all times. It encompasses the following:

• Monitoring. A constant pulse on the key metrics (response times, CPU usage, server memory, etc.) allows for quick identification of potential issues and can help prevent failures. Tools like Icinga can even automate the creation of monitors when setting up servers through Puppet, for example. I recommend making as much of these metrics visible using tools like Graphite, StatsD, and Grafana.
   
• Logging. Having additional details at hand help to give more insight into the various systems. Centralizing logging outputs using the ELK stack (Elasticsearch, Logstash, Kibana), or using tools like Takipi can help to reduce the time it takes to remediate issues.
  
  
• Alerting. In addition to visual dashboards, automated alerting of key thresholds plays a key role in ensuring timely resolution of issues. A tool like Seyren can be useful here in conjunction with Graphite.

Zero downtime
Who likes staying late or working on the weekend to push new code live? No one. One of the main reasons why this occurs is because many deployments incur downtime in some fashion. With a streamlined pipeline, and a little help from Docker, staying late may become a thing of the past.

When we push new code live, we launch a second Docker container in production and point only our internal network traffic to it using Vulcan. If all tests pass, we point all traffic to the new container (using Redis to maintain sessions) and we are live without any downtime! The same can be done in lower environments as well.


Conclusion
Ultimately we want to achieve a continuous delivery state, where code changes have the potential to go live very quickly, with high assurance of quality at each step. Visibility is key to this process, as it ensures everyone is on the same page.

Lastly, the term DevOps is the combination of Development and Operations. Traditionally development teams and operations teams have competing priorities: devs want to move code to production fast; ops wants to keep the environment stable. With DevOps, the developers take more ownership throughout the process, while operations get involved earlier, more automated tools, and better visibility of the pipeline. The partnership is what drives great business results.


*Side note on peer code reviews: There may be a times where code reviews seem a bit of a burden.

First, when refactoring is required/requested by the reviewing person. Refactoring is an important and natural part of keeping the code base in good order over time. There may be times when refactoring may not be possible due to time, which I would then suggest that a user story (requirement in Agile) is added to the top of the backlog and done in a subsequent sprint (keeping in mind that there is nothing more permanent than temporary code). If you are following Scrum, ensure your teams do not consider their user stories to be "done" until all the code review comments are addressed.

Second, when there are disagreements between the reviewer and the developer. This is pretty simple to resolve, especially when the reviewer is an architect -- the developer does what the architect says. Discussions are always welcome, but tie goes to the architect.

MVP: failing fast is a good thing

How many times has your team had a scenario similar to the following example?

A senior client says a product cannot launch until one particular feature is added. Her reasoning: "No one will ever use this product unless this feature is included."

Although small in size, the feature is quite complex and would require another 2 months of development effort. The team works tirelessly those two months, and launches the product shortly thereafter.

A few weeks after go-live, the analytics demonstrate an unfortunate reality: while the product overall is getting good traction, no customers are actually using the feature suggested by the senior exec.

This scenario is exactly what Eric Ries' book, The Lean Startup, tries to address. In it he assesses how successful software development companies build and launch products.

Minimum Viable Product (MVP)
Ries describes the MVP as the product whose minimum set of features allow for learning from early adopters. Using the MVP, we are able to avoid building products no one wants, and maximize the learning per dollar spent.

The image below takes us through Ries' build-measure-learn feedback loop. An idea is formed and then built and released as an MVP. That MVP contains measurements or ways to pull data which we can learn from. From there, the product team is prepared to act on that data, and pivot or iterate.

Through small increments we can continue to test hypotheses and build a better product by minimizing the time through the feedback loop. If a particular feature or iteration is not successful, we learn early in the process through facts (analytics, metrics, user feedback, etc.).

This means failing fast is a good thing! Validated learning means we do not have to wait months before we find out no one will use a particular feature. We spend more time on things we know the users will want.

Image credit: Eric Ries, TheLeanStartup.com

Putting it into practice
I like to think of MVP as happening at each phase of the software development life cycle, in addition to the product viewed as a whole.

Take the design phase, for instance. Low fidelity mock-ups (think black-and-white, hand-sketched) are key because they speed time through "the loop." The goal is to get feedback fast -- how can you get fast feedback if you're spending time perfecting the shade of blue a button should be?

When it comes to the product owner's vision for features in release planning, how many of them build on top of an unvalidated hypothesis? What can be built and released quickly as an MVP instead?

Teams must be prepared to iterate. This means we cannot launch something and forget about it. We must release our MVP's, analyze the results, and pivot (move in a different direction vs. our hypothesis) or iterate based on our learning.

It is important to remember software has no value until it is in the hands of the user. The MVP approach gets more engaging software to the users faster by adapting incrementally.

Creating the time to innovate -- Part I

I am frequently approached by leadership from other divisions asking how my teams find the time to be so innovative. I propose that it is not finding the time, but rather creating the time. We all need more time in the day, but if you create a culture which inspires quality, you will naturally have the time you've been looking for.

Culture
A culture of continuous improvement runs through my team's veins. When inefficiencies arise, the team identifies solutions to improve productivity.

Team members are encouraged to give back to the team (I call this "team community service") by proposing and implementing better ways of doing things. Generally it's about 20% of their time (equating to about 1 day per week).

The key is not assigning tasks nor me saying what to do, but rather giving each individual a blank slate to identify and contribute to the areas they are most passionate about. (See previous post about motivation.)

Quality
Where do we get the time to implement these solutions? We have a constant pulse on things which prevent us from working on value-add tasks. From here we identify where we need to simplify or improve quality. These improvements in quality add up to very large time savings.

At first the team uses this extra time to catch up on value-add work and achieve a consistent flow. However, once we achieve optimal flow, we use the extra time gained to continue to innovate, gradually reaching the magic 20% time for each individual.

Suggestions to get started
Analyze your team's errors, production bugs, defects, and other distractions which require someone to stop what they're doing and spend time fixing issues. Use the 80/20 rule to determine the 20% of items causing 80% of the issues, and start to eliminate them.

It may be difficult at first, but leverage key team member strengths and passions. Some folks will not mind putting in extra effort, especially if it means helping the team in the long run and working on something they enjoy.

Be sure to track your team's progress. Take a baseline of key metrics today (number of production defects, average time spent fixing issues, etc.), and track improvements along the way.

The key is to have a tipping point in mind: when do you stop giving the time saved back to "business as usual" work and start giving it to "team community service?" Some individuals may only be able to reach 10%, while others may reach 20% or more.

You will see that 10-20% of time spent on innovation and continuous improvement will produce 2-10x gains for your team in the long run. Create the time to do it.


Update: Read Part II of this topic here.