Data, data, data

"Data, data, data" is the new "location, location, location."

Uber owns no taxis, yet is the largest taxi company in the world. AirBNB owns no real estate, yet has the most accommodations in the world. These companies run their businesses on data, and lots of it.

Data is king, and it is only becoming more important. Proper analysis and utilization of data helps to uncover the what, the why, and even predict the future. As a result, data must be a core component of your digital strategy.

Hindsight
At the most basic level, data gives us hindsight. A simple example is how grocery stores utilize loyalty cards. Customers sign up for them with some basic personal information, and in return the store gives the customers discounts when they use their card. Data collected from these cards helps the grocer identify individual purchasing habits -- it gives them hindsight.

This is why online retailers encourage customers to create accounts. The data collected (which products are being viewed, which terms are being searched for, etc.) all help track what is happening in their store.

Insight
Understanding the "what" is just the basics when it comes to data analytics. Having the view into the "why" provides insight.

Why do certain customers buy one product over another? Why do certain products sell better at certain times of the year? These are the types of questions the data can help provide insight into.

Foresight
Being able to predict behavior is the next step; this is where the most positive transformation can occur for an organization.

Again using grocers as an example, stores can use big data to predict and suggest the price points of certain products at certain times to ensure the right amount is in stock and fresh. If the price of strawberries, for example, is too high grocers risk having too many in stock and the strawberries going bad. If they accurately predict the right price point, they can keep the right amount moving off the shelves at a pace that ensures each package is still fresh.

Lastly, there are some scenarios where proper data analysis can actually help to prescribe some actions. In other words, using data can help make things happen.

Let us use the car company Fiat as a fictitious example for this. If Fiat mined the publicly available social media posts specifically looking for terms which suggest a propensity to buy their car, they may be able to help drive more sales. The scenario could go something like this: John Smith posts to Twitter, "Thinking about buying the new Fiat. Can't decide between that or the Toyota Prius." That post will get picked up in Fiat's social media scanning algorithm, and alert the salesperson in John's region to contact him directly. That contact may help to influence John in purchasing a Fiat.

Making it happen in your organization
To leverage data effectively, naturally you need data. Determine the sources, and if none exist start setting up your data collection processes.

Once you have the data, it needs to be usable. Having it in 25 disparate systems will make life tough. Rather centralizing it and "cleansing" it for use (i.e. ensuring accuracy, removing duplicates, etc.) is key.

Additionally, data can help create a source of revenue. Identify any data which may be unique to your organization which others externally may pay to access. Ensure proper usage controls and governance are in place.

Also keep in mind potential external integrations or partnerships.

Ultimately, there are endless possibilities to how you can utilize data. Start small, take an MVP approach, and build from there as you learn what works for your organization.

Be open: Integrate and let integrate

A key principle I drive at my organization is technical openness. This means all the tech we leverage should be based on open standards and frameworks. There are many reasons for this, including:

• Superior interoperability and integration with other systems.
• Prevents "re-inventing the wheel."
• Avoids being locked into proprietary and costly technologies or vendors.
• Improves agility and choice; can select best-of-breed solutions for each job.
• Broadens support pool and timelines.
• Increases innovation, as open standards invite everyone to participate in providing feedback.

I believe the ability to integrate fast and effectively is a skill which all companies will need to survive over the next few years. This is why the first bullet above is most critical.

Examples of key business integrations 

Here are a few "integrations" which help drive business growth:

1. Pizza Hut can be ordered directly through Amazon Echo (Alexa). Amazon provides vendors a standard way of connecting to their Echo service, and companies like Pizza Hut are able to connect their ordering systems to allow for another potential revenue stream. Pizza Hut was one of the first onto the platform because their systems allow for integrations with external sources to place orders.
   
   
2. Uber is a great example of being able to integrate with various channels. Users can request rides directly within both Google Maps and Facebook Messenger. They try to capitalize on being available to request a ride at the exact moment when someone is likely to need one.

The examples above demonstrate the need for enabling the in-the-moment, simple, and fluid purchasing capabilities. None of which would be possible if the systems were closed and unable to move quickly to meet the changing dynamics of their users. 

There are other examples which do not include purchases, but rather provide information or other service more easily through atypical channels (see KLM's Messenger integration, for example). Those help to drive customer engagement, satisfaction, and loyalty. All wins for good business, and only possible with open technologies.

Integrate and let integrate!

Do DevOps

DevOps is not a buzzword; it is the way quality software gets deployed fast.

In order for software teams to truly embrace DevOps, they must have an inherent continuous improvement culture which embraces ruthless amounts of automation. Many of my examples below will be Java-specific, but this can apply to all types of software languages.

The deployment pipeline
Your deployment pipeline is critical to enabling speed, so I will expand a bit more here. Some questions to ask yourself: How often do you deploy code to production? How long are your builds? How long does it take to do a production deployment? How often do we have bugs in production? Staging/UAT? Dev? The answers may vary based on many factors, but odds are, you can improve dramatically in all areas.

• Continuous integration. Enabling a distributed group of developers to integrate their local code into a shared development environment as efficiently as possible is the key first step. Generally a build server (like Jenkins or Bamboo) can help to enable this. Most importantly, though, are the automated tests which run on the code before moving it to development. These can be things like PMD or SonarQube which check for best practice violations, standards, or bugs. Similarly, unit, integration, and security tests can and should be run here. The key is code is not allowed to move to development until all tests are passed. We strive for quality, production-ready code even in development.
  
  
• Peer code reviews.* This is probably the only manual step of the deployment process. Having an additional pair of (usually senior- or architect-level) eyes helps to drive team standards, code re-use, scalability, security, and efficiency. Some teams may find it hard to incorporate this critical step, but it must become part of the process.
  
  
• Automated testing. Automated tests can occur at each stage, either with each build (depending on speed), or some regular rhythm (like nightly). These tests can be regression, smoke tests, integration, or performance tests. Visibility of the results are key, as test failures must be addressed promptly. Regular testing also helps to ensure tests stay current. As the test suite grows to have a comfortable percentage of coverage, code can move faster to production with less manual testing.
  
  
• Auto-build, auto-deploy. The build servers mentioned above can automate the process of building and deploying code to each environment. Moving to production may require additional steps due to segregation of duties and change controls. As a result, I recommend making everything standard changes -- this way a change control ticket can be opened automatically by the deployment process rather than requiring manual change controls to be approved. In the lower environments, builds and deploys can be scheduled automatically or occur automatically once new code is committed.
  
  
• Same artifact in each environment. Consistency is key in ensuring quality. Using the same artifact (or Docker image if you use Docker) throughout each environment minimizes variability.
  
  
• Visibility. It is important that with all of the above it is easily accessible and visible to all stakeholders -- from the project managers to the developers. Broken builds, for example, need to be remediated fast as they prevent code from moving for the other developers.
  
  
• Forward and back. Getting to production quickly is important, but it is also imperative to have a way to revert deployments fast. Your pipeline should support this.

Configuration management
Configuring and managing environments in a streamlined and automated fashion enables speed and consistency. Configuration management tools like Puppet or Chef enable centralized management of multiple servers at once. This is key to being able to quickly spin up or down new environments as needed, patch, or ensure the same settings are applied to each without individually tending to each.

These tools can also be used to push software to desktops. This is useful for a team of developers looking to ensure everyone has the same version and configuration of tools on their machines at all times. It also helps with installing those tools as it can literally be a simple double click and go get a cup of coffee.


Containers & container orchestration
Step aside VM's, containers are the new thing. Docker containers wrap your software in a complete filesystem. It is more lightweight than a VM, and enable speed through ensuring standardization of the environment. Their small size means you can have several containers inside one VM. The key point being that containers enable true application portability, as they abstract the underlying infrastructure from the app itself.

As your environment grows with more and more containers, orchestration tools like Kubernetes become important to help manage them all from a central place.


Situational awareness
It is key for the team to know the health of the system at all times. It encompasses the following:

• Monitoring. A constant pulse on the key metrics (response times, CPU usage, server memory, etc.) allows for quick identification of potential issues and can help prevent failures. Tools like Icinga can even automate the creation of monitors when setting up servers through Puppet, for example. I recommend making as much of these metrics visible using tools like Graphite, StatsD, and Grafana.
   
• Logging. Having additional details at hand help to give more insight into the various systems. Centralizing logging outputs using the ELK stack (Elasticsearch, Logstash, Kibana), or using tools like Takipi can help to reduce the time it takes to remediate issues.
  
  
• Alerting. In addition to visual dashboards, automated alerting of key thresholds plays a key role in ensuring timely resolution of issues. A tool like Seyren can be useful here in conjunction with Graphite.

Zero downtime
Who likes staying late or working on the weekend to push new code live? No one. One of the main reasons why this occurs is because many deployments incur downtime in some fashion. With a streamlined pipeline, and a little help from Docker, staying late may become a thing of the past.

When we push new code live, we launch a second Docker container in production and point only our internal network traffic to it using Vulcan. If all tests pass, we point all traffic to the new container (using Redis to maintain sessions) and we are live without any downtime! The same can be done in lower environments as well.


Conclusion
Ultimately we want to achieve a continuous delivery state, where code changes have the potential to go live very quickly, with high assurance of quality at each step. Visibility is key to this process, as it ensures everyone is on the same page.

Lastly, the term DevOps is the combination of Development and Operations. Traditionally development teams and operations teams have competing priorities: devs want to move code to production fast; ops wants to keep the environment stable. With DevOps, the developers take more ownership throughout the process, while operations get involved earlier, more automated tools, and better visibility of the pipeline. The partnership is what drives great business results.


*Side note on peer code reviews: There may be a times where code reviews seem a bit of a burden.

First, when refactoring is required/requested by the reviewing person. Refactoring is an important and natural part of keeping the code base in good order over time. There may be times when refactoring may not be possible due to time, which I would then suggest that a user story (requirement in Agile) is added to the top of the backlog and done in a subsequent sprint (keeping in mind that there is nothing more permanent than temporary code). If you are following Scrum, ensure your teams do not consider their user stories to be "done" until all the code review comments are addressed.

Second, when there are disagreements between the reviewer and the developer. This is pretty simple to resolve, especially when the reviewer is an architect -- the developer does what the architect says. Discussions are always welcome, but tie goes to the architect.

High performing teams -- Part I: Culture

We all want our teams to be high performing. Here are just a few general traits I associate with high performing teams:

• Team members collaborate extremely well, with deep trust and openness;
• Consistently output high quality;
• Deliver at a rapid pace;
• Continuously learn and able to shift to new areas;
• Demonstrate innovation and creativity;
• Strong customer focus and knowledge;
• All team members contribute to their work and also proactively seek to improve the team;
• Has the same goals, moves in the same direction.

How do you create high performing teams? It takes a bit of effort across several different areas. I will try to use my experience to provide a framework, starting with culture.

CULTURE

Having the right culture in place is the first step toward achieving high performance. Here are some key areas I focus on:

• Relentless optimism. Doubt, fear, and negative outcomes from past experiences can hinder individuals, and bring down the entire team. With a positive outlook comes more possibilities. More possibilities bring infinite upside.
  
  Relentless optimism must start from the top. Leadership has to believe in positive change, positive results, and envision a future that is bright.
  
  This also means the senior members of the team (those whom the team look up to) need to pay attention to seemingly minor things like body language in meetings, and wise remarks that may spark doubt into others. There cannot be rolling of the eyes or anyone blurting out "Yeah, right!" Statements like, "This will never work," should be replaced with, "This could work if..."
  
• Ownership. When individuals are held accountable for their deliverables, they are more likely to ensure its quality.
  
  Leadership must identify areas where they want the team to take full ownership of their work. The team can also help to identify ways they want to be measured or demonstrate accountability. It is important for leadership to truly step away here; allow the team to be autonomous in their solutions while never micromanaging.
  
  Taking ownership should not be dreaded, rather done with pride. Leadership must help paint this light, recognizing that demonstrating a lack of ownership (perhaps as first examples for the team) could be challenging. So I remind leadership to ensure praise is given to those doing well.
  
• Failure + rapid learning is OK. A high performing team knows that it is fine to fail fast and cheap, as long as they learn from it. A team creating a new product, for example, does not want to find out it will not sell in the market after 18 months of development. They want to test their hypotheses and obtain feedback early to ensure the company does not waste time and money.
  
• Continuous improvement. The team must have a constant and proactive urge to improve processes, products, tools, and each other. I wrote about the bad words in a previous post -- those must all be removed in some fashion. I also wrote about finding the time to innovate through automation and elimination -- the essence of continuous improvement.
  
  Leadership must also make time for learning and development. Sending employees to courses can be beneficial, but how many of them are truly worth the time and money? Pick and choose wisely, and ensure results from training can be measured and demonstrated. Look for other avenues to embed training in ways which engage the team more. This takes a deep understanding of the individuals on the team, and how they operate.
  
• Need for speed. Each team member must have an eye on the speedometer. What is slowing down the process? How can we get something out the door faster? This is similar to a continuous improvement mindset, but focused strictly on speed to deliver.
  
• Safe to speak up. In order to achieve a lot of the above, the team must feel safe to speak up. Challenging status quo can be uncomfortable if leadership is not open and does not truly listen. New ideas will not emerge from team members if they are continuously stifled or ignored. Create an environment where everyone feels like their opinion is valued and they can make a difference.
  
• Have a purpose. With everyone moving toward the same goal, vision and standards of work, the team will move in lock-step even when leadership is not looking. Leadership must set out clear goals, measurements, and objectives. They must be continuously re-enforced and re-visited in various ways to demonstrate progress and positive impact.
  
• Have fun. A culture of friendliness, fun, and collaboration ensures everyone trusts each other, is willing to lend a helping hand, and enjoys coming to work. Smile, give high fives, and lighten up a little. :)

The bad words

Certain phrases are considered curse words on my teams. They represent the opposite culture which I try to instill. We strive for continuous improvement and innovation. We cannot settle or become overly comfortable, because technology moves at the speed of light. We must always be learning and thinking ahead.

Here are a few of those:

"That's the way we've always done it..."
Or also, "We've done it this way for years." If you hear this often, it generally means your team is probably far behind high-performing teams. Getting complacent or not having a constant pulse on improvement will eventually make your team irrelevant.

"Legacy system"
Why does this legacy system still exist? It is likely that managing it is painful, it contains critical security holes, and only a few employees understand it. Removing or upgrading it will reap many positive benefits. Quantify those, demonstrate the value, and kill the legacy stuff!

"Temporary code"
There is nothing more permanent than temporary code. We spend a little bit more time up front to get things right and not have to pay the price three-fold in the future (when things may break or require additional efforts due to earlier "shortcuts").

"Manual work"
We believe in automating everything. We want to be doing deep work tasks, letting the machines handle the trivial stuff.

Changing culture and creating the time to innovate does not come overnight, but demonstrating small wins along the way helps to reinforce the desired behavior.

MVP: failing fast is a good thing

How many times has your team had a scenario similar to the following example?

A senior client says a product cannot launch until one particular feature is added. Her reasoning: "No one will ever use this product unless this feature is included."

Although small in size, the feature is quite complex and would require another 2 months of development effort. The team works tirelessly those two months, and launches the product shortly thereafter.

A few weeks after go-live, the analytics demonstrate an unfortunate reality: while the product overall is getting good traction, no customers are actually using the feature suggested by the senior exec.

This scenario is exactly what Eric Ries' book, The Lean Startup, tries to address. In it he assesses how successful software development companies build and launch products.

Minimum Viable Product (MVP)
Ries describes the MVP as the product whose minimum set of features allow for learning from early adopters. Using the MVP, we are able to avoid building products no one wants, and maximize the learning per dollar spent.

The image below takes us through Ries' build-measure-learn feedback loop. An idea is formed and then built and released as an MVP. That MVP contains measurements or ways to pull data which we can learn from. From there, the product team is prepared to act on that data, and pivot or iterate.

Through small increments we can continue to test hypotheses and build a better product by minimizing the time through the feedback loop. If a particular feature or iteration is not successful, we learn early in the process through facts (analytics, metrics, user feedback, etc.).

This means failing fast is a good thing! Validated learning means we do not have to wait months before we find out no one will use a particular feature. We spend more time on things we know the users will want.

Image credit: Eric Ries, TheLeanStartup.com

Putting it into practice
I like to think of MVP as happening at each phase of the software development life cycle, in addition to the product viewed as a whole.

Take the design phase, for instance. Low fidelity mock-ups (think black-and-white, hand-sketched) are key because they speed time through "the loop." The goal is to get feedback fast -- how can you get fast feedback if you're spending time perfecting the shade of blue a button should be?

When it comes to the product owner's vision for features in release planning, how many of them build on top of an unvalidated hypothesis? What can be built and released quickly as an MVP instead?

Teams must be prepared to iterate. This means we cannot launch something and forget about it. We must release our MVP's, analyze the results, and pivot (move in a different direction vs. our hypothesis) or iterate based on our learning.

It is important to remember software has no value until it is in the hands of the user. The MVP approach gets more engaging software to the users faster by adapting incrementally.