Redefining the Boundaries of People, Process, and Platforms

Day two of Dynatrace Perform began with a great discussion between Kelsey Hightower, Distinguished Developer Advocate at Google Cloud Platform and Andi Grabner, DevOps Evangelist at Dynatrace. The theme of their discussion was redefining the boundaries of people, processes and platforms. 

Kelsey began the discussion by explaining that his career took off when he began focusing on the fundaments. When he sees something new, he takes a step back and thinks about the first time humans were involved in something like this.

Humans have been doing observability for thousands of years out of the necessity for survival. This was the first SRE position ever created. At night you need to be on the lookout for animals that may be trying to eat us.

We’ve been flying blind in IT for years. We now have the technology to put eyes and ears throughout the stack so people can respond to events. Data and technology help me understand the fundamentals of my technology and explain the things that are important. The convergence of observability and security ensures everyone stays safe

Kubernetes’ Growth and Boundaries

Andi: Looking at the data, we see a steady increase in K8s deployments as people move. to the cloud. But we see an even greater increase in auxiliary workloads.  No longer are we just deploying microservices for business apps but also for databases, DevOps tools, observability, and security. We have broken down the boundaries to put K8s into ecosystems.  We need to make it easy to deploy. Boundaries have been torn down. What are the technical and organizational boundaries we need to put in place to leverage K8s? 

Kelsey: The best way to remove boundaries is to create new ones. Know what you are trying to accomplish. People buy too much software and too many agents. Everyone rushes to buy everything rather than identifying what they’re trying to accomplish. They end up needing abstractions for everything they've bought.  

K8s provides infrastructure types that allow people to remove the barriers to entry necessary to give the system your desired state.

Arbitrary Deadlines Lead To Reverse Engineering

For more than 30 years we haven’t figured out how to put an app on the server. We gave people a thousand servers and lost our minds. We start running in circles. We created DevOps to work out the problems. 

Devs are given arbitrary deadlines to accomplish something in two months. In order to meet the deadlines, they go to StackOverflow to copy/paste some code. If it works, they have a head start. They have no idea how the code was written but it works and that’s what they care most about because you have to meet a made-up deadline. We take what works, without understanding why it works, and go on from there because we have a deadline to meet. 

If the QA team cares, they will test scenarios and tell you where your code doesn’t work. Given the time pressure, you only fix the things that are highlighted because you do not have the time to exercise every element of the code. Then we give it to the operations team. They have no idea why it works either. 

We spend our career reverse engineering these applications. We plug in all these tools to give us insight into how they’re supposed to work. So, effectively we are reverse engineering things we don’t understand. It's an impossible task.

I think at some point as we get better frameworks and encode best practices into the application stack. This is not a discussion between microservices or monoliths. It is a discussion around having the business logic you need to write it. The whole goal is to reduce the amount a human needs to give to the system to do what you want to do. Everything in the middle is friction.

Going forward we need better frameworks to encode best practices. Take the things you learn and put them into the platform so you don’t have to learn them again. The developer you hire in two years should be able to use the platform in its current state. 

The biggest challenge is taking the things we learn and putting them back into the platform so we don’t have to learn them again. Hopefully, 30 years from now, we’re not sitting here talking about how to put a binary on a platform. It’s been done and codified.

The Evolution of Roles

Roles are evolving. In 2012 Oracle DBAs were challenged to move to PostgreSQL. Oracle DBAs, there is no database better than Oracle. They were refusing to learn anything new. But the fundamentals of PostgreSQL were roughly the same. Some of the features were different. 

Today, I see the same roles evolving. Some people want to remain in their narrow role. Some people only see they are doing security work and they have their security hats on. Some people believe they are performance engineers and that’s the only thing that actually matters. 

If you ask them, how do you know if the system is secure, how do you know if the system is performant? If you’re an SRE, the question is how do you know the system is reliable? There is one common thread among all of these disciplines. It is the data you use to tell you these things are true.

What is the difference between a trace, a metric, and a log? We have entire conferences on all three islands of data. But if you think about role evolution, most of these disciplines will evolve to become data scientists. You understand physical analysis. You understand data has shaped. You will take that data and turn it into information that becomes knowledge. 

These platforms show that if you put all the data in one place you can analyze where the hidden problems may be. Your role will evolve. If you believe you’re only an SRE, the same data will tell you about security incidents, it will tell you about performance requirements and if you are meeting them or not. 

Eventually, these roles converge. We stop the islands. We realize there is a lot of value in what a data scientist does. How statistics are involved in your everyday role. And then you start removing HR boundaries. Those are good for LinkedIn but they're not good for production. 

Humans use data all of the time. We see a red light, we stop the car. Data gives us a feedback loop that you can use to make any decisions along the spectrum of performance and security. 

Cloud Arguments

The argument against clouds is uncertainty. When it’s your system you feel like you are in control. When you go to the cloud and have an outage, you cannot do anything about it. When there’s a black box it makes people feel uncomfortable.

However, let's look at it another way. Imagine the airline gifts you a 747. When you take ownership, you realize you don’t have a pilot’s license. You don't know how to get the fuel. You have no idea how to operationalize it. You do not succeed in the grand scheme of things by yourself, 

It takes a team of people. You don’t succeed by yourself. You need a partner to help you achieve observability that will insure a better UX and CX. 

Do you keep managing things on a spreadsheet or move to a platform that enables a team of people to manage together?

As AI Continues To Grow Exponentially, Does the People Aspect Change?

I hope it does. We will find a new appreciation for talking to another person and seeing their emotions, their body language, and learning about their experience. I believe AI will raise the bar for humanity. 

What Are You Most Excited About?

The things we’re able to do as individuals have never been more powerful. It is only going to increase with the tools we have access to. We will continue to marry smart people with new tools to create new breakthroughs.