Observability in 2022 – more open, more insight, more collaboration needed

Observability has been growing in importance for developers over the past few years. The term today describes the combination of application log data, metrics and tracing to tell you what is going on in your applications and your infrastructure. The term itself comes from an approach rooted in engineering and control theory, where you validate how your system is running from the output that you gather over time. This output helps you deduce any problems, but also where there are opportunities to improve your system over time.

Following this growth in popularity, more and more developers have got involved in how to use observability. This has led to a thriving community of developers that have created and supported open source projects around that data. These projects initially solved specific problems, but just like observability in theory, these projects have been joined up to deliver more in-depth results and more insights to those that use them.

The OpenTelemetry specification for tracing moved to version 1.0 in 2021, providing a base for multiple projects around application performance. This was used by multiple providers to offer their own distributions of OpenTelemetry, where they applied this framework to their services or based products on the open source version. This makes it easier for developers to adopt, and also gives them options if they ever want to change their approach.

SEE ALSO: No-code is SaaS 2.0

Looking ahead around observability

Now we have this in place, what will happen next around observability? The first prediction I would make is that use of OpenTelemetry will continue to expand, with more IT professionals taking advantage of it to help them in their work. Alongside developers, others will use OpenTelemetry too.

For example, IT Security has a lot of common ground with developers around observability data. The majority of enterprises will have security operations centres (SOCs) in place that have to look for potential attacks and stop them. This is a round-the-clock effort based on data coming in, being analysed and then potential threats earmarked for human follow-up. Using this information can help the SOC team understand what is taking place, without requiring them to run their own data gathering processes or store the same sets as a duplicate.

Observability can also be used for business decisions. With more companies adopting digital processes, companies can use information from their applications to see how they are performing. In the past, this ‘digital exhaust’ of data created over time would have been treated as something just for the tech teams to look at, or to be thrown out as soon as possible. However, that mindset has changed.

Observability can help developers see faults or things to fix, but it can show how those business decisions led to an impact on performance, and vice versa. For example, your company might decide to launch a new service as part of your application. You would obviously want observability data on that service and how it performs from a technical perspective, but that same information can be used to show how that service was used by customers and what their responses were to evaluate its success from a wider business perspective.

In this case, you can make inferences about customer behaviour over time based on the data that your applications and infrastructure provide back to you. This requires context and trend information to help. In 2022, providing this information back to the business will be a requirement for DevOps and developer teams alongside their technical use cases for observability.

This will mean that teams will have to develop more skills around data and storytelling. For example, developers may be happy to use logs, traces and metrics to look for things when an incident takes place, but this is not the same audience. Previously, this would involve looking at areas like service level objectives and service level indicators, but these are still very technical approaches to these problems. Simplifying this further will help the business to perform better, but it will take work to design and provide this data in the appropriate way.

Observability and collaboration

One response for this in 2022 will be that more enterprises will put together centres of excellence around observability. The role for the centre of excellence is to have a neutral, unbiased team that constantly works on collaboration across functions, and helps each team improve their results around the data that is coming in.

In order to achieve this, companies will need senior leadership to force the issue. The reason for this is that the software development and security teams involved, have to work quickly and deal with high volumes of data. As the amount of data they have to work with increases due to digital transformation, these teams can end up concentrating on their own work. While they focus on what they can deliver, they can ignore the bigger picture taking place around data. Instead, a senior leader will have to implement an approach that makes this collaboration take place.

Putting a centre of excellence in place can help for other reasons too. Having experts coach other teams on what they can use data for can make processes more efficient. Similarly, these teams will look at the platforms that their companies use. This will lead to consolidation around how to manage the physical set of data. Each team should use the same approach and platform, as this makes it easier to support multiple use cases across the organisation.

SEE ALSO: How Data Gateways Simplify Developers’ Lives

Observability and insight into AI-based decisions

More organisations are using artificial intelligence (AI) and machine learning (ML) capabilities across their infrastructure. According to Gartner, the worldwide market for AI software is forecast to be more than $62.5 billion in 2022, an increase of 21.3% from the previous year. That spend is based on AI helping companies work faster, be more efficient and automate their processes. AI and observability are beginning to cross over, and 2022 will see a big change in how developers analyse and monitor AI’s use in production systems.

The big question that these teams have to answer is how AI systems work, and how they get to the results that they put together. Alongside looking at the results and whether they are accurate and fair. Looking at how data is used, and whether those results are ethical, will be a high profile requirement in 2022.

For developers, this means extending existing observability approaches to cover how AI and ML capabilities work and how information gets used. This insight will explain how AI results are delivered, and highlight any potential problems around how data models function. This ethical side to AI will be critical as companies invest more around the technology, so explaining how results get created will be just as important as the results themselves. The open source community will work on how observability into AI models can be delivered.

Using observability, the aim should be to go back to any transaction and explain how the result came about. Just like observability for applications can tell us how the system itself works, observability for AI will use the results of transactions, and how they are processed, in order to gauge the effectiveness of each model. It should also show whether there are any issues to consider around personal data.

Observability has gone from being a niche topic for software developers into a source of consolidated, continuously updated intelligence for teams across the business. This continuous approach will grow into other areas. By getting the right processes and the right mindset in place, companies can take advantage of their digital exhaust to grow, to make better decisions, and to safeguard their customers better.

The post Observability in 2022 – more open, more insight, more collaboration needed appeared first on JAXenter.