Three Pillars of Observability – Logs

Introduction

Observability evaluates what’s happening in your software from the outside. The term describes one cohesive capability. The goal of observability is to help you see the condition of your entire system.

Observability needs information about metrics, traces, and logs – the three pillars. When you combine these three “pillars,” a remarkable ability to understand the whole state of your system also emerges. This information might go unnoticed within the pillars on their own. Some observability solutions will put all this information together. They do that as different capabilities, and it’s up to the observer to determine the differences. Observability isn’t just about monitoring each of these pillars at a time; it’s also the ability to see the whole picture and to see how these pieces combine to fit in a puzzle and show you the actual state of your system.

The Three Pillars of Observability

As mentioned earlier, there are three pillars of observability: Logs, Metrics, and Traces.

Logs are the archival records of your system functions and errors. They are always time-stamped and come in either binary or plain text and a structured format that combines text and metadata. Logs allow you to look through and see what went wrong and where within a system.

Metrics can be a wide range of values monitored over some time. Metrics are often vital performance indicators such as CPU capacity, memory usage, latency, or anything else that provides insights into the health and performance of your system. The changes in these metrics allow teams to understand the system’s end performance better. Metrics offer modern businesses a measurable means to improve the user experience.

Traces are a method to follow a user’s journey through your application. Trace documents the user’s interaction and requests within the system, starting from the user interface to the backend systems and then back to the user once their request is processed. 

This is a three-part blog series on these 3 pillars of observability.  In this first part, we will dive into logs.

The First Pillar – Logs

In this part of the blog, we will go through the first pillar of Observability – Logs. 

Logs consist of the system’s structured and unstructured data when specific programs run. Overall, you can think of a log as a database of events within an application. Logs help solve unpredictable and irregular behaviors of the components in a system.

They are relatively easy to generate. Almost all application frameworks, libraries, and languages support logging. In a distributed system, every component generates logs of actions and events at any point.

Log files entail complete system details, like fault and the specific time when the fault occurred. By examining the logs,  you can troubleshoot your program and identify where and why the error occurred. Logs are also helpful for troubleshooting security incidents in load balancers, caches, and databases.

Logs play a crucial role in understanding your system’s performance and health. Good logging practice is essential to power a good observability platform across your system design. Monitoring involves the collection and analysis of logs and system metrics. Log analysis is the process of deriving information from these logs. To conduct a proper log analysis, you first need to generate the logs, collect them, and store them. Two things that developers need to get better at logging are: what and how to log.

But one problem with logging is the sheer amount of logged data and the inability to search through it all efficiently. Storing and analyzing logs is expensive, so it’s essential to log only the necessary information to help you identify issues and manage them. It also helps to categorize log messages into priority buckets called logging levels. It’s vital to divide logs into various logging levels, such as Error, Warn, Info, Debug, and Trace. Logging helps us understand the system better and help set up necessary monitoring alerts. 

Insights from Logs

You need to know what happened in the software to troubleshoot system or software level issues. Logs give information about what happened before, during, and after an error occurred.

A trained eye monitoring log can tell what went wrong during a specific time segment within a particular piece of software.

Logs offer analysis at the granular level of the three pillars. You can use logs to discover the primary causes for your system’s errors and find why they occurred. There are many tools available for logs management like

• SolarWinds Log Analyzer
• Sematext Logs
• Datadog
• Splunk
• Fluentd

You can then monitor logs using Grafana or Kibana or any other visualization tool.

The Logs app in Kibana helps you to search, filter, and follow all your logs present in Elasticsearch. Also, Log panels in Grafana are very useful when you want to see the correlations between visualized data and logs at a given time. You can also filter your logs for a specific term, label, or time period.

Limitations of Logs

Logs show what is happening in a specific program. For companies running microservices, the issue may not lie within a given service but how different connected functions. Logs alone may show the problem but do not show how often the problem has occurred. Saving logs that go back a long time can increase costs due to the amount of storage required to keep all the information.

Similarly, coming up with new containers or instances to handle client activity means increasing the logging and storage cost. 

To solve this issue, you need to again look to another of the three pillars of observability—the solution for this: metrics. We will cover metrics in the second part of our observability series. Stay tuned to learn more about observability.

Observability 101 – How is it Different from Monitoring

Monitoring IT infrastructure was, in the past, a fairly complicated thing, because it required constant vigilance: software continuously scanned a network, looking for outages, inefficiencies, and other potential problems, and then logged them. Each of these logs would then have to be checked by a qualified SOC team, which would then identify any issues. This led to several common problems, such as alert fatigue and false flags – both of which we’ll discuss more later – and burnout was prevalent. In fact, these three issues (fatigue, flags, and burnout) have only increased as our interconnectivity has increased. Much like the pitfalls that have befallen the airline industry (such as increased security risks and tougher identification and authorization measures), our increasing connectivity is also presenting increased security risks, risks that require more stringent identification and authorization measures, adding to the workload of SOC teams.

What does monitoring do? It lets us know if there are latency issues; it lets us know if we’ve had a jump in TCP connections. And while these are important notifications, they are no longer enough. Secure systems do not remain secure unless they are also maintained. Security teams need a system that can monitor all of these interconnected components. This is where observability comes in.

What is monitoring?

Observability is the capacity to deduce a system’s internal states. Monitoring is the actions involved in observability: perceiving the quality of system performance over a time duration. The tools and processes which support monitoring can deduce the performance, health, and other relevant criteria of a system’s internal states. Monitoring specifically refers to the process of analyzing infrastructure log metrics data.

A system’s observability lets you know how well the infrastructure log metrics can extract the performance criteria connected with critical components. Monitoring helps to analyze the infra log metrics to take actions and deliver insights.

What is Observability?

Observability is the capacity to deduce the internal states of a system based on the system’s external outputs. In control theory, observability is a mathematical dual to controllability, which is the ability to control the internal states of a system by influencing external inputs. 

Infrastructure components that are distributed operate in multiple conceptual layers of software and virtualization. Therefore it is not feasible and challenging to analyze and compute system controllability.

Observability has three basic pillars:  metrics, logs, and tracing. As we noted a moment ago, observability employs all three of these to create a more holistic, end-to-end look at an entire system, using multiple-point tools to accomplish this. 

Comparing observability and monitoring

People are always curious about observability and its difference from monitoring. Let’s take a large, complex data center infrastructure system that is monitored using log analysis, monitoring, and ITSM tools. Monitoring multiple data points continuously will create a large number of unnecessary alerts, data, and red flags. Unless the correct metrics are evaluated and the redundant noise is carefully filtered monitoring solutions, the infrastructure may have low observability characteristics.

A single server machine can be easily monitored using metrics and parameters like energy consumption, temperature,  transfer rates, and speed. The health of internal system components is highly correlated with these parameters. Therefore, the system has high observability. Considering some basic monitoring criteria, such as energy and temperature measurement, the performance, life expectancy, and risk of potential performance incidents can be evaluated.

Observability in DevOps

The concept of observability is very important in DevOps methodologies. In earlier frameworks like waterfall and agile, developers created new features and product lines while separate teams worked on testing and operations for software dependability. This compartmentalized approach meant that operations and monitoring activities were outside the development’s scope. Projects were aimed for success and not for failure i.e debugging of the code was rarely a primary consideration. There was no proper understanding of infrastructure dependencies and application semantics by the developers. Apps and services were built with low dependability. 

Monitoring ultimately failed to give sufficient information of the distributed infrastructure system about the familiar unknowns, let alone the unfamiliar unknown.

The popularity of DevOps has transformed SDLC. Monitoring is no longer limited to just collecting and processing log data, metrics, and event traces but is now used to make the system more transparent I.e observable. 

The scope of observability encapsulates the development segment which is also aided by people, processes, and technologies operating across the pipeline.

Conclusion

Collaboration of cross-functional teams such as Devs, ITOps, and QA personnel is very important when designing a dependable system. Communication and feedback between developers and operations teams are necessary to achieve observability targets of the system that will help QA yield correct and insightful monitoring during the testing phase. In turn, DevOps teams can test systems and solutions for true real-world performance. Constant iteration based on feedback can further enhance IT’s ability to identify potential issues in the systems before the impact reaches end-users.

Observability has a strong human component involved, similar to DevOps. It’s not limited to technologies but also covers the approach, organizational culture, and priorities in reaching appropriate observability targets, and hence, the value of monitoring initiatives.

Episode 9 – Top 5 Challenges for Mobile Service Providers Today and How to Tackle Them with DevOps and Analytics

In episode 9 of Infralytics, Shankar spoke with John Griffiths. John is the Senior Product Manager for Openmind Networks, a leading provider of messaging infrastructure for mobile service operators and intercarriers. The subject of discussion was the “Top 5 challenges for Mobile Service Providers today and how to tackle them with devops and analytics.” 

Mobile Service Providers: The Interview

Telcos are planning for the 5G rollout and there are huge expectations among consumers and businesses regarding how 5G could transform and improve connectivity. Meeting such expectations is never easy. What are the top challenges faced by mobile service providers today?

Due to increased competition in the mobile sector in general and due to government regulation, operators are having to deal with decreasing revenues and shrinking margins for the same services. They have to do this in the face of usual challenges of the need to upgrade the network and invest with their network.

5G is the latest technology that requires serious investment.  Also mobile operators are not just competing among themselves anymore. New competitors are entering the space and offering over the top services. The risk in this type of climate of mobile operators being marginalized, with the worst case scenario being if they just become providers of data bandwidth over which messaging and streaming services could be carried. It’s a huge challenge for operators to stay relevant.

How are mobile service providers addressing these challenges around competition?

The more cutting edge operators have realized that to survive in this environment with the disruptive innovations that keep happening, they have to create a slim and efficient network. Mobile operators can gain efficiencies by reducing and optimizing their hardware footprint. This is done through Network Function Virtualization (NFV) which is the mobile sectors preferred system for maximizing efficiency. NFV architecture enables operators to plan for tomorrow’s systems and applications through hardware that can run multiple applications simultaneously. 

Another way mobile service providers are addressing these challenges is by becoming more IT centric. Network technologies are being reworked and moved into an IT centric and software driven environment. IT and Internet companies are much farther ahead than mobile operators as far as this process goes, and they have employed devops, continuous integrations, and continuous delivery enabled through automation and optimization of services. The most cutting edge mobile operators are beginning to learn from these IT and internet companies, and they are adopting these new techniques. DevOps involves releasing small incremental improvements weekly or monthly, so the cutting edge operators have done away with large upgrade projects.

DevOps also enables automated testing. Manual testing is replaced by automated testing. In addition to these efficiencies, DevOps is also adding value.

Have you seen adoption of container technologies by Mobile Service providers or is it too early?

Leading mobile operators are beginning to adopt container technologies.  For example, Openmind’s new platform is based on containers and it’s docker based. The advantage of containers is that it enables us to deploy in any environment. It’s also very devops friendly. So containerization makes everything smooth and easy from a mobile operator perspective.

There used to be a lot of testing for a major release but with automated testing you can immediately run testing whenever there is any small change.

Are Telcos implementing any monitoring tools since changes are so frequent?

At Openmind we provide all of the software updates, testing services, and monitoring to the operators. 

In terms of going from 4G LTE to 5G, are there more endpoints that they need to monitor?

Yes there are always a huge number of updates and software rollouts on the network with any new technology. From a messaging perspective, it’s yet to be determined whether the architecture will change in 5G.

I read about the RCS that is being adopted by all of the hardware vendors and Telcos. Any thoughts on that?

RCS has been promised as a great hope for a number of years, and Openmind is the first messaging vendor to have a GSMA accredited RCS product but despite this the industry pickup hasn’t been as great as what was hoped for many years. Even after Google came on board, the adaptation hasn’t been as great as many hoped it would be.

How is analytics being used to address the various challenges you mentioned?

With the new generation of messaging products, Big Data and Analytics are part of the products themselves. These services can be customized as customer based services and AI services. Mobile operators have valuable data that can be used by enterprises to communicate more effectively with their customers. This new generation of products is incorporating this customer data into the products themselves.

Big Data is also aiding the messaging space in artificial intelligence. The latest developments in machine learning and neural networks are now being applied to message categorization based on the content in the message. Mobile operators can then use this classification and categorization to make intelligent routing decisions about where to route different types of messages. This enables them to offer different levels of services and charge different rates for the different levels. 

As far as security is concerned, if you can categorize messages into spam and/or fraudulent you can block these types of messages. 

Is the concept of sending targeted messages to different demographics in the form of ads similar to what Google and Facebook are already doing with their ad businesses something that we are heading towards in the messaging space?

Yes. The categorization is really about targeting the messages and campaigns. It involves offering services to enterprises by making use of the knowledge you have about the consumers themselves. For example if you know that certain subscribers are roaming in a certain location because you can detect that, and they are in a shopping mall, it’s a possibility to send a campaign to this targeted group rather than spamming the whole group of subscribers. So using real-time data and customer profiling to target messages and campaigns. 

So at Openmind are you using your own stack or what do you use to offer these analytics capabilities?

Some operators have their own analytics systems, but for customers that want a messaging system plus analytics capabilities as well we base our products on what  Elasticsearch and Kibana are offering and we have built on top of that. One of the aspects that we have put on top of the standard Kibana is the Skedler Reporting Tool for sending scheduled reports to people who don’t need to access the analytics systems themselves…that just need a regular report being sent to them. 

Conclusion

What are your thoughts on the revelation that mobile service providers can send targeted messages based on real-time data collected and customer profiling similar to how companies like Facebook and Google use data to target ads? 

Openmind uses the Skedler reporting tool to send scheduled reports to their Telco customers. Are you interested in trying Skedler reports or Skedler alerts for your business? Start your free trial today!