Observability for Enterprise AI Agents: an open-source solution

12 min readMar 23, 2026

A practical deep dive into observability for GenAI agents using Langfuse and OCI

At a glance

For Enterprise AI Agents, observability cannot be limited to logs, metrics, and traces in the traditional sense. To truly understand how these systems behave, it is also necessary to observe prompts, context, tool calls, token usage, response times, and output quality.

In this article, I explain why systems based on Generative AI require a different approach to observability, and I present a concrete solution based on Langfuse that can be integrated easily with OCI Enterprise AI Agents thanks to compatibility with the OpenAI Responses API.

The goal is to show, from a practical perspective, what information can be collected, why it is useful, and how it can help improve the quality, reliability, and governability of an AI agent in production.

Introduction

Today, there is a great deal of discussion around Generative AI and AI agents. However, the term “agent” is often used to describe any workflow that relies on one or more Large Language Models to produce a final result. This simplification hides an important distinction: a true AI agent introduces autonomy, tool usage, context management, and therefore a much higher level of operational complexity.

There is also a second issue, even more relevant in enterprise contexts: attention is often focused on the “magic” of AI, while the aspects required to bring these solutions into production with adequate levels of reliability, control, and quality tend to be overlooked.

Among these aspects, observability is one of the most important. In complex systems, and especially in agents based on Generative AI, it is not a technical detail: it is an essential condition for understanding system behavior, analyzing errors, evaluating response quality, and keeping costs and latency under control.

In this article, I will start with a definition of observability and its role in complex systems. I will then examine the specific characteristics it takes on in the context of Generative AI and Enterprise AI Agents.

In the second part, I will present a concrete solution based on open-source components and easily implementable on OCI. Through practical examples, I will show what information can be collected and why it is useful for observing and improving an AI agent in production.

The proposed solution combines the interoperability capabilities offered by OCI Enterprise AI Agents, including compatibility with the OpenAI Responses API, with Langfuse, a widely used open-source platform for collecting, analyzing, and evaluating telemetry from LLM-based applications.

What observability means

Observability is a fundamental property of complex systems. In simple terms, a system is observable if we can understand its internal behavior based on the signals it produces externally and that we are able to collect and analyze.

More formally, a system is observable if two different internal states are distinguishable on the basis of the observable signals. In other words, what the system emits must allow us to understand reliably what is happening inside it.

This implies two things. First, observability matters most in complex systems, made up of many parts, dependencies, and interactions. Second, it is not enough to look at the system from the outside: it must be designed and instrumented so that it produces useful information.

To operate a complex system correctly, such as an Enterprise AI Agent, it is therefore necessary to collect telemetry that makes it possible to understand its state, behavior, anomalies, and performance.

This is where observability goes beyond traditional monitoring. Monitoring is mainly designed to answer known questions anticipated in advance, for example about resource consumption or service latency. Observability, by contrast, must also make it possible to investigate unexpected problems and questions that were not anticipated at design time, the so-called “unknown unknowns.”

Observability in traditional systems

By “traditional systems,” I mean application systems designed before the widespread adoption of Generative AI, roughly before 2023.

Historically, observability in these systems has been built around three main categories of information: logs, metrics, and traces.

Logs and metrics are generally familiar concepts. A trace, by contrast, represents the end-to-end view of a single request or interaction with the system and is composed of a series of spans, which describe the individual steps in the flow.

In a microservices architecture, for example, a trace can represent an entire application transaction, while spans describe an API call, a database query, or a call to an external service.

To be truly effective, this approach must make it possible to follow the entire path of the request across all the components involved.

In the case of agents based on Generative AI, however, this classical model is no longer sufficient on its own: in addition to logs, metrics, and traces, it also becomes necessary to observe the model’s behavior, the context, the tool calls, and the actual quality of the generated responses.

What changes with Generative AI

Traditional software systems are generally more predictable in their behavior. Systems based on Generative AI, especially when they include decision-making autonomy, are less so. The output produced by an LLM, in fact, depends not only on the user’s input, but also on a broader set of factors that must be made observable.

In an LLM-based system, an observability solution must therefore make it possible to collect at least four types of information:

the model and invocation parameters
the instructions and the actual prompt sent to the model
the context provided to the model, including any prior interaction history
the system’s execution behavior, including tool calls, intermediate outputs, latency, and token usage.

As a result, observing only the final output is not enough. To truly understand the behavior of a GenAI system, it must be possible to reconstruct, for each trace, all the elements that contributed to generating the response.

This need becomes even clearer when the model is given some degree of operational autonomy. If the system can decide which tools to invoke, in what order to execute actions, which sources to consult, or when to stop, then visibility into prompts and outputs alone is no longer sufficient. It also becomes necessary to observe the decision-making process, the sequence of tool calls, timing, and intermediate outputs.

In many cases, however, even a complete trace is still not enough. To truly evaluate the quality of the system’s behavior and the output it produces, it is often necessary to associate each trace with specific scores. These scores may be computed using traditional logic or generated in turn by an LLM, according to an approach that is now widely known as “LLM-as-a-judge.”

A concrete example is evaluating the correctness of a response produced by a RAG system. A response may in fact be only partially correct, or formally plausible but not well supported by the retrieved sources. In these cases, it must be possible to distinguish at least two aspects: the root cause of the problem, for example retrieval, reranking, or generation, and the grounding of the response, meaning its actual grounding in the retrieved documents, in order to determine whether unsupported statements are present and therefore whether there may be potential hallucinations.

All these characteristics clearly show that, in the world of Generative AI, adopting a generic observability platform is not enough. What is needed is a tool capable of natively capturing this kind of information while at the same time minimizing the effort required to instrument the system being observed.

This, ultimately, is the real difference: in GenAI systems, we must observe not only infrastructure and performance, but also context, decisions, quality, and the reliability of the model’s behavior.

From theory to practice

At this point, it should be clear why observability, in the case of Enterprise AI Agents, cannot be limited to traditional tools and approaches. It is not enough to collect logs, metrics, and traces in the classical sense of the term: it must also be possible to observe prompts, context, tool calls, response times, token consumption, output quality, and the possible causes of errors or poor-quality responses.

The question therefore becomes very concrete: how can all this be implemented in a simple, effective way that is also compatible with an enterprise context?

In the second part of this article, I will try to answer that question by presenting an open-source solution that can be adopted easily on OCI.

A concrete solution: Langfuse

To build an observability solution suitable for systems based on LLMs and AI agents, one of the most interesting open-source tools available today is Langfuse. It is a platform designed for tracing, observability, and evaluation (scoring) of LLM-based applications.

Its relevance in the context of this article is very concrete: Langfuse makes it possible to collect and analyze many of the types of information that, as discussed in the first part, are essential for truly observing a GenAI system. These include prompts, outputs, traces, metadata, response times, token usage, tool calls, and possible evaluation scores.

More than just a logging platform, Langfuse therefore provides a structured observation point into the actual behavior of a GenAI application.

Why OCI simplifies integration

One of the most interesting aspects in this scenario is that OCI Enterprise AI Agents provides an API that is fully compatible with the OpenAI Responses API. This compatibility is not just a technical detail: it makes it much easier to integrate tools, libraries, and development patterns that are already widely used across the OpenAI-compatible ecosystem.

In the specific case of Langfuse, this translates into a significant reduction in the effort required to collect telemetry from model calls and to build the corresponding observability layer. In practice, a substantial part of the integration work becomes much simpler, making it possible to focus more on analysis and less on implementation details.

This is one of the reasons why the combined use of OCI and Langfuse is particularly interesting in an enterprise context.

A quick OCI deployment for a PoC

For a proof of concept, Langfuse can be installed quite easily within OCI as well, for example on a Linux virtual machine, using the official deployment based on Docker Compose.

This approach does not necessarily represent the final architecture for a production-grade enterprise environment, but it is more than adequate for quickly validating the solution, collecting real telemetry, and evaluating the kind of analysis that becomes possible once the application has been integrated.

In my case, this approach proved sufficient to make a working platform available quickly, accessible through a browser and ready to receive traces from Python applications based on LLMs.

A simple code integration

Once a Langfuse instance is available, the next step is to integrate the application so that telemetry from model calls is sent automatically.

Thanks to compatibility with the OpenAI Responses API, this step can be quite straightforward for Python applications that are already built around an OpenAI-compatible invocation pattern. The application continues to call the model in a familiar way, while Langfuse makes it possible to trace the most relevant information for later analysis.

Integration example:

As the example code shows, the required changes are minimal:

A few environment variables are set.
The client provided by the Langfuse library is used, already instrumented to collect and send traces.
The conversation_id is propagated as the session_id, so that requests belonging to the same conversation can be correlated within the same session.
If needed, the flush can be forced to send the trace immediately. Otherwise, the flush is performed automatically after a few seconds.

The complete demo is available at this GitHub repository.

What data is collected

Once the integration is complete, Langfuse makes it possible to automatically collect a significant amount of information that is useful for observing an LLM-based system. In the case of AI agents, this is especially important because it provides visibility not only into model inputs and outputs, but also into the execution flow, including intermediate spans, tool calls, timing, and other signals that help reconstruct how the result was produced.

Among the most relevant information that can be collected, in the context of this article, are:

the model being used and its version
the prompt sent to the model
the generated output
the metadata associated with the call
the structure of the trace and any spans
latency metrics, including user experience metrics such as time-to-first-token
the number of input and output tokens, useful for cost analysis
any tool calls and their details
scores and evaluations that can be associated with the trace

Not all application-relevant information is necessarily collected automatically. Some, such as domain-specific scores, custom indicators, or business metadata, can and, in some cases, must be added explicitly by the application. It is precisely this combination of automatic collection and targeted enrichment that makes the tool particularly effective.

Sessions

Langfuse makes it possible to correlate multiple traces belonging to the same user session through the concept of a session and the use of a session_id. In the code example shown, the conversation is identified through the standard conversation_id of the OpenAI Responses API. For this reason, the session_id is set equal to the conversation_id, so that requests belonging to the same conversation can be correlated with one another.

Sessions can then be explored directly through the UI.

A session contains multiple traces:

Inside a trace

Once traces have been collected, the value of the solution becomes most evident during the analysis phase. The Langfuse UI makes it possible to explore each interaction in detail, making elements immediately visible that would otherwise be scattered across application logs, isolated metadata, and code.

Within a single trace, it is possible to observe, for example, the actual prompt sent, the model used, the parameters, execution times, the number of tokens consumed, the sequence of any calls made, and the outputs produced at different stages.

This information becomes particularly valuable when analyzing unexpected behavior, latency issues, rising costs, or cases where response quality is not satisfactory.

Limits and aspects to consider

Naturally, adopting a platform such as Langfuse does not automatically solve all the observability and evaluation challenges of a GenAI system. Collecting traces rich in context is a necessary condition, but it is not always sufficient.

The most useful scores often depend on the application domain and must be designed explicitly. In real enterprise environments, it is also necessary to consider carefully aspects such as security, sensitive data, telemetry retention, access control, and deployment architecture.

That said, the combination of an open-source platform such as Langfuse and the full API compatibility offered by OCI provides a very effective starting point for building a concrete observability solution for Enterprise AI Agents.

Conclusions

In the case of Enterprise AI Agents, observability is neither a technical detail nor an optional addition. It is a necessary condition for understanding how the system behaves, analyzing errors, evaluating response quality, and keeping response times and operating costs under control.

As we have seen, in systems based on Generative AI it is not enough to observe infrastructure, application logs, and traditional metrics. It also becomes necessary to collect information that is much closer to the agent’s actual behavior: prompts, context, tool calls, token consumption, user-perceived response times, and any evaluation scores.

In this scenario, Langfuse represents a particularly interesting open-source solution, because it makes it possible to collect and analyze an important part of this telemetry in a structured way. In addition, the availability within OCI Enterprise AI Agents of an API compatible with the OpenAI Responses API makes integration very straightforward and significantly reduces the effort required.

Naturally, an observability platform does not solve all problems on its own. In an enterprise context, aspects such as security, governance, sensitive data management, and the design of the most useful domain-specific scores remain essential.

If we want Enterprise AI Agents to move beyond the demonstration phase and become reliable components of real systems, we must stop treating observability as a secondary concern. The complexity of these systems cannot be eliminated. But it can, and must, be made observable.

In the next article

In this article, I have focused mainly on the observability problem and on a concrete solution for collecting the telemetry needed to understand the behavior of an Enterprise AI Agent. A natural next step is to show how traces can be enriched with specific scores that make it possible to evaluate system quality in a more structured way.

In particular, in a follow-up article I will show how to integrate scores produced through an “LLM-as-a-judge” approach, so that each trace can include not only the technical execution data, but also evaluations that are closer to the actual quality of the generated response.

This will make it possible to take a further step forward: not only to observe system behavior, but also to start measuring it more specifically in relation to response quality.