Logging on dotnetuniversity

GenAI Observability in .NET: Logging, Tracing, and Metrics with Langfuse

Sun, 05 Apr 2026 00:00:00 +0000

When you build a .NET application that calls an LLM, the hard part is rarely the first API call. The hard part starts after you deploy. How many tokens did each user consume? Which prompts are failing? How long do chat completions take? Why did a tool-calling chain produce the wrong result for one user but not another?

Traditional observability tools give you generic HTTP spans and CPU metrics, but they don’t understand LLM-specific signals like token usage, model name, temperature, prompt text, tool calls, or finish reason. That’s where Langfuse comes in.

Langfuse is an open-source LLM engineering platform that provides observability, evaluation, prompt management, and analytics. It supports OpenTelemetry natively, which means you can instrument your .NET GenAI application once and get structured LLM traces, metrics, and logs in one dashboard.

In this post, I’ll show how to instrument a .NET Web API that uses Semantic Kernel for chat completions via DeepSeek (OpenAI-compatible API), with Langfuse as the observability backend. The sample uses:

Microsoft.SemanticKernel — orchestration layer for LLM calls and tool invocation
DeepSeek — chat model via OpenAI-compatible endpoint
LangfuseDotnet — OpenTelemetry exporter for Langfuse
OpenTelemetry — traces, metrics, and structured logs
Custom meters — LLM-specific metrics (token count, latency, request rate)

What Langfuse Gives Us for GenAI Workloads

Standard observability backends treat all spans equally. A chat completion request looks like any other POST /api/v1/chat span. You lose the GenAI-specific context that matters for debugging and optimization.

Langfuse bridges that gap. It follows OpenTelemetry GenAI semantic conventions, so the SDK knows how to structure LLM telemetry:

Traces — full trace of a chat request with nested spans for tool calls, retrievals, and generations
Generations — LLM-specific observations with model name, provider, temperature, max tokens, input/output messages, token counts, and finish reason
Metrics — request counts, latency histograms, per-model token usage
Logs — structured application logs correlated with traces via trace IDs

All of this works through standard OpenTelemetry exporters. The Langfuse exporter is just another OTLP destination that understands GenAI conventions.

Getting Started: Package Setup

We need these packages in the project:

<!-- Semantic Kernel -->
<PackageReference Include="Microsoft.SemanticKernel" Version="1.35.0" />
<PackageReference Include="Microsoft.SemanticKernel.Connectors.OpenAI" Version="1.35.0" />

<!-- Langfuse + OpenTelemetry -->
<PackageReference Include="zborek.LangfuseDotnet" Version="0.10.0" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.15.3" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.15.2" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.15.1" />
<PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.15.1" />

The Langfuse SDK (zborek.Langfuse) bundles the OpenTelemetry exporter, GenAI semantic convention helpers, and the IOtelLangfuseTrace service for scoped tracing.

Semantic Kernel preview warnings are suppressed so the build stays clean:

`1`	`<NoWarn>SKEXP0010,SKEXP0050,SKEXP0001,SKEXP0055</NoWarn>`

Running Langfuse Locally with Docker Compose

Langfuse is an open-source platform, so you can run it locally instead of relying on a managed cloud instance. The sample repository includes a docker-compose.yml that starts everything needed for local development:

`1`	`docker compose up -d`

The compose file starts Postgres, ClickHouse, MinIO, Redis, the Langfuse worker, and the Langfuse web UI. The web UI is exposed on http://localhost:3001.

It also seeds a default organization, project, and user account:

Email: admin@example.com
Password: LangfuseDemo2026!

The sample’s appsettings.json already points to the seeded local project, so no manual key configuration is needed. The compose file sets both LANGFUSE_INIT_ORG_ID and LANGFUSE_INIT_PROJECT_ID; without these IDs Langfuse skips the seeding step.

{
    "Langfuse": {
        "Url": "http://localhost:3001",
        "PublicKey": "pk-lf-local-public-key",
        "SecretKey": "sk-lf-local-secret-key",
        "OnlyGenAiActivities": false
    }
}

The application checks for the Langfuse keys at startup. If they are empty, the exporter is not registered and the API still runs normally — it just does not send traces to Langfuse. This is useful for local development without the compose stack.

No Aspire orchestration is required. The sample is a plain ASP.NET Core project, and the compose stack is self-contained.

Configuring Langfuse with OpenTelemetry

The wiring happens in Program.cs. We register OpenTelemetry with the Langfuse exporter, and then register the scoped trace service:

using zborek.Langfuse.OpenTelemetry;

var builder = WebApplication.CreateBuilder(args);

var langfuseSection = builder.Configuration.GetSection("Langfuse");
var hasLangfuseKeys = !string.IsNullOrEmpty(langfuseSection["PublicKey"])
    && !string.IsNullOrEmpty(langfuseSection["SecretKey"]);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "LangfuseGenAI",
            serviceVersion: "1.0.0",
            serviceInstanceId: Environment.MachineName
        )
    )
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddSource("LangfuseGenAI.WeatherPlugin");

        if (hasLangfuseKeys)
            tracing.AddLangfuseExporter(langfuseSection);
    })
    .WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation()
            .AddMeter("LangfuseGenAI.Meters");
    });

if (hasLangfuseKeys)
    builder.Services.AddLangfuseTracing();

The AddLangfuseExporter call reads these settings from appsettings.json:

{
    "Langfuse": {
        "Url": "http://localhost:3001",
        "PublicKey": "pk-...",
        "SecretKey": "sk-...",
        "OnlyGenAiActivities": false
    }
}

Setting OnlyGenAiActivities to false keeps infrastructure spans (HTTP client, ASP.NET Core, and our custom tool-call activity source) visible alongside GenAI spans. If you set it to true, Langfuse filters out non-GenAI spans, which hides the tool-call child observations.

Setting Up Semantic Kernel with DeepSeek

Semantic Kernel handles the chat completion service, the plugin registry, and the automatic tool-calling loop. The DeepSeek API is OpenAI-compatible, so we use the OpenAI connector with a custom endpoint:

var apiKey = Environment.GetEnvironmentVariable("APP_API_KEY")
    ?? builder.Configuration["AI:ApiKey"]
    ?? throw new InvalidOperationException("AI API key is required.");

var endpoint = aiConfig["Endpoint"] ?? "https://api.deepseek.com";
var chatModel = aiConfig["ChatModel"] ?? "deepseek-chat";

var kernelBuilder = builder.Services.AddKernel();
kernelBuilder.AddOpenAIChatCompletion(
    modelId: chatModel,
    apiKey: apiKey,
    endpoint: new Uri(endpoint));

kernelBuilder.Plugins.AddFromType<WeatherPlugin>();

builder.Services.AddScoped<ChatService>();

The same pattern works for any OpenAI-compatible provider — just change the Endpoint URL and model name. Because Semantic Kernel owns the abstraction, switching providers is a one-line change.

System Architecture

Here is how the components fit together:

%%{init: {"theme":"base", "themeVariables": { "primaryColor":"#A855F7", "primaryTextColor":"#FFFFFF", "primaryBorderColor":"#7C3AED", "lineColor":"#6B7280", "secondaryColor":"#F59E0B", "secondaryTextColor":"#FFFFFF", "secondaryBorderColor":"#D97706", "tertiaryColor":"#06B6D4", "tertiaryTextColor":"#FFFFFF" }}}%% flowchart TD classDef client fill:#a855f7,stroke:#7e22ce,color:#fff,font-size:14px classDef service fill:#06b6d4,stroke:#0891b2,color:#fff,font-size:14px classDef ai fill:#f59e0b,stroke:#d97706,color:#fff,font-size:14px classDef obs fill:#10b981,stroke:#059669,color:#fff,font-size:14px classDef metric fill:#8b5cf6,stroke:#6d28d9,color:#fff,font-size:14px subgraph Client["🖥️ Client"] C1["HTTP Client"] end C1 --> API subgraph App["📦 .NET GenAI API"] API["ASP.NET Core \nMinimal API"] CS["ChatService"] SK["Semantic Kernel\n+ WeatherPlugin"] OTel["OpenTelemetry \nSDK"] Meters["Custom Meters \n📊 Histogram/Counter"] API --> CS CS --> SK CS --> OTel CS --> Meters Meters --> OTel end subgraph AI["🧠 AI Provider"] DS["DeepSeek API\ndeepseek-chat"] end SK --> DS subgraph Observability["🔭 Observability"] LF["Langfuse"] direction LR end OTel -->|"OTLP Traces (GenAI)"| LF class C1 client class API,CS,SK service class DS ai class LF,OTel obs class Meters metric

The flow is straightforward: HTTP requests hit the minimal API endpoints, which delegate to ChatService. That service calls DeepSeek through Semantic Kernel and emits OpenTelemetry spans, metrics, and logs. The Langfuse exporter picks up GenAI-specific spans and sends them to Langfuse.

Instrumenting Chat Completions with Langfuse

The ChatService integrates Langfuse tracing alongside Semantic Kernel calls. Here is the full pattern:

public sealed class WeatherPlugin
{
    private static readonly ActivitySource ActivitySource = new("LangfuseGenAI.WeatherPlugin");

    [KernelFunction("get_weather")]
    [Description("Get the current weather for a city")]
    public string GetWeather(string city)
    {
        using var activity = ActivitySource.StartActivity("get_weather", ActivityKind.Internal);
        activity?.SetTag("tool.name", "get_weather");
        activity?.SetTag("tool.city", city);

        var result = city switch
        {
            "Tokyo" => "Sunny, 22°C",
            "Paris" => "Rainy, 15°C",
            "New York" => "Cloudy, 18°C",
            "London" => "Foggy, 12°C",
            _ => "Sunny, 20°C"
        };

        activity?.SetTag("tool.weather", result);

        return result;
    }
}

public sealed class ChatService(
    Kernel kernel,
    ILogger<ChatService> logger,
    IServiceProvider serviceProvider)
{
    private static readonly Meter ChatMeter = new("LangfuseGenAI.Meters");
    private static readonly Counter<int> ChatRequestCounter =
        ChatMeter.CreateCounter<int>("genai.chat.requests", description: "Total chat requests");
    private static readonly Histogram<double> ChatLatency =
        ChatMeter.CreateHistogram<double>("genai.chat.latency_ms", description: "Chat request latency in ms");
    private static readonly Counter<int> TokenCounter =
        ChatMeter.CreateCounter<int>("genai.chat.tokens", description: "Total tokens used");

    public async Task<string> ProcessChatAsync(
        string userMessage,
        string? sessionId = null,
        string? userId = null,
        CancellationToken cancellationToken = default)
    {
        var sw = Stopwatch.StartNew();
        ChatRequestCounter.Add(1);

        serviceProvider.GetService<IOtelLangfuseTrace>()?.SetPublic(true);

        logger.LogInformation(
            "Processing chat request. SessionId: {SessionId}, UserId: {UserId}",
            sessionId, userId);

        try
        {
            var chatCompletion = kernel.GetRequiredService<IChatCompletionService>();
            var chatHistory = new ChatHistory();
            chatHistory.AddSystemMessage(
                "You are a weather assistant. For any weather question, you must call the get_weather tool for each requested city and use the tool results in your answer. Do not guess weather.");
            chatHistory.AddUserMessage(userMessage);

            var executionSettings = new OpenAIPromptExecutionSettings
            {
                Temperature = 0.7f,
                MaxTokens = 1000,
                ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
            };

            var result = await chatCompletion.GetChatMessageContentAsync(
                chatHistory,
                executionSettings: executionSettings,
                kernel: kernel,
                cancellationToken: cancellationToken);

            sw.Stop();
            ChatLatency.Record(sw.ElapsedMilliseconds);

            var outputTokens = result.Content?.Length / 4 ?? 0;
            TokenCounter.Add(outputTokens);

            logger.LogInformation(
                "Chat completed in {ElapsedMs}ms. Output tokens (est): {OutputTokens}",
                sw.ElapsedMilliseconds, outputTokens);

            return result.Content ?? string.Empty;
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Chat request failed for session {SessionId}", sessionId);
            throw;
        }
    }
}

The pattern covers all three pillars of observability:

Traces — Semantic Kernel’s built-in instrumentation emits a trace with nested generation observations, and the custom ActivitySource in WeatherPlugin emits child spans for each tool call.
Metrics — custom counters track request rate and token usage. A histogram tracks latency distribution. These can be scraped by Prometheus and visualized in Grafana for operational dashboards.
Logs — structured logs with session IDs and user IDs make it easy to correlate log entries with specific traces.

Adding Tool Calling with Semantic Kernel

Most real-world GenAI applications need more than text generation — they need the model to query databases, call APIs, or fetch live data. Semantic Kernel handles this with plugins and ToolCallBehavior.

The pattern has three parts:

1. Define a plugin with `KernelFunction`

Use a public method decorated with [KernelFunction] and [Description] to expose a callable tool. The method’s parameters and return type are automatically described to the model:

public sealed class WeatherPlugin
{
    [KernelFunction("get_weather")]
    [Description("Get the current weather for a city")]
    public string GetWeather(string city)
    {
        return city switch
        {
            "Tokyo" => "Sunny, 22°C",
            "Paris" => "Rainy, 15°C",
            _ => "Sunny, 20°C"
        };
    }
}

2. Register the plugin in `Program.cs`

Add the plugin to the kernel:

1
2

var kernelBuilder = builder.Services.AddKernel();
kernelBuilder.Plugins.AddFromType<WeatherPlugin>();

3. Enable auto-invocation in execution settings

Tell Semantic Kernel to automatically call the tool when the model requests it:

var executionSettings = new OpenAIPromptExecutionSettings
{
    Temperature = 0.7f,
    MaxTokens = 1000,
    ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
};

When the model decides it needs weather data, the flow is:

Model returns a get_weather function call request instead of a text response
Semantic Kernel invokes the WeatherPlugin.GetWeather method
The result is sent back to the model in a second round
Model produces the final text response using the tool output
The whole exchange appears as one trace with nested tool-call observations

Langfuse captures the tool call as a separate child observation. You can see the tool inputs (city name) and outputs (weather string) alongside the final model response — all correlated in one trace.

Tool Calling in the Langfuse Trace

This is the dashboard view readers should focus on: one root chat request, one or more generation spans, and get_weather child span(s) under the same trace.

Below is the Langfuse traces list. Each row is one request; the observation count tells you whether tool calls were captured as separate spans.

Opening the newest trace shows the full tree: the root POST /api/v1/chat/ span, the chat-completion span, the HTTP POST spans to DeepSeek, and the nested get_weather tool-call activity.

The Observations view lists every individual span across all traces, making it easy to filter for tool calls or generations.

What You Get in the Langfuse Dashboard

After running the sample and sending a few chat requests, Langfuse shows:

Traces view: Each chat request appears as a trace with the full input/output, model metadata, token counts, latency, and nested tool-call observations. When weather tool calling is active, the trace shows the get_weather child span under the root request.

Observations view: Every span is listed independently. You can filter by type (SPAN, GENERATION), name, or time range to inspect tool calls, HTTP calls, or model generations.

Sessions view: By passing sessionId in the request, Langfuse groups all traces from a single user session. This helps debug multi-turn conversations and identify UX issues.

Users view: Passing userId lets you see per-user activity, total events, and estimated cost over any time range.

Metrics: Custom metrics like request count, latency histograms, and token counters integrate with Prometheus for alerting. Set an alert when token consumption exceeds a threshold or when p95 latency goes above 5 seconds.

Connecting Logs to Traces

The structured logs emitted by ILogger<ChatService> contain SessionId and UserId properties. When combined with OpenTelemetry logging, each log record inherits the trace ID from the current activity context. This means you can jump from a Langfuse trace directly to the correlating log entries in Loki or Elasticsearch, even when the trace includes tool calls.

To enable OpenTelemetry logging, add the OTLP log exporter in Program.cs:

`1`	`builder.Logging.AddOpenTelemetry(logging => logging.AddOtlpExporter());`

With that in place, every LogInformation call is enriched with TraceId, SpanId, and resource attributes automatically.

Conclusion

LLM observability needs more than generic HTTP tracing. Langfuse gives .NET developers a purpose-built platform that understands GenAI semantics — model names, token counts, prompt/response pairs, temperature, finish reasons, and tool-call spans — all through standard OpenTelemetry exporters.

The integration with Semantic Kernel means you don’t need to choose between clean abstractions and deep observability. You get both: the kernel handles chat completions and the tool-calling loop, while OpenTelemetry and Langfuse capture the full execution graph in the dashboard.

You can find the sample code in this repository:

https://github.com/meysamhadeli/blog-samples/tree/main/src/langfuse-genai-observability-sample

Reference

Full Observability with OpenTelemetry, Grafana, Prometheus, Tempo, Loki, and Kibana in .NET

Thu, 26 Mar 2026 00:00:00 +0000

Introduction

Modern distributed systems break in surprising ways. When something goes wrong, you need more than just logs — you need to know what happened, when, and why, across all your services at once. That’s what observability is about.

There are three main types of telemetry data, each answers a different question:

Metrics — What’s happening? (how many requests, how many errors, how fast)
Traces — Where is it happening? (follow a request across all services)
Logs — Why is it happening? (detailed event messages)

In this article, we’ll build a complete observability setup for two .NET microservices — a Catalog API and an Order API — using:

OpenTelemetry — a standard way to collect telemetry from any stack
OpenTelemetry Collector — receives, processes, and routes data to backends
Prometheus — stores and queries metrics
Tempo — stores and queries traces (with span metrics generation)
Loki — stores and queries logs (native OTLP support in v3.x)
Elasticsearch + Kibana — second log backend with rich search
Grafana — one dashboard for metrics, traces, and logs
PostgreSQL — a separate database per microservice

Architecture Overview

OpenTelemetry is a CNCF (Cloud Native Computing Foundation) project that gives you one standard way to collect telemetry. Instead of wiring each backend separately, you instrument once and the collector sends data wherever you need.

The diagram below shows our full setup. Two .NET services (Catalog on :5000, Order on :5002) each have their own PostgreSQL database. Both send telemetry to the OpenTelemetry Collector, which fans it out: metrics to Prometheus, traces to Tempo, logs to Loki and Elasticsearch. Grafana queries all three backends for a unified view; Kibana provides a dedicated log interface.

%%{init: {"theme":"base", "themeVariables": { "primaryColor":"#A855F7", "primaryTextColor":"#FFFFFF", "primaryBorderColor":"#7C3AED", "lineColor":"#6B7280", "secondaryColor":"#F59E0B", "secondaryTextColor":"#FFFFFF", "secondaryBorderColor":"#D97706", "tertiaryColor":"#06B6D4", "tertiaryTextColor":"#FFFFFF", "tertiaryBorderColor":"#0891B2", "clusterBkg":"#1F2937", "clusterBorder":"#374151", "fontFamily":"arial", "fontSize":"14px", "edgeLabelBackground":"#374151", "nodeBorder":"#4B5563" }}}%% flowchart LR subgraph Apps["🚀 .NET Services"] Catalog["Catalog API
:5000"] Order["Order API
:5002"] end subgraph Collector["📡 OpenTelemetry Collector"] OTLP["OTLP Receiver
gRPC :4317"] Batch["Batch Processor"] end subgraph Backends["💾 Storage Backends"] Prom["📊 Prometheus
Metrics"] Tempo["🔍 Tempo
Traces"] Loki["📋 Loki
Logs"] ES["🔎 Elasticsearch
Logs"] end subgraph Viz["🖥️ Visualization"] Grafana["📈 Grafana"] Kibana["🔎 Kibana"] end Catalog & Order -->|"OTLP gRPC"| OTLP OTLP --> Batch Batch -->|"remote write"| Prom Batch -->|"OTLP gRPC"| Tempo Batch -->|"OTLP HTTP"| Loki Batch -->|"elasticsearch"| ES Prom --> Grafana Tempo --> Grafana Loki --> Grafana Tempo -.->|"span metrics"| Prom ES --> Kibana

How Data Moves Through the Stack

Let’s walk through each connection to see how telemetry flows:

1. Apps → OpenTelemetry Collector

Both .NET apps send traces, metrics, and logs to the collector using the OTLP protocol over gRPC (port 4317). One endpoint handles all three — no need for separate agents.

2. Collector → Prometheus (Metrics)

The collector pushes metrics to Prometheus at /api/v1/write using the prometheusremotewrite exporter. Prometheus needs the --web.enable-remote-write-receiver flag to accept pushes. This is different from the usual Prometheus “pull” model — the collector controls when data is sent, which works better in container environments where services come and go.

3. Collector → Tempo (Traces)

Traces go to Tempo via OTLP gRPC on port 4317. Tempo stores them and exposes a query API on port 3200. Tempo’s metrics generator is important: it looks at incoming spans and creates span metrics (request rate, error rate, duration) and service graph data, then writes these to Prometheus. You get service dashboards and dependency graphs without extra code.

4. Collector → Loki (Logs)

Logs are sent via OTLP HTTP to Loki at /otlp. Since Loki 3.x, Loki natively accepts OTLP-formatted logs — no need for a Promtail or fluent-bit intermediary. The collector batches logs with a 1s timeout for efficiency.

5. Collector → Elasticsearch (Logs — Alternative Path)

The same logs are also exported to Elasticsearch via the collector’s elasticsearch exporter. This gives you a second log storage backend, queryable through Kibana. Having both Loki and Elasticsearch lets you compare approaches or use Grafana for operational dashboards (via Loki) and Kibana for deep log analytics (via Elasticsearch).

6. Tempo → Prometheus (Span Metrics)

This is a key interaction that often goes unnoticed. Tempo’s metrics_generator reads incoming spans and computes:

Service graph edges — which services call which, request count, error rate
Span metrics — RED metrics per endpoint, per status code
DB query metrics — if spans contain DB span attributes

These computed metrics are remote-written to Prometheus, enabling the Service Graph view in Grafana’s Tempo datasource without any custom instrumentation.

7. Grafana → Prometheus, Tempo, Loki

Grafana is provisioned with three datasources:

Prometheus (default) — for metric queries and alerting
Tempo — for trace exploration via TraceQL, with the Prometheus datasource linked for service-map lookups
Loki — for log exploration via LogQL, with derived fields that extract trace IDs from log lines and link directly to Tempo traces

8. Kibana → Elasticsearch

Kibana connects to Elasticsearch on port 9200 and provides the familiar Discover UI for log searching, filtering, and dashboarding. Both Loki and Elasticsearch receive the same log data from the collector, so you can choose your preferred tool.

Here’s a summary of all ports and protocols:

Component	Port	Protocol	Used By
OTel Collector	4317	OTLP gRPC	.NET apps (traces + metrics + logs)
OTel Collector	4318	OTLP HTTP	Fallback ingestion
Prometheus	9090	HTTP API	Grafana queries + remote write
Tempo	4317	OTLP gRPC	Trace ingestion
Tempo	3200	HTTP	Grafana queries
Loki	3100	HTTP	OTLP logs + Grafana queries
Elasticsearch	9200	HTTP	Log indexing + Kibana queries
Kibana	5601	HTTP	User-facing log UI
Grafana	3000	HTTP	User-facing dashboard UI
Catalog API	5000	HTTP	Business API
Order API	5002	HTTP	Business API
PostgreSQL	5432	PostgreSQL	Data persistence

Infrastructure Setup with Docker Compose

Here’s the docker-compose infrastructure configuration. Save this as docker-compose.infrastructure.yaml:

name: otel-observability-stack
services:
  #######################################################
  #  Database: PostgreSQL (separate DBs per microservice)
  #######################################################
  postgres:
    image: postgres:16
    container_name: postgres
    restart: unless-stopped
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - infrastructure

  #######################################################
  #  App Services: Catalog API
  #######################################################
  catalog-api:
    build:
      context: ../
      dockerfile: Catalog.Api/Dockerfile
    container_name: catalog-api
    restart: unless-stopped
    ports:
      - "5000:5000"
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:5000
      - ConnectionStrings__CatalogDb=Host=postgres;Port=5432;Database=catalogdb;Username=postgres;Password=postgres
      - OpenTelemetry__OtlpExporter__Endpoint=http://otel-collector:4317
    depends_on:
      - postgres
      - otel-collector
    networks:
      - infrastructure

  #######################################################
  #  App Services: Order API
  #######################################################
  order-api:
    build:
      context: ../
      dockerfile: Order.Api/Dockerfile
    container_name: order-api
    restart: unless-stopped
    ports:
      - "5002:5002"
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:5002
      - ConnectionStrings__OrderDb=Host=postgres;Port=5432;Database=orderdb;Username=postgres;Password=postgres
      - OpenTelemetry__OtlpExporter__Endpoint=http://otel-collector:4317
    depends_on:
      - postgres
      - otel-collector
    networks:
      - infrastructure

  #######################################################
  #  OpenTelemetry Collector
  #######################################################
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.116.1
    container_name: otel-collector
    restart: unless-stopped
    command: ['--config=/etc/otelcol-contrib/config.yaml']
    volumes:
      - ./configs/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
      - "8888:8888"  # Prometheus self-metrics
      - "8889:8889"  # Prometheus exporter metrics
      - "13133:13133" # Health check
    networks:
      - infrastructure

  #######################################################
  #  Metrics: Prometheus
  #######################################################
  prometheus:
    image: prom/prometheus:v3.1.0
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./configs/prometheus.yaml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-remote-write-receiver'
    networks:
      - infrastructure

  #######################################################
  #  Traces: Tempo
  #######################################################
  tempo:
    image: grafana/tempo:2.7.0
    container_name: tempo
    restart: unless-stopped
    command: ['-config.file=/etc/tempo.yaml']
    volumes:
      - ./configs/tempo.yaml:/etc/tempo.yaml
    ports:
      - "3200:3200"
    networks:
      - infrastructure

  #######################################################
  #  Logs: Loki
  #######################################################
  loki:
    image: grafana/loki:3.3.2
    container_name: loki
    restart: unless-stopped
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./configs/loki-config.yaml:/etc/loki/local-config.yaml
    ports:
      - "3100:3100"
    networks:
      - infrastructure

  #######################################################
  #  Storage: Elasticsearch (for Kibana)
  #######################################################
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.16.0
    container_name: elasticsearch
    restart: unless-stopped
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
    ports:
      - "9200:9200"
    networks:
      - infrastructure

  #######################################################
  #  Visualization: Logs (Kibana)
  #######################################################
  kibana:
    image: docker.elastic.co/kibana/kibana:8.16.0
    container_name: kibana
    restart: unless-stopped
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - SERVER_NAME=kibana
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - infrastructure

  #######################################################
  #  Visualization: Grafana
  #######################################################
  grafana:
    image: grafana/grafana:11.4.0
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor
    depends_on:
      - prometheus
      - tempo
      - loki
    ports:
      - "3000:3000"
    volumes:
      - ./configs/grafana/provisioning:/etc/grafana/provisioning
    networks:
      - infrastructure

volumes:
  postgres_data:

networks:
  infrastructure:
    driver: bridge

OpenTelemetry Collector Configuration

The collector is the brain of the observability pipeline. It receives OTLP telemetry on port 4317 (gRPC) and 4318 (HTTP), batches it, then fans out to each backend. Save this as configs/otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
    tls:
      insecure: true

  otlphttp/loki:
    endpoint: "http://loki:3100/otlp"
    tls:
      insecure: true

  elasticsearch:
    endpoint: "http://elasticsearch:9200"
    tls:
      insecure: true

  otlp/tempo:
    endpoint: "tempo:4317"
    tls:
      insecure: true

  debug:
    verbosity: basic

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, debug]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite, debug]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki, elasticsearch, debug]

How the Collector Config Works

The collector has three parts — receivers (inbound), processors (buffer/batch), exporters (outbound) — wired together in pipelines:

Receivers: The otlp receiver accepts gRPC and HTTP. The .NET SDK sends all signals over gRPC by default.
Processors: The batch processor buffers up to 1024 items or waits 1 second, then sends to exporters. This cuts down API calls to backends.
Exporters: prometheusremotewrite → metrics, otlp/tempo → traces, otlphttp/loki + elasticsearch → logs, debug → console output for dev.
Pipelines: Three separate paths — traces, metrics, logs. Each pipeline runs receivers → processors → exporters independently.

Prometheus Configuration

Prometheus runs with --web.enable-remote-write-receiver, so it accepts pushed metrics from both the collector and Tempo — not just the usual pull/scrape model.

configs/prometheus.yaml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8888']

The only scrape target is the collector’s own metrics port (8888). All app metrics come via remote write.

Three sources feed Prometheus:

Source	Data	How
OTel Collector (remote write)	App metrics (HTTP duration, GC, etc.)	`prometheusremotewrite` exporter → `/api/v1/write`
Tempo metrics generator	Span RED metrics, service graph	Tempo computes from traces → remote writes to Prometheus
Scrape (port 8888)	Collector internal metrics	Prometheus scrapes `/metrics` periodically

Tempo Configuration

Tempo receives OTLP traces from the collector, stores them, and runs a metrics generator that produces service-level RED metrics from traces.

configs/tempo.yaml:

stream_over_http_enabled: true
server:
  http_listen_port: 3200
  log_level: info

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 'tempo:4317'
        http:
          endpoint: 'tempo:4318'

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 1h

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

Tempo Config Interactions

The metrics_generator section connects traces to metrics. It reads every span and creates:

Span metrics — request rate, error rate, latency per endpoint
Service graph metrics — which service calls which, with counts and durations

These are written to Prometheus. Grafana’s Tempo datasource uses them for Service Graph and RED metrics panels — no custom code.

stream_over_http_enabled: true lets Grafana stream large traces in real time.

Loki Configuration

Loki 3.x accepts OTLP logs natively — no log shipper needed. Save as configs/loki-config.yaml:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /tmp/loki

schema_config:
  configs:
    - from: 2020-05-15
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  allow_structured_metadata: true

Key settings: auth_enabled: false for local dev, TSDB index store (v13 schema), and allow_structured_metadata: true so you can filter logs by service.name or service.version.

Elasticsearch & Kibana

Elasticsearch runs single-node with security disabled for local development. The collector sends the same logs here as it does to Loki.

Kibana connects on port 9200 and provides a Discover UI for log search. Open http://localhost:5601, create a data view with pattern logs-* or otel-*, and you’ll see the same logs Grafana shows from Loki.

Grafana Provisioning

Grafana is auto-provisioned with datasources linking to Prometheus, Tempo, and Loki. Save this as configs/grafana/provisioning/datasources/datasource.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    uid: DS_PROMETHEUS

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    uid: tempo-uid
    jsonData:
      httpMethod: GET
      serviceMap:
        datasourceUid: DS_PROMETHEUS

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    uid: loki-uid
    jsonData:
      derivedFields:
        - datasourceUid: tempo-uid
          matcherRegex: "^.*?traceId=(\\w+).*$"
          name: traceId
          url: '$${__value.raw}'

Grafana Config Details

Tempo datasource: serviceMap.datasourceUid links to Prometheus so Grafana renders the Service Graph tab — shows Catalog→Order topology automatically.
Loki datasource: derivedFields extracts trace IDs from log lines. Click a trace ID in a log → jumps directly to that trace in Tempo. This is logs → traces correlation: spot an error, see the full request path.

Dashboard Provisioning

Create configs/grafana/provisioning/dashboards/dashboard.yml:

apiVersion: 1

providers:
  - name: "default"
    orgId: 1
    folder: ""
    type: file
    disableDeletion: false
    editable: true
    allowUiUpdates: true
    updateIntervalSeconds: 5
    options:
      path: /var/lib/grafana/dashboards

Add community dashboards (aspnetcore.json, aspnetcore-endpoint.json) to configs/grafana/dashboards/ — they visualize request rates, errors, latency, and .NET runtime metrics out of the box.

.NET Application Integration

Now let’s instrument our .NET microservices with OpenTelemetry, using Vertical Slice Architecture.

Project Structure

Two services, same pattern. Each has a class library + an API host:

src/
├── Catalog.API/
│   ├── Program.cs
│   └── Catalog.API.csproj
├── Ordering.API/
│   ├── Program.cs
│   └── Ordering.API.csproj
├── OTel.Shared/
│   ├── Extensions.cs
│   └── GlobalUsings.cs
└── docker-compose/
    ├── docker-compose.yml
    └── .env

Each service follows the same pattern: vertical slice features, clean Program.cs, infrastructure wiring in InfrastructureExtensions.cs.

Clean Program.cs (both services)

Both API projects use the exact same pattern — minimal Program.cs that delegates to AddInfrastructure():

Catalog.Api/Program.cs:

using Catalog.Extensions.Infrastructure;

var builder = WebApplication.CreateBuilder(args);
builder.AddInfrastructure();
var app = builder.Build();
app.UseInfrastructure();
app.Run();

Order.Api/Program.cs:

using Order.Extensions.Infrastructure;

var builder = WebApplication.CreateBuilder(args);
builder.AddInfrastructure();
var app = builder.Build();
app.UseInfrastructure();
app.Run();

The only difference is which namespace’s InfrastructureExtensions is used.

AddInfrastructure Extension (Catalog)

Here’s the full AddInfrastructure() for the Catalog service. The Order service follows the same pattern with its own DbContext and connection string:

public static class InfrastructureExtensions
{
    public static WebApplicationBuilder AddInfrastructure(
        this WebApplicationBuilder builder)
    {
        // Swagger / OpenAPI
        builder.Services.AddEndpointsApiExplorer();
        builder.Services.AddSwaggerGen();

        // Entity Framework + PostgreSQL
        var connectionString = builder.Configuration
            .GetConnectionString("CatalogDb");
        builder.Services.AddDbContext<CatalogDbContext>(opt =>
            opt.UseNpgsql(connectionString)
        );

        // MediatR
        builder.Services.AddMediatR(cfg =>
            cfg.RegisterServicesFromAssembly(
                typeof(CatalogRoot).Assembly)
        );

        // AutoMapper
        builder.Services.AddAutoMapper(cfg =>
            cfg.AddMaps(typeof(CatalogRoot).Assembly));

        // ---- OpenTelemetry ----
        var serviceVersion = Assembly.GetEntryAssembly()?
            .GetName().Version?.ToString() ?? "1.0.0";

        // Export metrics every 10s so Prometheus rate() queries work
        builder.Services.Configure<MetricReaderOptions>(o =>
            o.PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds = 10_000);

        builder.Services.AddOpenTelemetry()
            .ConfigureResource(resource => resource
                .AddService(
                    serviceName: builder.Environment.ApplicationName,
                    serviceVersion: serviceVersion,
                    serviceInstanceId: Environment.MachineName
                )
            )
            .WithTracing(tracing => tracing
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddGrpcClientInstrumentation()
                .AddOtlpExporter()
            )
            .WithMetrics(metrics => metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation()
                .AddOtlpExporter()
            )
            .WithLogging(logging => logging
                .AddOtlpExporter()
            );

        // ---- Service Discovery ----
        builder.Services.AddServiceDiscovery();

        // ---- Resilient HTTP Client to Order API ----
        builder.Services.AddHttpClient("OrderApi", client =>
        {
            client.BaseAddress = new Uri("http://localhost:5002");
        })
        .AddStandardResilienceHandler();

        // ---- Health Checks ----
        builder.Services.AddHealthChecks()
            .AddUrlGroup(
                new Uri("http://localhost:5000/healthz"),
                name: "Self Health Check",
                tags: ["healthz"]
            );

        return builder;
    }

    public static WebApplication UseInfrastructure(
        this WebApplication app)
    {
        if (app.Environment.IsDevelopment())
        {
            app.UseSwagger();
            app.UseSwaggerUI();
        }

        // Map vertical slice endpoints
        app.MapCreateProductEndpoint();
        app.MapGetProductByIdEndpoint();
        app.MapGetProductsEndpoint();

        // Health check endpoint
        app.MapHealthChecks("/healthz");

        return app;
    }
}

What each instrumentation does

Instrumentation	What it captures
`AddAspNetCoreInstrumentation()`	Incoming HTTP — duration, status, route
`AddHttpClientInstrumentation()`	Outgoing HTTP — dependencies, latency
`AddGrpcClientInstrumentation()`	gRPC client calls
`AddRuntimeInstrumentation()`	.NET runtime — GC, CPU, memory, thread pool
`.WithLogging().AddOtlpExporter()`	Structured logs via OTLP to collector

AddOtlpExporter() sends everything to the collector at http://localhost:4317 (default OTLP gRPC).

Connection Strings

Each service has its own database. Catalog uses CatalogDb, Order uses OrderDb.

Catalog.Api/appsettings.json:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    "CatalogDb": "Host=localhost;Port=5432;Database=catalogdb;Username=postgres;Password=postgres"
  },
  "OpenTelemetry": {
    "OtlpExporter": {
      "Endpoint": "http://localhost:4317"
    }
  }
}

Order.Api/appsettings.json — same structure, but ConnectionStrings.OrderDb points to orderdb.

Cross-Service Tracing

When Catalog calls Order, AddHttpClientInstrumentation() automatically propagates the traceparent header. This creates a distributed trace across both services:

%%{init: {"theme":"base", "themeVariables": { "primaryColor":"#A855F7", "primaryTextColor":"#FFFFFF", "primaryBorderColor":"#7C3AED", "lineColor":"#6B7280", "secondaryColor":"#F59E0B", "secondaryTextColor":"#FFFFFF", "secondaryBorderColor":"#D97706", "tertiaryColor":"#06B6D4", "tertiaryTextColor":"#FFFFFF", "tertiaryBorderColor":"#0891B2", "clusterBkg":"#1F2937", "clusterBorder":"#374151", "fontFamily":"arial", "fontSize":"14px", "edgeLabelBackground":"#374151", "nodeBorder":"#4B5563" }}}%% sequenceDiagram participant Client as 🖥️ Client participant Catalog as 🚀 Catalog API participant Order as 🚀 Order API participant DB as 🗄️ PostgreSQL Client->>Catalog: POST /api/products activate Catalog Catalog->>Order: POST /api/orders
(traceparent propagated) activate Order Order->>DB: INSERT INTO orders activate DB DB-->>Order: OK deactivate DB Order-->>Catalog: 200 OK deactivate Order Catalog-->>Client: 201 Created deactivate Catalog

In Grafana’s Tempo you see one trace with spans from both services — full request path visible across process boundaries.

Running the Stack

Start infrastructure:

`1`	`docker compose -f docker-compose.infrastructure.yaml up -d`

Run the APIs (separate terminals):

1
2

# Catalog API (port 5000)
cd Catalog.Api && dotnet run

1
2

# Order API (port 5002)
cd Order.Api && dotnet run

Access UIs:

Service	URL	Credentials
Grafana	`http://localhost:3000`	admin / admin
Kibana	`http://localhost:5601`	—
Prometheus	`http://localhost:9090`	—
Catalog Swagger	`http://localhost:5000/swagger`	—
Order Swagger	`http://localhost:5002/swagger`	—

Generate traffic by calling the API endpoints:

# Create a product
curl -X POST http://localhost:5000/api/products \
  -H "Content-Type: application/json" \
  -d '{"name":"Laptop","description":"Gaming laptop","price":1500,"stockQuantity":10,"category":"Electronics"}'

# List products
curl http://localhost:5000/api/products

# Create an order
curl -X POST http://localhost:5002/api/orders \
  -H "Content-Type: application/json" \
  -d '{"customerName":"John Doe","shippingAddress":"123 Main St","items":[{"productId":"...","productName":"Laptop","quantity":1,"unitPrice":1500}]}'

Exploring Data in Grafana and Kibana

Metrics (Prometheus)

Go to Explore → Prometheus. Try:

http_server_request_duration_seconds_count{http_route=~"/api/.*"} — request rate
dotnet_gc_collection_count — GC pressure
Pre-built ASP.NET Core dashboard shows rates, errors, latency, and runtime metrics.

Traces (Tempo)

Select Tempo → Search → filter by Catalog.Api or Order.Api. Each trace shows the waterfall: HTTP → DB calls → downstream HTTP. Create a product in Catalog, and you’ll see the cross-service trace spanning both services in one view.

Logs (Loki)

Select Loki → query:

{service_name="Catalog.Api"} — Catalog logs
{service_name="Order.Api"} |= "error" — errors only
Click a traceId in any log → jumps to the full trace in Tempo.

Logs (Kibana)

Open http://localhost:5601 → Discover → create a data view (logs-*). Filter by service.name, search for keywords. Kibana gives full-text log search over the same logs Loki ingests.

Key Takeaways

🛠 Instrumentation

Instrument once, route everywhere — OTel SDK sends traces, metrics, and logs to the collector; change backends without touching app code
Metric export interval matters — reduce PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds from 60s to 10s so Prometheus rate() and histogram_quantile() have enough samples

🔗 Correlation

Log correlation works end-to-end — Loki derived fields extract traceId from log lines and link directly to Tempo traces, closing the logs→traces loop in one click
Tempo metrics generator — automatically produces RED metrics and service graphs from spans, no custom instrumentation; enables Grafana Service Graph view

⚙ Architecture

Collector pipelines decouple concerns — three independent pipelines (traces → Tempo, metrics → Prometheus, logs → Loki + Elasticsearch) let you manage each signal separately
Dual log backends for different jobs — Loki for Grafana dashboards and operational queries, Elasticsearch/Kibana for full-text search and long-term retention

Conclusion

OpenTelemetry + Grafana stack gives you metrics, traces, and logs across your microservices — with zero vendor lock-in. Instrument once with the OTel SDK, the collector routes signals to all backends.

The complete sample code is available in the repository below, including both microservices, all deployment configs, and pre-built Grafana dashboards.

You can find the sample code in this repository:

🔗 https://github.com/meysamhadeli/blog-samples/tree/main/src/otel-observability-sample

Logging on dotnetuniversity

GenAI Observability in .NET: Logging, Tracing, and Metrics with Langfuse

What Langfuse Gives Us for GenAI Workloads

Getting Started: Package Setup

Running Langfuse Locally with Docker Compose

Configuring Langfuse with OpenTelemetry

Setting Up Semantic Kernel with DeepSeek

System Architecture

Instrumenting Chat Completions with Langfuse

Adding Tool Calling with Semantic Kernel

1. Define a plugin with KernelFunction

2. Register the plugin in Program.cs

3. Enable auto-invocation in execution settings

Tool Calling in the Langfuse Trace

What You Get in the Langfuse Dashboard

Connecting Logs to Traces

Conclusion

You can find the sample code in this repository:

Reference

Full Observability with OpenTelemetry, Grafana, Prometheus, Tempo, Loki, and Kibana in .NET

Introduction

Architecture Overview

How Data Moves Through the Stack

1. Apps → OpenTelemetry Collector

2. Collector → Prometheus (Metrics)

3. Collector → Tempo (Traces)

4. Collector → Loki (Logs)

5. Collector → Elasticsearch (Logs — Alternative Path)

6. Tempo → Prometheus (Span Metrics)

7. Grafana → Prometheus, Tempo, Loki

8. Kibana → Elasticsearch

Infrastructure Setup with Docker Compose

OpenTelemetry Collector Configuration

How the Collector Config Works

Prometheus Configuration

Tempo Configuration

Tempo Config Interactions

Loki Configuration

Elasticsearch & Kibana

Grafana Provisioning

Grafana Config Details

Dashboard Provisioning

.NET Application Integration

Project Structure

Clean Program.cs (both services)

AddInfrastructure Extension (Catalog)

What each instrumentation does

Connection Strings

Cross-Service Tracing

Running the Stack

Exploring Data in Grafana and Kibana

Metrics (Prometheus)

Traces (Tempo)

Logs (Loki)

Logs (Kibana)

Key Takeaways

🛠 Instrumentation

🔗 Correlation

⚙ Architecture

Conclusion

You can find the sample code in this repository:

1. Define a plugin with `KernelFunction`

2. Register the plugin in `Program.cs`