Introduction

Modern distributed systems break in surprising ways. When something goes wrong, you need more than just logs — you need to know what happened, when, and why, across all your services at once. That’s what observability is about.

There are three main types of telemetry data, each answers a different question:

Metrics — What’s happening? (how many requests, how many errors, how fast)
Traces — Where is it happening? (follow a request across all services)
Logs — Why is it happening? (detailed event messages)

In this article, we’ll build a complete observability setup for two .NET microservices — a Catalog API and an Order API — using:

OpenTelemetry — a standard way to collect telemetry from any stack
OpenTelemetry Collector — receives, processes, and routes data to backends
Prometheus — stores and queries metrics
Tempo — stores and queries traces (with span metrics generation)
Loki — stores and queries logs (native OTLP support in v3.x)
Elasticsearch + Kibana — second log backend with rich search
Grafana — one dashboard for metrics, traces, and logs
PostgreSQL — a separate database per microservice

Architecture Overview

OpenTelemetry is a CNCF (Cloud Native Computing Foundation) project that gives you one standard way to collect telemetry. Instead of wiring each backend separately, you instrument once and the collector sends data wherever you need.

The diagram below shows our full setup. Two .NET services (Catalog on :5000, Order on :5002) each have their own PostgreSQL database. Both send telemetry to the OpenTelemetry Collector, which fans it out: metrics to Prometheus, traces to Tempo, logs to Loki and Elasticsearch. Grafana queries all three backends for a unified view; Kibana provides a dedicated log interface.

Observability Stack Architecture

How Data Moves Through the Stack

Let’s walk through each connection to see how telemetry flows:

1. Apps → OpenTelemetry Collector

Both .NET apps send traces, metrics, and logs to the collector using the OTLP protocol over gRPC (port 4317). One endpoint handles all three — no need for separate agents.

2. Collector → Prometheus (Metrics)

The collector pushes metrics to Prometheus at /api/v1/write using the prometheusremotewrite exporter. Prometheus needs the --web.enable-remote-write-receiver flag to accept pushes. This is different from the usual Prometheus “pull” model — the collector controls when data is sent, which works better in container environments where services come and go.

3. Collector → Tempo (Traces)

Traces go to Tempo via OTLP gRPC on port 4317. Tempo stores them and exposes a query API on port 3200. Tempo’s metrics generator is important: it looks at incoming spans and creates span metrics (request rate, error rate, duration) and service graph data, then writes these to Prometheus. You get service dashboards and dependency graphs without extra code.

4. Collector → Loki (Logs)

Logs are sent via OTLP HTTP to Loki at /otlp. Since Loki 3.x, Loki natively accepts OTLP-formatted logs — no need for a Promtail or fluent-bit intermediary. The collector batches logs with a 1s timeout for efficiency.

5. Collector → Elasticsearch (Logs — Alternative Path)

The same logs are also exported to Elasticsearch via the collector’s elasticsearch exporter. This gives you a second log storage backend, queryable through Kibana. Having both Loki and Elasticsearch lets you compare approaches or use Grafana for operational dashboards (via Loki) and Kibana for deep log analytics (via Elasticsearch).

6. Tempo → Prometheus (Span Metrics)

This is a key interaction that often goes unnoticed. Tempo’s metrics_generator reads incoming spans and computes:

Service graph edges — which services call which, request count, error rate
Span metrics — RED metrics per endpoint, per status code
DB query metrics — if spans contain DB span attributes

These computed metrics are remote-written to Prometheus, enabling the Service Graph view in Grafana’s Tempo datasource without any custom instrumentation.

7. Grafana → Prometheus, Tempo, Loki

Grafana is provisioned with three datasources:

Prometheus (default) — for metric queries and alerting
Tempo — for trace exploration via TraceQL, with the Prometheus datasource linked for service-map lookups
Loki — for log exploration via LogQL, with derived fields that extract trace IDs from log lines and link directly to Tempo traces

8. Kibana → Elasticsearch

Kibana connects to Elasticsearch on port 9200 and provides the familiar Discover UI for log searching, filtering, and dashboarding. Both Loki and Elasticsearch receive the same log data from the collector, so you can choose your preferred tool.

Here’s a summary of all ports and protocols:

Component	Port	Protocol	Used By
OTel Collector	4317	OTLP gRPC	.NET apps (traces + metrics + logs)
OTel Collector	4318	OTLP HTTP	Fallback ingestion
Prometheus	9090	HTTP API	Grafana queries + remote write
Tempo	4317	OTLP gRPC	Trace ingestion
Tempo	3200	HTTP	Grafana queries
Loki	3100	HTTP	OTLP logs + Grafana queries
Elasticsearch	9200	HTTP	Log indexing + Kibana queries
Kibana	5601	HTTP	User-facing log UI
Grafana	3000	HTTP	User-facing dashboard UI
Catalog API	5000	HTTP	Business API
Order API	5002	HTTP	Business API
PostgreSQL	5432	PostgreSQL	Data persistence

Infrastructure Setup with Docker Compose

Here’s the docker-compose infrastructure configuration. Save this as docker-compose.infrastructure.yaml:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190


name: otel-observability-stack
services:
  #######################################################
  #  Database: PostgreSQL (separate DBs per microservice)
  #######################################################
  postgres:
    image: postgres:16
    container_name: postgres
    restart: unless-stopped
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - infrastructure

  #######################################################
  #  App Services: Catalog API
  #######################################################
  catalog-api:
    build:
      context: ../
      dockerfile: Catalog.Api/Dockerfile
    container_name: catalog-api
    restart: unless-stopped
    ports:
      - "5000:5000"
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:5000
      - ConnectionStrings__CatalogDb=Host=postgres;Port=5432;Database=catalogdb;Username=postgres;Password=postgres
      - OpenTelemetry__OtlpExporter__Endpoint=http://otel-collector:4317
    depends_on:
      - postgres
      - otel-collector
    networks:
      - infrastructure

  #######################################################
  #  App Services: Order API
  #######################################################
  order-api:
    build:
      context: ../
      dockerfile: Order.Api/Dockerfile
    container_name: order-api
    restart: unless-stopped
    ports:
      - "5002:5002"
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://+:5002
      - ConnectionStrings__OrderDb=Host=postgres;Port=5432;Database=orderdb;Username=postgres;Password=postgres
      - OpenTelemetry__OtlpExporter__Endpoint=http://otel-collector:4317
    depends_on:
      - postgres
      - otel-collector
    networks:
      - infrastructure

  #######################################################
  #  OpenTelemetry Collector
  #######################################################
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.116.1
    container_name: otel-collector
    restart: unless-stopped
    command: ['--config=/etc/otelcol-contrib/config.yaml']
    volumes:
      - ./configs/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"  # OTLP gRPC
      - "4318:4318"  # OTLP HTTP
      - "8888:8888"  # Prometheus self-metrics
      - "8889:8889"  # Prometheus exporter metrics
      - "13133:13133" # Health check
    networks:
      - infrastructure

  #######################################################
  #  Metrics: Prometheus
  #######################################################
  prometheus:
    image: prom/prometheus:v3.1.0
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./configs/prometheus.yaml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-remote-write-receiver'
    networks:
      - infrastructure

  #######################################################
  #  Traces: Tempo
  #######################################################
  tempo:
    image: grafana/tempo:2.7.0
    container_name: tempo
    restart: unless-stopped
    command: ['-config.file=/etc/tempo.yaml']
    volumes:
      - ./configs/tempo.yaml:/etc/tempo.yaml
    ports:
      - "3200:3200"
    networks:
      - infrastructure

  #######################################################
  #  Logs: Loki
  #######################################################
  loki:
    image: grafana/loki:3.3.2
    container_name: loki
    restart: unless-stopped
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./configs/loki-config.yaml:/etc/loki/local-config.yaml
    ports:
      - "3100:3100"
    networks:
      - infrastructure

  #######################################################
  #  Storage: Elasticsearch (for Kibana)
  #######################################################
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.16.0
    container_name: elasticsearch
    restart: unless-stopped
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
    ports:
      - "9200:9200"
    networks:
      - infrastructure

  #######################################################
  #  Visualization: Logs (Kibana)
  #######################################################
  kibana:
    image: docker.elastic.co/kibana/kibana:8.16.0
    container_name: kibana
    restart: unless-stopped
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - SERVER_NAME=kibana
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - infrastructure

  #######################################################
  #  Visualization: Grafana
  #######################################################
  grafana:
    image: grafana/grafana:11.4.0
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor
    depends_on:
      - prometheus
      - tempo
      - loki
    ports:
      - "3000:3000"
    volumes:
      - ./configs/grafana/provisioning:/etc/grafana/provisioning
    networks:
      - infrastructure

volumes:
  postgres_data:

networks:
  infrastructure:
    driver: bridge

OpenTelemetry Collector Configuration

The collector is the brain of the observability pipeline. It receives OTLP telemetry on port 4317 (gRPC) and 4318 (HTTP), batches it, then fans out to each backend. Save this as configs/otel-collector-config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
    tls:
      insecure: true

  otlphttp/loki:
    endpoint: "http://loki:3100/otlp"
    tls:
      insecure: true

  elasticsearch:
    endpoint: "http://elasticsearch:9200"
    tls:
      insecure: true

  otlp/tempo:
    endpoint: "tempo:4317"
    tls:
      insecure: true

  debug:
    verbosity: basic

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, debug]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite, debug]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/loki, elasticsearch, debug]

How the Collector Config Works

The collector has three parts — receivers (inbound), processors (buffer/batch), exporters (outbound) — wired together in pipelines:

Receivers: The otlp receiver accepts gRPC and HTTP. The .NET SDK sends all signals over gRPC by default.
Processors: The batch processor buffers up to 1024 items or waits 1 second, then sends to exporters. This cuts down API calls to backends.
Exporters: prometheusremotewrite → metrics, otlp/tempo → traces, otlphttp/loki + elasticsearch → logs, debug → console output for dev.
Pipelines: Three separate paths — traces, metrics, logs. Each pipeline runs receivers → processors → exporters independently.

Prometheus Configuration

Prometheus runs with --web.enable-remote-write-receiver, so it accepts pushed metrics from both the collector and Tempo — not just the usual pull/scrape model.

configs/prometheus.yaml:

1
2
3
4
5
6
7
8
9


global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8888']

The only scrape target is the collector’s own metrics port (8888). All app metrics come via remote write.

Three sources feed Prometheus:

Source	Data	How
OTel Collector (remote write)	App metrics (HTTP duration, GC, etc.)	`prometheusremotewrite` exporter → `/api/v1/write`
Tempo metrics generator	Span RED metrics, service graph	Tempo computes from traces → remote writes to Prometheus
Scrape (port 8888)	Collector internal metrics	Prometheus scrapes `/metrics` periodically

Tempo Configuration

Tempo receives OTLP traces from the collector, stores them, and runs a metrics generator that produces service-level RED metrics from traces.

configs/tempo.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


stream_over_http_enabled: true
server:
  http_listen_port: 3200
  log_level: info

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 'tempo:4317'
        http:
          endpoint: 'tempo:4318'

ingester:
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 1h

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true
  traces_storage:
    path: /var/tempo/generator/traces

storage:
  trace:
    backend: local
    wal:
      path: /var/tempo/wal
    local:
      path: /var/tempo/blocks

Tempo Config Interactions

The metrics_generator section connects traces to metrics. It reads every span and creates:

Span metrics — request rate, error rate, latency per endpoint
Service graph metrics — which service calls which, with counts and durations

These are written to Prometheus. Grafana’s Tempo datasource uses them for Service Graph and RED metrics panels — no custom code.

stream_over_http_enabled: true lets Grafana stream large traces in real time.

Loki Configuration

Loki 3.x accepts OTLP logs natively — no log shipper needed. Save as configs/loki-config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /tmp/loki

schema_config:
  configs:
    - from: 2020-05-15
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  allow_structured_metadata: true

Key settings: auth_enabled: false for local dev, TSDB index store (v13 schema), and allow_structured_metadata: true so you can filter logs by service.name or service.version.

Elasticsearch & Kibana

Elasticsearch runs single-node with security disabled for local development. The collector sends the same logs here as it does to Loki.

Kibana connects on port 9200 and provides a Discover UI for log search. Open http://localhost:5601, create a data view with pattern logs-* or otel-*, and you’ll see the same logs Grafana shows from Loki.

Grafana Provisioning

Grafana is auto-provisioned with datasources linking to Prometheus, Tempo, and Loki. Save this as configs/grafana/provisioning/datasources/datasource.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    uid: prometheus-uid

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    uid: tempo-uid
    jsonData:
      httpMethod: GET
      serviceMap:
        datasourceUid: prometheus-uid

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    uid: loki-uid
    jsonData:
      derivedFields:
        - datasourceUid: tempo-uid
          matcherRegex: "^.*?traceId=(\\w+).*$"
          name: traceId
          url: '$${__value.raw}'

Grafana Config Details

Tempo datasource: serviceMap.datasourceUid links to Prometheus so Grafana renders the Service Graph tab — shows Catalog→Order topology automatically.
Loki datasource: derivedFields extracts trace IDs from log lines. Click a trace ID in a log → jumps directly to that trace in Tempo. This is logs → traces correlation: spot an error, see the full request path.

Dashboard Provisioning

Create configs/grafana/provisioning/dashboards/dashboard.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


apiVersion: 1
providers:
  - name: 'ASP.NET Core Dashboards'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /var/lib/grafana/dashboards

Add community dashboards (aspnetcore.json, aspnetcore-endpoint.json) to configs/grafana/dashboards/ — they visualize request rates, errors, latency, and .NET runtime metrics out of the box.

.NET Application Integration

Now let’s instrument our .NET microservices with OpenTelemetry, using Vertical Slice Architecture.

Project Structure

Two services, same pattern. Each has a class library + an API host:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


src/
├── Catalog.API/
│   ├── Program.cs
│   └── Catalog.API.csproj
├── Ordering.API/
│   ├── Program.cs
│   └── Ordering.API.csproj
├── OTel.Shared/
│   ├── Extensions.cs
│   └── GlobalUsings.cs
└── docker-compose/
    ├── docker-compose.yml
    └── .env

Each service follows the same pattern: vertical slice features, clean Program.cs, infrastructure wiring in InfrastructureExtensions.cs.

Clean Program.cs (both services)

Both API projects use the exact same pattern — minimal Program.cs that delegates to AddInfrastructure():

Catalog.Api/Program.cs:

1
2
3
4
5
6
7


using Catalog.Extensions.Infrastructure;

var builder = WebApplication.CreateBuilder(args);
builder.AddInfrastructure();
var app = builder.Build();
app.UseInfrastructure();
app.Run();

Order.Api/Program.cs:

1
2
3
4
5
6
7


using Order.Extensions.Infrastructure;

var builder = WebApplication.CreateBuilder(args);
builder.AddInfrastructure();
var app = builder.Build();
app.UseInfrastructure();
app.Run();

The only difference is which namespace’s InfrastructureExtensions is used.

AddInfrastructure Extension (Catalog)

Here’s the full AddInfrastructure() for the Catalog service. The Order service follows the same pattern with its own DbContext and connection string:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92


public static class InfrastructureExtensions
{
    public static WebApplicationBuilder AddInfrastructure(
        this WebApplicationBuilder builder)
    {
        // Swagger / OpenAPI
        builder.Services.AddEndpointsApiExplorer();
        builder.Services.AddSwaggerGen();

        // Entity Framework + PostgreSQL
        var connectionString = builder.Configuration
            .GetConnectionString("CatalogDb");
        builder.Services.AddDbContext<CatalogDbContext>(opt =>
            opt.UseNpgsql(connectionString)
        );

        // MediatR
        builder.Services.AddMediatR(cfg =>
            cfg.RegisterServicesFromAssembly(
                typeof(CatalogRoot).Assembly)
        );

        // AutoMapper
        builder.Services.AddAutoMapper(cfg =>
            cfg.AddMaps(typeof(CatalogRoot).Assembly));

        // ---- OpenTelemetry ----
        var serviceVersion = Assembly.GetEntryAssembly()?
            .GetName().Version?.ToString() ?? "1.0.0";

        builder.Services.AddOpenTelemetry()
            .ConfigureResource(resource => resource
                .AddService(
                    serviceName: builder.Environment.ApplicationName,
                    serviceVersion: serviceVersion,
                    serviceInstanceId: Environment.MachineName
                )
            )
            .WithTracing(tracing => tracing
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddGrpcClientInstrumentation()
                .AddOtlpExporter()
            )
            .WithMetrics(metrics => metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation()
                .AddOtlpExporter()
            );

        // ---- Service Discovery ----
        builder.Services.AddServiceDiscovery();

        // ---- Resilient HTTP Client to Order API ----
        builder.Services.AddHttpClient("OrderApi", client =>
        {
            client.BaseAddress = new Uri("http://localhost:5002");
        })
        .AddStandardResilienceHandler();

        // ---- Health Checks ----
        builder.Services.AddHealthChecks()
            .AddUrlGroup(
                new Uri("http://localhost:5000/healthz"),
                name: "Self Health Check",
                tags: ["healthz"]
            );

        return builder;
    }

    public static WebApplication UseInfrastructure(
        this WebApplication app)
    {
        if (app.Environment.IsDevelopment())
        {
            app.UseSwagger();
            app.UseSwaggerUI();
        }

        // Map vertical slice endpoints
        app.MapCreateProductEndpoint();
        app.MapGetProductByIdEndpoint();
        app.MapGetProductsEndpoint();

        // Health check endpoint
        app.MapHealthChecks("/healthz");

        return app;
    }
}

What each instrumentation does

Instrumentation	What it captures
`AddAspNetCoreInstrumentation()`	Incoming HTTP — duration, status, route
`AddHttpClientInstrumentation()`	Outgoing HTTP — dependencies, latency
`AddGrpcClientInstrumentation()`	gRPC client calls
`AddRuntimeInstrumentation()`	.NET runtime — GC, CPU, memory, thread pool

AddOtlpExporter() sends everything to the collector at http://localhost:4317 (default OTLP gRPC).

Connection Strings

Each service has its own database. Catalog uses CatalogDb, Order uses OrderDb.

Catalog.Api/appsettings.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    "CatalogDb": "Host=localhost;Port=5432;Database=catalogdb;Username=postgres;Password=postgres"
  },
  "OpenTelemetry": {
    "OtlpExporter": {
      "Endpoint": "http://localhost:4317"
    }
  }
}

Order.Api/appsettings.json — same structure, but ConnectionStrings.OrderDb points to orderdb.

Cross-Service Tracing

When Catalog calls Order, AddHttpClientInstrumentation() automatically propagates the traceparent header. This creates a distributed trace across both services:

catalog-api: POST /api/products
  └─ http → order-api: POST /api/orders
       └─ npgsql: INSERT INTO orders

In Grafana’s Tempo you see one trace with spans from both services — full request path visible across process boundaries.

Running the Stack

Start infrastructure:

1

docker compose -f docker-compose.infrastructure.yaml up -d

Run the APIs (separate terminals):

1
2


# Catalog API (port 5000)
cd Catalog.Api && dotnet run

1
2


# Order API (port 5002)
cd Order.Api && dotnet run

Access UIs:

Service	URL	Credentials
Grafana	`http://localhost:3000`	admin / admin
Kibana	`http://localhost:5601`	—
Prometheus	`http://localhost:9090`	—
Catalog Swagger	`http://localhost:5000/swagger`	—
Order Swagger	`http://localhost:5002/swagger`	—

Generate traffic by calling the API endpoints:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Create a product
curl -X POST http://localhost:5000/api/products \
  -H "Content-Type: application/json" \
  -d '{"name":"Laptop","description":"Gaming laptop","price":1500,"stockQuantity":10,"category":"Electronics"}'

# List products
curl http://localhost:5000/api/products

# Create an order
curl -X POST http://localhost:5002/api/orders \
  -H "Content-Type: application/json" \
  -d '{"customerName":"John Doe","shippingAddress":"123 Main St","items":[{"productId":"...","productName":"Laptop","quantity":1,"unitPrice":1500}]}'

Exploring Data in Grafana and Kibana

Metrics (Prometheus)

Go to Explore → Prometheus. Try:

http_server_duration_milliseconds_count{http_route=~"/api/.*"} — request rate
dotnet_gc_collection_count — GC pressure
Pre-built ASP.NET Core dashboard shows rates, errors, latency, and runtime metrics.

Traces (Tempo)

Select Tempo → Search → filter by Catalog.Api or Order.Api. Each trace shows the waterfall: HTTP → DB calls → downstream HTTP. Create a product in Catalog, and you’ll see the cross-service trace spanning both services in one view.

Logs (Loki)

Select Loki → query:

{service_name="Catalog.Api"} — Catalog logs
{service_name="Order.Api"} |= "error" — errors only
Click a traceId in any log → jumps to the full trace in Tempo.

Logs (Kibana)

Open http://localhost:5601 → Discover → create a data view (logs-*). Filter by service.name, search for keywords. Kibana gives full-text log search over the same logs Loki ingests.

Debugging Flow

Dashboard shows error spike
Click → Explore → Tempo: find the trace
Trace waterfall shows where time was spent
Click log line → Loki: see the error with stack trace
Deep dive → Kibana full-text search across historical logs

Conclusion

OpenTelemetry + Grafana stack gives you metrics, traces, and logs across your microservices — with zero vendor lock-in. Instrument once with the OTel SDK, the collector routes signals to all backends.

Key takeaways:

OTel Collector decouples your code from backends — three independent pipelines for traces, metrics, logs
Tempo’s metrics generator turns spans into RED metrics and service graphs automatically
Loki 3.x OTLP means no log shipper — the collector sends logs directly
Dual log backends — Loki for dashboards, Elasticsearch/Kibana for full-text search
One AddInfrastructure() call wires up OpenTelemetry, EF Core, MediatR, Swagger per service
Cross-service tracing works automatically with AddHttpClientInstrumentation()

The complete sample code is available in the repository below, including both microservices, all deployment configs, and pre-built Grafana dashboards.

You can find the sample code in this repository:

🔗 https://github.com/meysamhadeli/blog-samples/tree/main/src/otel-observability-sample

Full Observability with OpenTelemetry, Grafana, Prometheus, Tempo, Loki, and Kibana in .NET

Set up a full observability stack for .NET microservices using OpenTelemetry instrumentation, OpenTelemetry Collector data pipeline, and Grafana with Prometheus, Tempo, and Loki — plus Kibana for log visualization.