---
name: observability
description: Monitoring, logging, và tracing patterns. Use for Prometheus metrics, Grafana dashboards, Loki logging, và distributed tracing.
compatibility: ".NET 10+, Serilog, OpenTelemetry, Prometheus, Grafana"
metadata:
  author: Velik Ho
  version: "1.0"
---

# Observability Patterns / Mẫu Quan Sát

Monitoring, logging, và tracing cho GoodGo microservices.

## When to Use This Skill / Khi Nào Sử Dụng

Use this skill when:
- Implementing health checks / Triển khai health checks
- Configuring structured logging / Cấu hình structured logging
- Setting up Prometheus metrics / Cài đặt Prometheus metrics
- Implementing distributed tracing / Triển khai distributed tracing
- Creating Grafana dashboards / Tạo Grafana dashboards

## Core Concepts / Khái Niệm Cốt Lõi

### Three Pillars of Observability / Ba Trụ Cột của Observability

```
┌─────────────────────────────────────────────────────────────┐
│                      OBSERVABILITY                          │
├───────────────────┬───────────────────┬───────────────────┤
│     METRICS       │      LOGS         │     TRACES        │
│   (Prometheus)    │   (Loki/ELK)      │  (OpenTelemetry)  │
├───────────────────┼───────────────────┼───────────────────┤
│ What's happening  │ What happened     │ How it happened   │
│ (Số liệu)         │ (Nhật ký)         │ (Theo dõi)        │
└───────────────────┴───────────────────┴───────────────────┘
```

### Health Check Types / Các Loại Health Check

| Type | Purpose | Endpoint |
|------|---------|----------|
| **Liveness** | Is the app running? | `/health/live` |
| **Readiness** | Can the app accept traffic? | `/health/ready` |
| **Startup** | Has the app initialized? | `/health/startup` |

### Logging Levels / Các Mức Log

| Level | Purpose | Example |
|-------|---------|---------|
| **Trace** | Detailed debugging | Variable values |
| **Debug** | Development info | Method entry/exit |
| **Information** | Normal operations | Request processed |
| **Warning** | Potential problems | Slow query |
| **Error** | Errors handled | Exception caught |
| **Critical** | System failures | Database down |

## Key Patterns / Mẫu Chính

### Health Checks Configuration

```csharp
/// <summary>
/// EN: Configure comprehensive health checks.
/// VI: Cấu hình health checks toàn diện.
/// </summary>

// Program.cs
builder.Services.AddHealthChecks()
    // EN: Database health / VI: Health database
    .AddNpgSql(
        connectionString: builder.Configuration.GetConnectionString("DefaultConnection")!,
        name: "postgresql",
        failureStatus: HealthStatus.Unhealthy,
        tags: new[] { "db", "ready" })
    
    // EN: Redis cache / VI: Redis cache
    .AddRedis(
        redisConnectionString: builder.Configuration["Redis:ConnectionString"]!,
        name: "redis",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "cache", "ready" })
    
    // EN: Custom health check / VI: Health check tùy chỉnh
    .AddCheck<ExternalServiceHealthCheck>(
        "external-api",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "external", "ready" });

// EN: Map health endpoints / VI: Map health endpoints
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = _ => false, // EN: Just check app is running
    ResponseWriter = WriteMinimalResponse
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready"),
    ResponseWriter = WriteDetailedResponse
});

app.MapHealthChecks("/health", new HealthCheckOptions
{
    ResponseWriter = WriteDetailedResponse
});
```

### Structured Logging with Serilog

```csharp
/// <summary>
/// EN: Configure Serilog with structured logging.
/// VI: Cấu hình Serilog với structured logging.
/// </summary>

// Program.cs
Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Information()
    .MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
    .MinimumLevel.Override("Microsoft.Hosting.Lifetime", LogEventLevel.Information)
    .Enrich.FromLogContext()
    .Enrich.WithMachineName()
    .Enrich.WithEnvironmentName()
    .Enrich.WithProperty("Application", "IamService")
    .WriteTo.Console(new JsonFormatter())
    .WriteTo.Seq(builder.Configuration["Seq:ServerUrl"]!)
    .CreateLogger();

builder.Host.UseSerilog();

// EN: Usage in code / VI: Sử dụng trong code
public class OrderService
{
    private readonly ILogger<OrderService> _logger;

    public async Task<Order> CreateOrderAsync(CreateOrderCommand cmd)
    {
        // EN: Structured logging with properties
        // VI: Structured logging với properties
        _logger.LogInformation(
            "Creating order for user {UserId} with {ItemCount} items",
            cmd.UserId,
            cmd.Items.Count);

        try
        {
            var order = await ProcessOrderAsync(cmd);
            
            _logger.LogInformation(
                "Order {OrderId} created successfully. Total: {TotalAmount}",
                order.Id,
                order.TotalAmount);

            return order;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
                "Failed to create order for user {UserId}",
                cmd.UserId);
            throw;
        }
    }
}
```

### Prometheus Metrics

```csharp
/// <summary>
/// EN: Configure Prometheus metrics.
/// VI: Cấu hình Prometheus metrics.
/// </summary>

// Program.cs
builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddRuntimeInstrumentation()
            .AddPrometheusExporter();
    });

// EN: Map Prometheus endpoint / VI: Map Prometheus endpoint
app.MapPrometheusScrapingEndpoint();

/// <summary>
/// EN: Custom metrics for business logic.
/// VI: Metrics tùy chỉnh cho business logic.
/// </summary>
public class OrderMetrics
{
    private readonly Counter<int> _ordersCreated;
    private readonly Histogram<double> _orderProcessingDuration;
    private readonly UpDownCounter<int> _activeOrders;

    public OrderMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("GoodGo.Orders");

        _ordersCreated = meter.CreateCounter<int>(
            "orders_created_total",
            description: "Total number of orders created");

        _orderProcessingDuration = meter.CreateHistogram<double>(
            "order_processing_duration_seconds",
            description: "Duration of order processing in seconds");

        _activeOrders = meter.CreateUpDownCounter<int>(
            "active_orders",
            description: "Number of orders currently being processed");
    }

    public void OrderCreated(string status)
    {
        _ordersCreated.Add(1, new KeyValuePair<string, object?>("status", status));
    }

    public void RecordProcessingDuration(double seconds)
    {
        _orderProcessingDuration.Record(seconds);
    }

    public void IncrementActiveOrders() => _activeOrders.Add(1);
    public void DecrementActiveOrders() => _activeOrders.Add(-1);
}
```

### Distributed Tracing with OpenTelemetry

```csharp
/// <summary>
/// EN: Configure OpenTelemetry distributed tracing.
/// VI: Cấu hình OpenTelemetry distributed tracing.
/// </summary>

// Program.cs
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService("iam-service")
                .AddAttributes(new[]
                {
                    new KeyValuePair<string, object>("deployment.environment", 
                        builder.Environment.EnvironmentName)
                }))
            .AddAspNetCoreInstrumentation(options =>
            {
                options.RecordException = true;
                options.Filter = ctx => 
                    !ctx.Request.Path.StartsWithSegments("/health");
            })
            .AddHttpClientInstrumentation()
            .AddEntityFrameworkCoreInstrumentation()
            .AddSource("GoodGo.Orders")
            .AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"]!);
            });
    });

/// <summary>
/// EN: Custom activity for business operations.
/// VI: Activity tùy chỉnh cho business operations.
/// </summary>
public class OrderService
{
    private static readonly ActivitySource ActivitySource = new("GoodGo.Orders");

    public async Task<Order> ProcessOrderAsync(CreateOrderCommand cmd)
    {
        using var activity = ActivitySource.StartActivity("ProcessOrder");
        activity?.SetTag("user.id", cmd.UserId);
        activity?.SetTag("order.items_count", cmd.Items.Count);

        try
        {
            // EN: Create child span for validation
            // VI: Tạo child span cho validation
            using var validationActivity = ActivitySource.StartActivity("ValidateOrder");
            await ValidateOrderAsync(cmd);
            validationActivity?.SetStatus(ActivityStatusCode.Ok);

            // EN: Create child span for persistence
            // VI: Tạo child span cho persistence
            using var persistActivity = ActivitySource.StartActivity("PersistOrder");
            var order = await SaveOrderAsync(cmd);
            persistActivity?.SetTag("order.id", order.Id.ToString());

            activity?.SetTag("order.id", order.Id.ToString());
            activity?.SetStatus(ActivityStatusCode.Ok);

            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}
```

### Request Logging Middleware

```csharp
/// <summary>
/// EN: Middleware for request/response logging.
/// VI: Middleware cho logging request/response.
/// </summary>
public class RequestLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<RequestLoggingMiddleware> _logger;

    public RequestLoggingMiddleware(
        RequestDelegate next,
        ILogger<RequestLoggingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        var stopwatch = Stopwatch.StartNew();
        var requestId = context.TraceIdentifier;

        // EN: Log request / VI: Log request
        _logger.LogInformation(
            "Request {RequestId} started: {Method} {Path}",
            requestId,
            context.Request.Method,
            context.Request.Path);

        try
        {
            await _next(context);
        }
        finally
        {
            stopwatch.Stop();

            // EN: Log response / VI: Log response
            _logger.LogInformation(
                "Request {RequestId} completed: {StatusCode} in {ElapsedMs}ms",
                requestId,
                context.Response.StatusCode,
                stopwatch.ElapsedMilliseconds);
        }
    }
}
```

## Common Mistakes / Lỗi Thường Gặp

### 1. Logging Sensitive Data

```csharp
// ❌ BAD: Logging password
_logger.LogInformation("Login: {Email} {Password}", email, password);

// ✅ GOOD: Redact sensitive data
_logger.LogInformation("Login attempt for {Email}", email);
```

### 2. Missing Correlation ID

```csharp
// ❌ BAD: No correlation between logs
_logger.LogInformation("Processing order");
_logger.LogInformation("Order completed");

// ✅ GOOD: Include correlation ID
using (_logger.BeginScope(new Dictionary<string, object>
{
    ["OrderId"] = orderId,
    ["UserId"] = userId
}))
{
    _logger.LogInformation("Processing order");
    _logger.LogInformation("Order completed");
}
```

### 3. No Health Check for Dependencies

```csharp
// ❌ BAD: Only check if app runs
builder.Services.AddHealthChecks();

// ✅ GOOD: Check all dependencies
builder.Services.AddHealthChecks()
    .AddNpgSql(connectionString, name: "database")
    .AddRedis(redisConnection, name: "cache")
    .AddUrlGroup(new Uri(externalApi), name: "external-api");
```

## Quick Reference / Tham Chiếu Nhanh

### Serilog Configuration (appsettings.json)

```json
{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.Hosting.Lifetime": "Information",
        "System": "Warning"
      }
    },
    "WriteTo": [
      { "Name": "Console", "Args": { "formatter": "Serilog.Formatting.Json.JsonFormatter" } },
      { "Name": "Seq", "Args": { "serverUrl": "http://seq:5341" } }
    ],
    "Enrich": ["FromLogContext", "WithMachineName", "WithEnvironmentName"]
  }
}
```

### Health Check Endpoints

| Endpoint | Purpose | K8s Probe |
|----------|---------|-----------|
| `/health/live` | App is running | livenessProbe |
| `/health/ready` | App can accept traffic | readinessProbe |
| `/health/startup` | App has initialized | startupProbe |

### Common Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `http_requests_total` | Counter | Total HTTP requests |
| `http_request_duration_seconds` | Histogram | Request duration |
| `dotnet_gc_collections_total` | Counter | GC collections |
| `process_cpu_seconds_total` | Counter | CPU usage |

## Resources / Tài Nguyên

- [Detailed Examples](./references/REFERENCE.md) - Full configurations
- [Error Handling](../error-handling-patterns/SKILL.md) - Health checks
- [Docker Traefik](../docker-traefik/SKILL.md) - Container monitoring
- [Project Rules](../project-rules/SKILL.md) - Logging standards
