# Khả Năng Quan Sát & Giám Sát

Observability and monitoring patterns for GoodGo microservices. Use when adding metrics, implementing logging, setting up tracing, creating health checks, or debugging production issues.
> Các pattern observability và monitoring cho microservices GoodGo. Sử dụng khi thêm metrics, triển khai logging, thiết lập tracing, tạo health checks, hoặc debug các vấn đề production.

## Tổng Quan

This skill covers the three pillars of observability (logs, metrics, traces) and how to implement them in GoodGo microservices. It includes structured logging, Prometheus metrics, distributed tracing with OpenTelemetry, health checks, and error tracking.

Skill này bao gồm ba trụ cột của observability (logs, metrics, traces) và cách triển khai chúng trong microservices GoodGo. Nó bao gồm structured logging, Prometheus metrics, distributed tracing với OpenTelemetry, health checks, và error tracking.

## Khi Nào Sử Dụng

Use this skill when:
- Setting up logging infrastructure
- Implementing metrics collection
- Adding distributed tracing
- Creating health check endpoints
- Setting up monitoring dashboards
- Debugging production issues
- Implementing alerting rules
- Analyzing performance bottlenecks

Sử dụng skill này khi:
- Thiết lập hạ tầng logging
- Triển khai thu thập metrics
- Thêm distributed tracing
- Tạo health check endpoints
- Thiết lập monitoring dashboards
- Debug các vấn đề production
- Triển khai alerting rules
- Phân tích performance bottlenecks

## Khái Niệm Chính

### Ba Trụ Cột Của Observability

1. **Logs**: Event records for debugging and auditing
2. **Metrics**: Numerical measurements over time (counters, gauges, histograms)
3. **Traces**: Request flow across services (distributed tracing)

**VI**:
1. **Logs**: Bản ghi sự kiện để debug và audit
2. **Metrics**: Đo lường số học theo thời gian (counters, gauges, histograms)
3. **Traces**: Luồng request qua các services (distributed tracing)

### Công Nghệ

- **Logging**: `@goodgo/logger` (Pino-based structured logging)
- **Metrics**: Prometheus + Grafana
- **Tracing**: OpenTelemetry + Jaeger (`@goodgo/tracing`)
- **Correlation IDs**: Request tracking across services

**VI**:
- **Logging**: `@goodgo/logger` (structured logging dựa trên Pino)
- **Metrics**: Prometheus + Grafana
- **Tracing**: OpenTelemetry + Jaeger (`@goodgo/tracing`)
- **Correlation IDs**: Theo dõi request qua các services

### Kiến Trúc Observability Stack

Observability stack bao gồm ba trụ cột hoạt động cùng nhau để cung cấp khả năng quan sát toàn diện vào hành vi hệ thống:

```mermaid
graph TB
    subgraph "Application Layer"
        App[Microservice]
    end
    
    subgraph "Three Pillars of Observability"
        Logs[Logs<br/>Winston/Pino]
        Metrics[Metrics<br/>Prometheus]
        Traces[Traces<br/>OpenTelemetry]
    end
    
    subgraph "Aggregation & Storage"
        Loki[Loki<br/>Log Aggregation]
        Prom[Prometheus<br/>Metrics Storage]
        Jaeger[Jaeger<br/>Trace Storage]
    end
    
    subgraph "Visualization & Alerting"
        Grafana[Grafana<br/>Dashboards]
        AlertManager[AlertManager<br/>Alerts]
    end
    
    App -->|Structured Logs| Logs
    App -->|HTTP Metrics| Metrics
    App -->|Distributed Spans| Traces
    
    Logs -->|Collect| Loki
    Metrics -->|Scrape /metrics| Prom
    Traces -->|Export| Jaeger
    
    Loki -->|Query| Grafana
    Prom -->|Query| Grafana
    Prom -->|Alerts| AlertManager
    Jaeger -->|Query| Grafana
    
    style App fill:#e1f5ff
    style Logs fill:#fff4e1
    style Metrics fill:#e1ffe1
    style Traces fill:#ffe1f5
    style Grafana fill:#e1e1ff
```

## Các Pattern Thường Dùng

### Logging Có Cấu Trúc

Use structured logging with correlation IDs for request tracking.

Sử dụng structured logging với correlation IDs để theo dõi request.

**Ví dụ từ codebase**: [`services/iam-service/src/middlewares/logger.middleware.ts`](../../../services/iam-service/src/middlewares/logger.middleware.ts)

```typescript
import { Request, Response, NextFunction } from 'express';
import { logger } from '@goodgo/logger';
import { getCorrelationId, getRequestId } from './correlation.middleware';

export const requestLogger = (req: Request, res: Response, next: NextFunction): void => {
  // Bỏ qua logging chi tiết cho health checks và metrics
  if (req.path.startsWith('/health') || req.path.startsWith('/metrics')) {
    return next();
  }

  const start = Date.now();

  res.on('finish', () => {
    const duration = Date.now() - start;
    const correlationId = getCorrelationId(req);
    const requestId = getRequestId(req);

    logger.info('Request processed / Request đã xử lý', {
      correlationId,
      requestId,
      method: req.method,
      path: req.path,
      query: req.query,
      statusCode: res.statusCode,
      duration: `${duration}ms`,
      contentLength: res.get('Content-Length') || 0,
      userAgent: req.get('User-Agent'),
      ip: req.ip,
      userId: (req as any).user?.userId,
    });
  });

  next();
};
```

### Luồng Logging

Luồng logging cho thấy cách các request được log với correlation IDs và chảy qua hệ thống:

```mermaid
sequenceDiagram
    participant Client
    participant Service as Microservice
    participant Logger as Winston/Pino Logger
    participant Aggregator as Log Aggregator<br/>(Loki)
    participant Dashboard as Grafana Dashboard
    
    Client->>Service: HTTP Request<br/>(with x-correlation-id)
    Service->>Service: Generate/Extract<br/>Correlation ID
    Service->>Logger: Log Request Start<br/>{correlationId, method, url}
    Service->>Service: Process Request
    Service->>Logger: Log Business Event<br/>{correlationId, event, data}
    Service->>Client: HTTP Response<br/>(with x-correlation-id)
    Service->>Logger: Log Request End<br/>{correlationId, status, duration}
    
    Logger->>Aggregator: Send Structured Logs<br/>(JSON format)
    Aggregator->>Dashboard: Index & Store Logs
    Dashboard->>Dashboard: Query by correlationId<br/>to trace request flow
```

### Correlation IDs

Use correlation IDs to track requests across services.

Sử dụng correlation IDs để theo dõi request qua các services.

**Ví dụ từ codebase**: [`services/iam-service/src/middlewares/correlation.middleware.ts`](../../../services/iam-service/src/middlewares/correlation.middleware.ts)

```typescript
import { Request, Response, NextFunction } from 'express';
import { randomUUID } from 'crypto';
import { logger } from '@goodgo/logger';

export const CORRELATION_ID_HEADER = 'x-correlation-id';
export const REQUEST_ID_HEADER = 'x-request-id';

export const correlationMiddleware = (
  options: {
    headerName?: string;
    generateId?: () => string;
    skipPaths?: string[];
  } = {}
) => {
  const {
    headerName = CORRELATION_ID_HEADER,
    generateId = randomUUID,
    skipPaths = ['/health', '/metrics', '/favicon.ico'],
  } = options;

  return (req: Request, res: Response, next: NextFunction) => {
    // Lấy correlation ID từ header hoặc tạo mới
    const correlationId = req.headers[headerName.toLowerCase()] as string || generateId();
    const requestId = generateId();

    // Gắn vào request object
    req.correlationId = correlationId;
    req.requestId = requestId;

    // Thêm vào response headers
    res.setHeader(headerName, correlationId);
    res.setHeader(REQUEST_ID_HEADER, requestId);

    // Log request start
    logger.info('Request started / Request bắt đầu', {
      correlationId,
      requestId,
      method: req.method,
      url: req.url,
      userAgent: req.get('User-Agent'),
      ip: req.ip,
    });

    next();
  };
};
```

### Thu Thập Metrics

Expose Prometheus metrics for monitoring and alerting.

Expose Prometheus metrics để monitoring và alerting.

**Ví dụ từ codebase**: [`services/iam-service/src/middlewares/metrics.middleware.ts`](../../../services/iam-service/src/middlewares/metrics.middleware.ts)

```typescript
import { Request, Response, NextFunction } from 'express';
import client from 'prom-client';
import { getCorrelationId } from './correlation.middleware';

// Tạo Registry để đăng ký các metrics
const register = client.register;

// Thu thập các metrics mặc định
client.collectDefaultMetrics({ register });

// Tạo histogram cho thời lượng request HTTP
const httpRequestDurationSeconds = new client.Histogram({
    name: 'http_request_duration_seconds',
    help: 'Duration of HTTP requests in seconds',
    labelNames: ['method', 'route', 'status_code', 'correlation_id'],
    buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 2, 5, 10],
});

// Tạo counter cho tổng số request HTTP
const httpRequestsTotal = new client.Counter({
    name: 'http_requests_total',
    help: 'Total number of HTTP requests',
    labelNames: ['method', 'route', 'status_code'],
});

// Tạo gauge cho active requests
const activeRequests = new client.Gauge({
    name: 'http_active_requests',
    help: 'Number of active HTTP requests',
});

// Tạo counter cho lỗi HTTP request
const httpRequestErrors = new client.Counter({
    name: 'http_request_errors_total',
    help: 'Total number of HTTP request errors',
    labelNames: ['method', 'route', 'error_type'],
});

export const metricsMiddleware = (req: Request, res: Response, next: NextFunction) => {
    // Tăng active requests
    activeRequests.inc();

    // Bắt đầu bấm giờ
    const start = process.hrtime.bigint();

    res.on('finish', () => {
        // Giảm active requests
        activeRequests.dec();

        // Tính toán thời lượng
        const end = process.hrtime.bigint();
        const durationInSeconds = Number(end - start) / 1e9;

        // Chuẩn hóa path để tránh high cardinality
        const route = normalizeRoutePath(req);
        const correlationId = getCorrelationId(req) || 'unknown';

        // Ghi nhận thời lượng
        httpRequestDurationSeconds
            .labels(req.method, route, res.statusCode.toString(), correlationId)
            .observe(durationInSeconds);

        // Tăng bộ đếm request
        httpRequestsTotal
            .labels(req.method, route, res.statusCode.toString())
            .inc();

        // Theo dõi lỗi
        if (res.statusCode >= 400) {
            const errorType = res.statusCode >= 500 ? 'server_error' : 'client_error';
            httpRequestErrors
                .labels(req.method, route, errorType)
                .inc();
        }
    });

    next();
};

// Chuẩn hóa route path để ngăn high cardinality metrics
function normalizeRoutePath(req: Request): string {
    if (req.route && req.route.path) {
        return req.route.path;
    }

    let path = req.path;
    // Thay thế UUIDs và numeric IDs bằng placeholders
    path = path.replace(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, ':uuid');
    path = path.replace(/\d+/g, ':id');

    return path;
}
```

### Luồng Thu Thập Metrics

Metrics được thu thập từ services và expose cho Prometheus để monitoring và alerting:

```mermaid
graph LR
    subgraph "Service Instance"
        App[Application]
        Middleware[Metrics Middleware]
        Registry[Prometheus Registry]
        Endpoint[/metrics Endpoint]
    end
    
    subgraph "Metrics Types"
        Counter[Counter<br/>http_requests_total]
        Gauge[Gauge<br/>active_users]
        Histogram[Histogram<br/>request_duration]
    end
    
    subgraph "Collection"
        Prometheus[Prometheus<br/>Scraper]
    end
    
    subgraph "Storage & Query"
        PromDB[(Prometheus<br/>Time Series DB)]
    end
    
    subgraph "Visualization"
        Grafana[Grafana<br/>Dashboards]
        Alerts[AlertManager<br/>Rules]
    end
    
    App -->|HTTP Request| Middleware
    Middleware -->|Record| Counter
    Middleware -->|Record| Histogram
    App -->|Update| Gauge
    
    Counter --> Registry
    Gauge --> Registry
    Histogram --> Registry
    Registry --> Endpoint
    
    Prometheus -->|Scrape every 15s| Endpoint
    Prometheus -->|Store| PromDB
    
    PromDB -->|Query| Grafana
    PromDB -->|Evaluate| Alerts
    Alerts -->|Trigger| Grafana
    
    style App fill:#e1f5ff
    style Prometheus fill:#ffe1e1
    style Grafana fill:#e1e1ff
```

### Distributed Tracing

Use OpenTelemetry for distributed tracing across services.

Sử dụng OpenTelemetry cho distributed tracing qua các services.

**Ví dụ từ codebase**: [`packages/tracing/src/index.ts`](../../../packages/tracing/src/index.ts)

```typescript
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { JaegerExporter } from '@opentelemetry/exporter-jaeger';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

export interface TracingConfig {
  serviceName: string;
  jaegerEndpoint?: string;
  enabled?: boolean;
}

export const initTracing = (config: TracingConfig): NodeSDK | null => {
  if (config.enabled === false) {
    return null;
  }

  // Tạo Jaeger exporter nếu endpoint được cung cấp
  const jaegerExporter = config.jaegerEndpoint
    ? new JaegerExporter({
        endpoint: config.jaegerEndpoint,
      })
    : undefined;

  // Khởi tạo OpenTelemetry NodeSDK với auto-instrumentations
  const sdk = new NodeSDK({
    resource: new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: config.serviceName,
    }),
    traceExporter: jaegerExporter,
    instrumentations: [getNodeAutoInstrumentations()],
  });

  // Khởi động tracing SDK
  sdk.start();

  return sdk;
};
```

**Cách sử dụng trong service**:

```typescript
// services/iam-service/src/main.ts
import { initTracing } from '@goodgo/tracing';

// Khởi tạo tracing
if (process.env.TRACING_ENABLED === 'true') {
  initTracing({
    serviceName: process.env.SERVICE_NAME || 'iam-service',
    jaegerEndpoint: process.env.JAEGER_ENDPOINT,
    enabled: true,
  });
}
```

### Luồng Distributed Tracing

Distributed tracing theo dõi requests qua nhiều services sử dụng OpenTelemetry:

```mermaid
sequenceDiagram
    participant Client
    participant Gateway as API Gateway
    participant ServiceA as Service A<br/>(User Service)
    participant ServiceB as Service B<br/>(Order Service)
    participant DB as Database
    participant Jaeger as Jaeger<br/>Collector
    
    Client->>Gateway: Request<br/>(Trace ID: abc123)
    Gateway->>Gateway: Create Root Span<br/>Span: gateway.request
    Gateway->>ServiceA: HTTP Call<br/>(Trace ID: abc123,<br/>Span ID: span-1)
    
    ServiceA->>ServiceA: Create Child Span<br/>Span: user.getById
    ServiceA->>DB: Query User<br/>(Trace ID: abc123,<br/>Span ID: span-2)
    DB-->>ServiceA: User Data
    ServiceA->>ServiceA: End Span span-2
    ServiceA->>ServiceB: HTTP Call<br/>(Trace ID: abc123,<br/>Span ID: span-3)
    
    ServiceB->>ServiceB: Create Child Span<br/>Span: order.getByUserId
    ServiceB->>DB: Query Orders<br/>(Trace ID: abc123,<br/>Span ID: span-4)
    DB-->>ServiceB: Orders Data
    ServiceB->>ServiceB: End Span span-4
    ServiceB->>ServiceB: End Span span-3
    ServiceB-->>ServiceA: Response
    ServiceA->>ServiceA: End Span span-1
    ServiceA-->>Gateway: Response
    Gateway->>Gateway: End Span gateway.request
    Gateway-->>Client: Final Response
    
    ServiceA->>Jaeger: Export Spans<br/>(Trace ID: abc123)
    ServiceB->>Jaeger: Export Spans<br/>(Trace ID: abc123)
    Gateway->>Jaeger: Export Spans<br/>(Trace ID: abc123)
    
    Note over Jaeger: All spans linked by<br/>Trace ID: abc123
```

### Kiểm Tra Sức Khỏe

Implement liveness and readiness probes for Kubernetes.

Triển khai liveness và readiness probes cho Kubernetes.

**Ví dụ từ codebase**: [`services/iam-service/src/modules/health/health.controller.ts`](../../../services/iam-service/src/modules/health/health.controller.ts)

```typescript
import { Request, Response } from 'express';
import { prisma } from '../../config/database.config';
import { ApiResponse } from '@goodgo/types';

export class HealthController {
  /**
Basic liveness probe
   * VI: Kiểm tra liveness cơ bản
   */
  health = async (_req: Request, res: Response): Promise<void> => {
    const response: ApiResponse<{ status: string; timestamp: string }> = {
      success: true,
      data: {
        status: 'ok',
        timestamp: new Date().toISOString(),
      },
      timestamp: new Date().toISOString(),
    };

    res.json(response);
  };

  /**
Readiness probe (checks database connection)
   * VI: Kiểm tra readiness (kiểm tra kết nối database)
   */
  ready = async (_req: Request, res: Response): Promise<void> => {
    try {
      // Kiểm tra kết nối database
      await prisma.$queryRaw`SELECT 1`;
      res.json({
        success: true,
        data: { status: 'ready' },
        timestamp: new Date().toISOString(),
      });
    } catch (error) {
      // Trả về 503 nếu database chưa sẵn sàng
      res.status(503).json({
        success: false,
        error: {
          code: 'HEALTH_001',
          message: 'Service not ready',
        },
        timestamp: new Date().toISOString(),
      });
    }
  };

  /**
Alias for health check
   * VI: Alias cho kiểm tra sức khỏe
   */
  live = async (_req: Request, res: Response): Promise<void> => {
    res.json({
      success: true,
      data: { status: 'live' },
      timestamp: new Date().toISOString(),
    });
  };
}
```

## Thực Hành Tốt Nhất

### Logging

- Use structured logging (JSON format)
- Include correlation IDs for request tracing
- Log at appropriate levels (ERROR, WARN, INFO, DEBUG)
- Avoid logging sensitive data (passwords, tokens, PII)
- Use consistent log format across services

**VI**:
- Sử dụng structured logging (định dạng JSON)
- Bao gồm correlation IDs để theo dõi request
- Log ở mức độ phù hợp (ERROR, WARN, INFO, DEBUG)
- Tránh log dữ liệu nhạy cảm (mật khẩu, tokens, PII)
- Sử dụng format log nhất quán giữa các services

### Metrics

- Use standard metric types (Counter, Gauge, Histogram)
- Keep cardinality low (avoid high-cardinality labels)
- Define SLIs and SLOs for critical paths
- Monitor business metrics, not just technical ones
- Normalize route paths to prevent high cardinality

**VI**:
- Sử dụng các loại metric chuẩn (Counter, Gauge, Histogram)
- Giữ cardinality thấp (tránh high-cardinality labels)
- Định nghĩa SLIs và SLOs cho các đường dẫn quan trọng
- Giám sát business metrics, không chỉ technical metrics
- Chuẩn hóa route paths để tránh high cardinality

### Tracing

- Add traces for critical operations
- Include relevant context in spans
- Sample appropriately to control costs
- Use distributed tracing for microservices
- Propagate correlation IDs across service boundaries

**VI**:
- Thêm traces cho các thao tác quan trọng
- Bao gồm context liên quan trong spans
- Sample phù hợp để kiểm soát chi phí
- Sử dụng distributed tracing cho microservices
- Truyền correlation IDs qua ranh giới service

### Cảnh Báo

- Alert on symptoms, not causes
- Include runbook links in alerts
- Avoid alert fatigue with proper thresholds
- Test alerting rules regularly
- Use correlation IDs in alert context

**VI**:
- Cảnh báo về triệu chứng, không phải nguyên nhân
- Bao gồm links runbook trong alerts
- Tránh alert fatigue với thresholds phù hợp
- Test alerting rules thường xuyên
- Sử dụng correlation IDs trong alert context

## Ví Dụ Từ Dự Án

### Triển Khai Logging

- **Request Logger**: [`services/iam-service/src/middlewares/logger.middleware.ts`](../../../services/iam-service/src/middlewares/logger.middleware.ts)
- **Correlation Middleware**: [`services/iam-service/src/middlewares/correlation.middleware.ts`](../../../services/iam-service/src/middlewares/correlation.middleware.ts)

### Triển Khai Metrics

- **Metrics Middleware**: [`services/iam-service/src/middlewares/metrics.middleware.ts`](../../../services/iam-service/src/middlewares/metrics.middleware.ts)
- **Metrics Endpoint**: Exposed at `/metrics` in all services

### Triển Khai Tracing

- **Tracing Package**: [`packages/tracing/src/index.ts`](../../../packages/tracing/src/index.ts)
- **Service Integration**: [`services/iam-service/src/main.ts`](../../../services/iam-service/src/main.ts)

### Health Checks

- **Health Controller**: [`services/iam-service/src/modules/health/health.controller.ts`](../../../services/iam-service/src/modules/health/health.controller.ts)

## Tham Khảo Nhanh

### Mức Độ Log

- `ERROR`: Errors that require immediate attention
- `WARN`: Warnings that may indicate issues
- `INFO`: Informational messages (default)
- `DEBUG`: Detailed debugging information

**VI**:
- `ERROR`: Lỗi cần chú ý ngay lập tức
- `WARN`: Cảnh báo có thể chỉ ra vấn đề
- `INFO`: Thông điệp thông tin (mặc định)
- `DEBUG`: Thông tin debug chi tiết

### Loại Metrics

- **Counter**: Monotonically increasing value (e.g., request count)
- **Gauge**: Value that can go up or down (e.g., active connections)
- **Histogram**: Distribution of values (e.g., request duration)

**VI**:
- **Counter**: Giá trị tăng đơn điệu (ví dụ: số lượng request)
- **Gauge**: Giá trị có thể tăng hoặc giảm (ví dụ: kết nối đang hoạt động)
- **Histogram**: Phân phối giá trị (ví dụ: thời lượng request)

### Endpoints Health Check

- `/health` or `/health/live`: Liveness probe (service is running)
- `/health/ready`: Readiness probe (service is ready to accept traffic)

**VI**:
- `/health` hoặc `/health/live`: Liveness probe (service đang chạy)
- `/health/ready`: Readiness probe (service sẵn sàng nhận traffic)

### Truy Vấn Prometheus

```promql
# Request rate
rate(http_requests_total[5m])

# Error rate
rate(http_requests_total{status_code=~"5.."}[5m])

# 95th percentile latency
histogram_quantile(0.95, http_request_duration_seconds)

# Active requests
http_active_requests
```

## Skills Liên Quan

- [Kubernetes Deployment](./deployment-kubernetes.md) - Để cấu hình health checks trong K8s
- [Security](./security.md) - Để logging và monitoring an toàn
- [Project Rules](./project-rules.md) - Cho cấu trúc và tiêu chuẩn service

## Tài Nguyên

### Tài Liệu Chính Thức

- [OpenTelemetry Documentation](https://opentelemetry.io/docs/)
- [Prometheus Documentation](https://prometheus.io/docs/)
- [Jaeger Documentation](https://www.jaegertracing.io/docs/)

### Tài Nguyên GoodGo

- [Observability Guide](../guides/observability.md)
- [Troubleshooting Guide](../guides/troubleshooting.md)
- [Logger Package](../../../packages/logger/README.md)
- [Tracing Package](../../../packages/tracing/README.md)
