---
trigger: always_on
---

# Observability & Monitoring Patterns

## When to Use This Skill

Use this skill when:
- Setting up logging infrastructure
- Implementing metrics collection
- Adding distributed tracing
- Creating health check endpoints
- Setting up monitoring dashboards
- Debugging production issues
- Implementing alerting rules
- Analyzing performance bottlenecks

## Core Concepts

### Three Pillars of Observability
1. **Logs**: Event records for debugging
2. **Metrics**: Numerical measurements over time
3. **Traces**: Request flow across services

### Tech Stack
- **Logging**: Winston, Pino
- **Metrics**: Prometheus + Grafana
- **Tracing**: OpenTelemetry + Jaeger
- **APM**: DataDog or New Relic (optional)

## Key Patterns

### Structured Logging

```typescript
import winston from 'winston';

export const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  defaultMeta: { service: process.env.SERVICE_NAME }
});

// Request logging middleware
export const requestLogger = (req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    logger.info('HTTP Request', {
      method: req.method, url: req.url,
      status: res.statusCode, duration: Date.now() - start,
      correlationId: req.headers['x-correlation-id']
    });
  });
  next();
};
```

### Metrics Collection

```typescript
import { Counter, Histogram, Gauge } from 'prom-client';

export const httpRequestDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.1, 0.3, 0.5, 0.7, 1, 3, 5, 7, 10]
});

export const httpRequestTotal = new Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status']
});
```

### Distributed Tracing

```typescript
import { NodeSDK } from '@opentelemetry/sdk-node';
import { trace, SpanStatusCode } from '@opentelemetry/api';

export const tracedOperation = async (name: string, fn: Function) => {
  const span = trace.getTracer('app').startSpan(name);
  try {
    const result = await fn();
    span.setStatus({ code: SpanStatusCode.OK });
    return result;
  } catch (error) {
    span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
    throw error;
  } finally {
    span.end();
  }
};
```

### Health Checks

```typescript
// Liveness - is the service running?
app.get('/health/live', (req, res) => res.json({ status: 'ok' }));

// Readiness - is the service ready for traffic?
app.get('/health/ready', async (req, res) => {
  const dbOk = await prisma.$queryRaw`SELECT 1`.then(() => true).catch(() => false);
  const redisOk = await redis.ping().then(() => true).catch(() => false);
  const ready = dbOk && redisOk;
  res.status(ready ? 200 : 503).json({ ready, db: dbOk, redis: redisOk });
});
```

## Best Practices

- **Logging**: Use structured JSON format with correlation IDs
- **Metrics**: Use standard types (Counter, Gauge, Histogram) with low-cardinality labels
- **Tracing**: Add traces for critical operations, sample appropriately
- **Alerting**: Alert on symptoms, include runbook links, avoid alert fatigue
- **Security**: Never log sensitive data (passwords, tokens, PII)

## Common Mistakes

1. **Logging Sensitive Data**: Exposing PII in logs
   ```typescript
   // BAD: logger.info('User login', { email, password, token });
   // GOOD: logger.info('User login', { email, userId });
   ```

2. **High Cardinality Labels**: Too many metric label values
   ```typescript
   // BAD: httpRequests.labels(method, route, userId).inc();
   // GOOD: httpRequests.labels(method, route, statusCode).inc();
   ```

3. **No Correlation IDs**: Can't trace requests across services
   ```typescript
   // GOOD: Include correlationId in all logs
   logger.info('Processing', { correlationId: req.headers['x-correlation-id'] });
   ```

4. **Wrong Log Levels**: Using ERROR for non-errors
   ```typescript
   // BAD: logger.error('User not found');
   // GOOD: logger.info('User not found', { userId });
   ```

5. **No Health Checks**: Service status unknown
   ```typescript
   // GOOD: Implement both endpoints
   app.get('/health/live', livenessCheck);
   app.get('/health/ready', readinessCheck);
   ```

## Quick Reference

| Pillar | Tool | Endpoint |
|--------|------|----------|
| **Logs** | Winston/Loki | stdout -> Loki |
| **Metrics** | Prometheus | `/metrics` |
| **Traces** | Jaeger | `http://jaeger:14268` |

**Log Levels:**
| Level | When to Use |
|-------|-------------|
| `error` | Errors requiring attention |
| `warn` | Potential issues, degradation |
| `info` | Business events, state changes |
| `debug` | Development details |

**Essential Metrics:**
```typescript
rate(http_requests_total[5m])                           // Request rate
rate(http_requests_total{status=~"5.."}[5m])            // Error rate
histogram_quantile(0.95, http_request_duration_bucket)  // Latency p95
```

**Health Check Endpoints:**
| Endpoint | Purpose | Used By |
|----------|---------|---------|
| `/health/live` | Is process running? | K8s liveness probe |
| `/health/ready` | Ready for traffic? | K8s readiness probe |
| `/health` | Full status | Monitoring |

## Resources

- [OpenTelemetry](https://opentelemetry.io/docs/) - Distributed tracing standard
- [Prometheus](https://prometheus.io/docs/) - Metrics and alerting
- [Grafana](https://grafana.com/docs/) - Visualization
- [Detailed Code Examples](./references/REFERENCE.md)
- [Deployment Kubernetes](../deployment-kubernetes/SKILL.md) - K8s health probes
- [Resilience Patterns](../resilience-patterns/SKILL.md) - Circuit breaker metrics
