pos-system/docs/en/guides/observability.md

Observability Stack Guide

This guide explains how to use the observability stack (Grafana, Prometheus, Loki, Promtail) included in the infrastructure.

Architecture Overview

Components

The stack consists of the following components:

• Prometheus: Metrics collection and storage
• Loki: Log aggregation system
• Promtail: Log collector agent
• Grafana: Unified visualization dashboard

Architecture Diagram

flowchart LR
    subgraph Services["Microservices"]
        IAM[IAM Service]
        USER[User Service]
        TRAEFIK[Traefik Gateway]
    end
    
    subgraph Collection["Data Collection"]
        PROM[Prometheus]
        PROMTAIL[Promtail]
    end
    
    subgraph Storage["Data Storage"]
        PROM_DB[(Prometheus DB)]
        LOKI_DB[(Loki DB)]
    end
    
    subgraph Visualization["Visualization"]
        GRAFANA[Grafana Dashboard]
    end
    
    IAM -->|Metrics| PROM
    USER -->|Metrics| PROM
    TRAEFIK -->|Metrics| PROM
    
    IAM -.->|Logs| PROMTAIL
    USER -.->|Logs| PROMTAIL
    TRAEFIK -.->|Logs| PROMTAIL
    
    PROM -->|Store| PROM_DB
    PROMTAIL -->|Push| LOKI_DB
    
    PROM_DB -->|Query| GRAFANA
    LOKI_DB -->|Query| GRAFANA
    
    style Services fill:#2d3748
    style Collection fill:#2c5282
    style Storage fill:#2f855a
    style Visualization fill:#744210
    style IAM fill:#4a5568
    style USER fill:#4a5568
    style TRAEFIK fill:#4a5568
    style PROM fill:#3182ce
    style PROMTAIL fill:#3182ce
    style PROM_DB fill:#38a169
    style LOKI_DB fill:#38a169
    style GRAFANA fill:#d69e2e

Data Flow

sequenceDiagram
    participant S as Service
    participant PT as Promtail
    participant P as Prometheus
    participant L as Loki
    participant G as Grafana
    
    Note over S,G: Metrics Flow
    S->>P: Expose /metrics endpoint
    P->>P: Scrape metrics (15s interval)
    G->>P: Query PromQL
    P-->>G: Return metrics data
    
    Note over S,G: Logs Flow
    S->>PT: Write logs to stdout
    PT->>PT: Parse & Label logs
    PT->>L: Push logs via HTTP
    G->>L: Query LogQL
    L-->>G: Return log data

Getting Started

Prerequisites

• Docker and Docker Compose installed
• Existing microservices-network (created by the main application stack or manually)

Starting the Stack

You can easily start the stack using the provided script:

./scripts/observability/start.sh

Or manually:

docker network create microservices-network || true

cd infra/observability
docker-compose -f docker-compose.observability.yml up -d

Check if all containers are running:

docker ps

You should see grafana, prometheus, loki, and promtail.

Accessing Services

Service	URL	Credentials	Description
Grafana	http://localhost:3001	admin / admin	Main dashboard for visualization
Prometheus	http://localhost:9090	N/A	Raw metrics and target status
Loki	http://localhost:3100	N/A	Log aggregation API (no UI)

Using Grafana

Initial Setup

1. Login: Access http://localhost:3001 and login with admin/admin
2. Change Password: You'll be prompted to change the default password (recommended)
3. Verify Datasources:
   • Navigate to Configuration → Data Sources
   • Ensure both Prometheus and Loki are connected

Exploring Data

Go to Explore (compass icon) in the sidebar:

• Select Loki from the datasource dropdown to search logs
• Select Prometheus from the datasource dropdown to query metrics

Viewing Logs (Loki)

In the Explore view with Loki selected:

1. Click Label browser
2. Select a label, e.g., container
3. Choose a specific container (e.g., iam-service or traefik)
4. Click Show logs

LogQL Query Examples:

{container="iam-service"}
{container="iam-service"} |= "error"
{container="iam-service"} |= "error" | json

Viewing Metrics (Prometheus)

In the Explore view with Prometheus selected:

1. Type a metric name in the query field (e.g., up, container_memory_usage_bytes)
2. Click Run query

PromQL Query Examples:

up

rate(http_requests_total[5m])

container_memory_usage_bytes{container="iam-service"}

Configuration

File Locations

• Prometheus: infra/observability/prometheus/prometheus.yml
• Promtail: infra/observability/promtail/promtail-config.yml
• Grafana: infra/observability/grafana/

Custom Metrics

To expose custom metrics from your service:

import { Counter, Histogram } from 'prom-client';

const requestCounter = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status']
});

const requestDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route']
});

Custom Dashboards

Create custom dashboards in Grafana:

1. Click + → Dashboard
2. Add Panel
3. Configure query (Prometheus or Loki)
4. Save dashboard

Color Palette Reference

Diagrams use a dark color palette for better readability:

Component Type	Fill Color	Stroke Color	Purpose
🚀 Services	#e94560	#c81e3b	Microservices (red)
📊 Collectors	#f39c12	#d68910	Data collection (orange)
💾 Storage	#3498db	#2874a6	Storage (blue)
📊 Visualization	#9b59b6	#7d3c98	Visualization (purple)
📦 Subgraphs	#1a1a2e - #533483	#16213e - #0f3460	Logical groups

All text uses color:#ffffff (white) for readability on dark backgrounds

Quick Tips

Mermaid Common Issues

✅ DO:

• Use flowchart LR for left-to-right flow
• Use sequenceDiagram for time-based interactions
• Apply dark colors for better contrast
• Use descriptive node IDs

❌ DON'T:

• Mix graph and flowchart syntax
• Use special characters in node IDs without quotes
• Forget closing brackets for subgraphs

LogQL Quick Reference

{label="value"}
{label="value"} |= "search"
{label="value"} |= "error" | json
{label="value"} | logfmt

PromQL Quick Reference

metric_name
metric_name{label="value"}
rate(metric_name[5m])
sum(metric_name) by (label)

Visual Indicators

• 📊 Metrics: Numerical time-series data
• 📝 Logs: Text-based event records
• 🎯 Query: Search/filter operations
• 🔍 Explore: Investigation interface
• 📈 Dashboard: Pre-configured visualizations

Troubleshooting

Common Issues

Issue	Symptoms	Solution
⚠️ No logs visible	Grafana Explore shows no logs	Check Promtail is running: docker ps | grep promtail
📊 Missing metrics	Services don't appear in Prometheus targets	Check service /metrics endpoint
🔴 Container won't start	docker ps doesn't show container	View logs: docker-compose logs <service-name>
🌐 Network issue	Services can't connect	Create network: docker network create microservices-network

Logs Not Appearing in Loki

1. Check Promtail logs: docker logs promtail
2. Verify container labels are correct
3. Ensure services are on microservices-network

Metrics Not Appearing in Prometheus

1. Check Prometheus targets: http://localhost:9090/targets
2. Verify service exposes /metrics endpoint
3. Check Prometheus scrape config

Grafana Shows "No Data"

1. Verify datasource connection (Configuration → Data Sources)
2. Check time range in query
3. Ensure data exists in Prometheus/Loki