# Observability Stack Guide

This guide explains how to use the observability stack (Grafana, Prometheus, Loki, Promtail) included in the infrastructure.

## Architecture Overview

The stack consists of the following components:

-   **Prometheus**: Collects metrics from services.
-   **Loki**: Collects logs.
-   **Promtail**: Scrapes logs from Docker containers and pushes them to Loki.
-   **Grafana**: Visualization dashboard for metrics (from Prometheus) and logs (from Loki).

## Getting Started

### Prerequisites

-   Docker and Docker Compose installed.
-   Existing `microservices-network` (created by the main application stack or manually).

### Starting the Stack

You can easily start the stack using the provided script:

```bash
./scripts/observability/start.sh
```

Or manually:

```bash
# Ensure network exists
docker network create microservices-network || true

cd infra/observability
docker-compose -f docker-compose.observability.yml up -d
```

Check if all containers are running:

```bash
docker ps
```

You should see `grafana`, `prometheus`, `loki`, and `promtail`.

## Accessing Services

| Service | URL | Credentials (if applicable) | Description |
| :--- | :--- | :--- | :--- |
| **Grafana** | [http://localhost:3001](http://localhost:3001) | `admin` / `admin` | Main dashboard for visualization. |
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | N/A | Raw metrics and target status. |
| **Loki** | [http://localhost:3100](http://localhost:3100) | N/A | Log aggregation API (no UI). |

## Using Grafana

1.  **Login**: Access [http://localhost:3001](http://localhost:3001) and login with `admin`/`admin`.
2.  **Explore Data**:
    -   Go to **Explore** (compass icon) in the sidebar.
    -   Select **Loki** from the datasource dropdown to search logs.
    -   Select **Prometheus** from the datasource dropdown to query metrics.

### Viewing Logs (Loki)

In the **Explore** view with **Loki** selected:

1.  Click **Label browser**.
2.  Select a label, e.g., `container`.
3.  Choose a specific container (e.g., `iam-service` or `traefik`).
4.  Click **Show logs**.

You can also write LogQL queries manually, for example:

```logql
{container="iam-service"}
```

### Viewing Metrics (Prometheus)

In the **Explore** view with **Prometheus** selected:

1.  Type a metric name in the query field (e.g., `up`, `container_memory_usage_bytes`).
2.  Click **Run query**.

## Configuration

-   **Prometheus**: Rules and targets are configured in `infra/observability/prometheus/prometheus.yml`.
-   **Promtail**: Log scraping rules are configured in `infra/observability/promtail/promtail-config.yml`.
-   **Grafana**: Datasources and dashboards provisioning are in `infra/observability/grafana/`.