goodgo-platform/docs/runbooks/sentry.md

# Sentry Alert Routing — Runbook

> Audience: SRE on-call & platform engineers.
> Scope: How Sentry P1/P2 alerts are wired into Slack & email for the GoodGo platform, and how to verify routing end-to-end.
> Related ticket: [GOO-178](/GOO/issues/GOO-178) (gap surfaced by [GOO-117](/GOO/issues/GOO-117) audit).

---

## 1. Sentry projects & SDK wiring

The SDK is initialised in three Next.js entrypoints (server, client, edge):

| File | DSN env | Notes |
|------|---------|-------|
| `apps/web/sentry.server.config.ts` | `SENTRY_DSN` | Server-side errors, traces (`tracesSampleRate=0.2` in prod). |
| `apps/web/sentry.client.config.ts` | `NEXT_PUBLIC_SENTRY_DSN` | Browser errors + session replay on error. |
| `apps/web/sentry.edge.config.ts` | `SENTRY_DSN` | Edge runtime. |

Required env (see `.env.example`):

- `SENTRY_DSN`, `NEXT_PUBLIC_SENTRY_DSN`
- `SENTRY_AUTH_TOKEN` (CI source-map upload)
- `SENTRY_ORG`, `SENTRY_PROJECT`
- `SENTRY_RELEASE` / `NEXT_PUBLIC_SENTRY_RELEASE` (set per deploy by CI)

`environment` is set from `NODE_ENV` on every init call, so all events are tagged `production` / `staging` / `development` automatically.

---

## 2. Severity model

We classify Sentry events into two routing tiers using event tags. Tag `severity` is set either by the SDK (`Sentry.setTag('severity', 'P1')`) at capture time, or via Sentry **Issue tagging rules** based on event.level / fingerprint.

| Tier | Trigger criteria | Routing target | SLA |
|------|------------------|----------------|-----|
| **P1** | `level:fatal` OR `tags[severity]=P1` OR fingerprint in critical group (payments, auth, data loss) | Slack `#alerts-p1-sentry` (immediate) + email to `oncall@goodgo.vn` | Page on-call within 5 min |
| **P2** | `level:error` AND `environment:production` AND not P1 | Slack `#alerts-p2-sentry` (digest, every 30 min) | Triage within 4 business hours |
| Below P2 (warn / info / staging-only) | Everything else | No paging — visible only in Sentry UI dashboards | Best effort |

---

## 3. Required Sentry integrations

Verify in **Sentry → Settings → Integrations**:

- [ ] **Slack** — workspace `goodgo`, channels `#alerts-p1-sentry` and `#alerts-p2-sentry` authorised.
- [ ] **Email** — built-in; `oncall@goodgo.vn` is a verified team mailing list and a member of the GoodGo team for the project.

If either integration is missing, install it before configuring rules below.

---

## 4. Alert rule configuration

Configure under **Sentry → Alerts → Create Alert → Issues**.

### 4.1 P1 — immediate page

- **Name:** `P1 — immediate page`
- **Environment:** `production`
- **When:** "An issue is first seen" **OR** "An event is captured"
- **If (all):**
  - `event.level` equals `fatal`
  - **OR** `event.tags[severity]` equals `P1`
- **Then:**
  - Send a Slack notification to `#alerts-p1-sentry`
  - Send an email notification to Team `oncall`
- **Frequency:** every 1 minute (no batching)
- **Owner:** SRE team

### 4.2 P2 — production errors digest

- **Name:** `P2 — production errors`
- **Environment:** `production`
- **When:** "An issue changes state from resolved to unresolved" OR "A new issue is created"
- **If (all):**
  - `event.level` equals `error`
  - `event.tags[severity]` does not equal `P1`
- **Then:**
  - Send a Slack notification to `#alerts-p2-sentry`
- **Frequency:** every 30 minutes (batched/digest)

### 4.3 Noise control

- Suppress alerts for environments other than `production` (a separate `staging` rule may post to `#alerts-staging` only).
- Use **Inbound filters** to drop browser extension errors, ResizeObserver loops, and known third-party noise.
- Per-issue **Mute → ignore until** for known-noisy fingerprints.

---

## 5. Test fire procedure (`Sentry.captureException`)

Use this whenever rules change or a quarterly verification is due.

### 5.1 Staging fire

1. Deploy the staging branch with `SENTRY_DSN` populated.
2. Trigger the dedicated test endpoint (or via curl in staging shell):
   ```ts
   // apps/web/app/api/_debug/sentry-test/route.ts (staging only — guard with env)
   import * as Sentry from '@sentry/nextjs';
   export async function GET() {
     Sentry.captureException(new Error('GOO-178 routing test — P1'), {
       tags: { severity: 'P1', source: 'manual_routing_test' },
     });
     return Response.json({ ok: true });
   }
   ```
   ```bash
   curl -fsS https://staging.goodgo.vn/api/_debug/sentry-test
   ```
3. **Expected:**
   - Event appears in Sentry within ~30s tagged `severity:P1`, `source:manual_routing_test`.
   - Slack `#alerts-p1-sentry` receives the notification.
   - Email lands at `oncall@goodgo.vn` within ~2 min.
4. Repeat without the `severity:P1` tag and with `level:error` to validate the P2 batched route (expect Slack message in next 30-min digest).

### 5.2 CLI alternative (no deploy)

```bash
SENTRY_DSN=$STAGING_SENTRY_DSN npx -y @sentry/cli send-event \
  --message "GOO-178 routing test (P1)" \
  --level fatal \
  --tag severity:P1 \
  --tag source:manual_routing_test \
  --release "$(git rev-parse --short HEAD)"
```

### 5.3 Cleanup

- Resolve the test issue in Sentry.
- Add `source:manual_routing_test` to the inbound filter ignore list if it generates additional noise.

---

## 6. Verification checklist (run quarterly)

- [ ] Slack integration shows "Connected" in Sentry → Integrations.
- [ ] Email integration: open one P1 alert, confirm `oncall@goodgo.vn` received it.
- [ ] Both rules listed under Alerts with correct environment/scope.
- [ ] Test fire produces messages in `#alerts-p1-sentry` AND `#alerts-p2-sentry` per tier.
- [ ] On-call rotation in PagerDuty / Slack matches Sentry team membership.
- [ ] Inbound filters review — drop new noise sources surfaced since last review.

Record verification results as a comment on a tracking issue and update the **last verified** date in section 7.

---

## 7. Change log

| Date | Author | Change |
|------|--------|--------|
| 2026-04-23 | SRE Engineer ([GOO-178](/GOO/issues/GOO-178)) | Initial runbook documenting P1/P2 routing model, alert rule definitions, and test fire procedure. |

---

## 8. Outstanding

- **Sentry org/project access not yet provisioned for the agent.** Live verification of integrations + a real test fire is blocked until `SENTRY_AUTH_TOKEN`, `SENTRY_ORG`, `SENTRY_PROJECT` are provided to staging and to the SRE agent. See [GOO-178](/GOO/issues/GOO-178) for the open blocker.