goodgo-platform/docs/runbooks/sentry.md

Sentry Alert Routing — Runbook

Audience: SRE on-call & platform engineers. Scope: How Sentry P1/P2 alerts are wired into Slack & email for the GoodGo platform, and how to verify routing end-to-end. Related ticket: GOO-178 (gap surfaced by GOO-117 audit).

---

1. Sentry projects & SDK wiring

The SDK is initialised in three Next.js entrypoints (server, client, edge):

File	DSN env	Notes
apps/web/sentry.server.config.ts	SENTRY_DSN	Server-side errors, traces (tracesSampleRate=0.2 in prod).
apps/web/sentry.client.config.ts	NEXT_PUBLIC_SENTRY_DSN	Browser errors + session replay on error.
apps/web/sentry.edge.config.ts	SENTRY_DSN	Edge runtime.

Required env (see .env.example):

• SENTRY_DSN, NEXT_PUBLIC_SENTRY_DSN
• SENTRY_AUTH_TOKEN (CI source-map upload)
• SENTRY_ORG, SENTRY_PROJECT
• SENTRY_RELEASE / NEXT_PUBLIC_SENTRY_RELEASE (set per deploy by CI)

environment is set from NODE_ENV on every init call, so all events are tagged production / staging / development automatically.

---

2. Severity model

We classify Sentry events into two routing tiers using event tags. Tag severity is set either by the SDK (Sentry.setTag('severity', 'P1')) at capture time, or via Sentry Issue tagging rules based on event.level / fingerprint.

Tier	Trigger criteria	Routing target	SLA
P1	level:fatal OR tags[severity]=P1 OR fingerprint in critical group (payments, auth, data loss)	Slack #alerts-p1-sentry (immediate) + email to oncall@goodgo.vn	Page on-call within 5 min
P2	level:error AND environment:production AND not P1	Slack #alerts-p2-sentry (digest, every 30 min)	Triage within 4 business hours
Below P2 (warn / info / staging-only)	Everything else	No paging — visible only in Sentry UI dashboards	Best effort

---

3. Required Sentry integrations

Verify in Sentry → Settings → Integrations:

• Slack — workspace goodgo, channels #alerts-p1-sentry and #alerts-p2-sentry authorised.
• Email — built-in; oncall@goodgo.vn is a verified team mailing list and a member of the GoodGo team for the project.

If either integration is missing, install it before configuring rules below.

---

4. Alert rule configuration

Configure under Sentry → Alerts → Create Alert → Issues.

4.1 P1 — immediate page

• Name: P1 — immediate page
• Environment: production
• When: "An issue is first seen" OR "An event is captured"
• If (all):
  • event.level equals fatal
  • OR event.tags[severity] equals P1
• Then:
  • Send a Slack notification to #alerts-p1-sentry
  • Send an email notification to Team oncall
• Frequency: every 1 minute (no batching)
• Owner: SRE team

4.2 P2 — production errors digest

• Name: P2 — production errors
• Environment: production
• When: "An issue changes state from resolved to unresolved" OR "A new issue is created"
• If (all):
  • event.level equals error
  • event.tags[severity] does not equal P1
• Then:
  • Send a Slack notification to #alerts-p2-sentry
• Frequency: every 30 minutes (batched/digest)

4.3 Noise control

• Suppress alerts for environments other than production (a separate staging rule may post to #alerts-staging only).
• Use Inbound filters to drop browser extension errors, ResizeObserver loops, and known third-party noise.
• Per-issue Mute → ignore until for known-noisy fingerprints.

---

5. Test fire procedure (Sentry.captureException)

Use this whenever rules change or a quarterly verification is due.

5.1 Staging fire

1. Deploy the staging branch with SENTRY_DSN populated.
2. Trigger the dedicated test endpoint (or via curl in staging shell):

   // apps/web/app/api/_debug/sentry-test/route.ts (staging only — guard with env)
   import * as Sentry from '@sentry/nextjs';
   export async function GET() {
     Sentry.captureException(new Error('GOO-178 routing test — P1'), {
       tags: { severity: 'P1', source: 'manual_routing_test' },
     });
     return Response.json({ ok: true });
   }
   

   curl -fsS https://staging.goodgo.vn/api/_debug/sentry-test
   

3. Expected:
   • Event appears in Sentry within ~30s tagged severity:P1, source:manual_routing_test.
   • Slack #alerts-p1-sentry receives the notification.
   • Email lands at oncall@goodgo.vn within ~2 min.
4. Repeat without the severity:P1 tag and with level:error to validate the P2 batched route (expect Slack message in next 30-min digest).

5.2 CLI alternative (no deploy)

SENTRY_DSN=$STAGING_SENTRY_DSN npx -y @sentry/cli send-event \
  --message "GOO-178 routing test (P1)" \
  --level fatal \
  --tag severity:P1 \
  --tag source:manual_routing_test \
  --release "$(git rev-parse --short HEAD)"

5.3 Cleanup

• Resolve the test issue in Sentry.
• Add source:manual_routing_test to the inbound filter ignore list if it generates additional noise.

---

6. Verification checklist (run quarterly)

• Slack integration shows "Connected" in Sentry → Integrations.
• Email integration: open one P1 alert, confirm oncall@goodgo.vn received it.
• Both rules listed under Alerts with correct environment/scope.
• Test fire produces messages in #alerts-p1-sentry AND #alerts-p2-sentry per tier.
• On-call rotation in PagerDuty / Slack matches Sentry team membership.
• Inbound filters review — drop new noise sources surfaced since last review.

Record verification results as a comment on a tracking issue and update the last verified date in section 7.

---

7. Change log

Date	Author	Change
2026-04-23	SRE Engineer (GOO-178)	Initial runbook documenting P1/P2 routing model, alert rule definitions, and test fire procedure.

---

8. Outstanding

• Sentry org/project access not yet provisioned for the agent. Live verification of integrations + a real test fire is blocked until SENTRY_AUTH_TOKEN, SENTRY_ORG, SENTRY_PROJECT are provided to staging and to the SRE agent. See GOO-178 for the open blocker.