726 lines
23 KiB
Markdown
726 lines
23 KiB
Markdown
# Data Consistency Patterns / Patterns Đồng bộ Dữ liệu
|
|
|
|
> **EN**: Patterns for maintaining data consistency in distributed microservices architecture
|
|
> **VI**: Các patterns để duy trì tính nhất quán dữ liệu trong kiến trúc microservices phân tán
|
|
|
|
## Overview Diagram / Sơ đồ Tổng quan
|
|
|
|
```mermaid
|
|
graph TD
|
|
subgraph "Consistency Patterns"
|
|
Saga[Saga Pattern<br/>Distributed Transactions]
|
|
Outbox[Outbox Pattern<br/>Reliable Events]
|
|
Idempotency[Idempotency<br/>Retry Safety]
|
|
OptimisticLock[Optimistic Locking<br/>Concurrent Updates]
|
|
CQRS[CQRS<br/>Read/Write Separation]
|
|
end
|
|
|
|
Service1[Service A] --> Saga
|
|
Service2[Service B] --> Outbox
|
|
Service3[Service C] --> Idempotency
|
|
|
|
Saga --> EventualConsistency[Eventual Consistency]
|
|
Outbox --> EventualConsistency
|
|
Idempotency --> EventualConsistency
|
|
OptimisticLock --> StrongConsistency[Strong Consistency]
|
|
CQRS --> EventualConsistency
|
|
|
|
style Saga fill:#e1f5ff
|
|
style Outbox fill:#fff4e1
|
|
style Idempotency fill:#f0e1ff
|
|
style CQRS fill:#d4edda
|
|
```
|
|
|
|
## Architecture Description / Mô tả Kiến trúc
|
|
|
|
### EN: Architecture Overview
|
|
|
|
GoodGo platform uses multiple consistency patterns to handle distributed data:
|
|
|
|
**Core Challenges**:
|
|
- No distributed transactions (2PC too slow)
|
|
- Services own their data (database per service)
|
|
- Network failures can cause partial completion
|
|
- Need to maintain data integrity across services
|
|
|
|
**Pattern Selection**:
|
|
- **Saga**: For multi-service workflows
|
|
- **Outbox**: For guaranteed event publishing
|
|
- **Idempotency**: For safe retries
|
|
- **Optimistic Locking**: For concurrent updates
|
|
- **CQRS**: For read/write optimization
|
|
|
|
### VI: Tổng quan Kiến trúc
|
|
|
|
Nền tảng GoodGo sử dụng nhiều consistency patterns để xử lý dữ liệu phân tán:
|
|
|
|
**Thách thức Cốt lõi**:
|
|
- Không có distributed transactions (2PC quá chậm)
|
|
- Services sở hữu dữ liệu riêng (database per service)
|
|
- Network failures có thể gây partial completion
|
|
- Cần maintain data integrity giữa các services
|
|
|
|
**Lựa chọn Pattern**:
|
|
- **Saga**: Cho workflows nhiều services
|
|
- **Outbox**: Cho event publishing đảm bảo
|
|
- **Idempotency**: Cho retries an toàn
|
|
- **Optimistic Locking**: Cho concurrent updates
|
|
- **CQRS**: Cho tối ưu read/write
|
|
|
|
## System Context / Bối cảnh Hệ thống
|
|
|
|
```mermaid
|
|
C4Context
|
|
title System Context for Data Consistency in GoodGo Platform
|
|
|
|
Person(user, "User", "End user performing actions")
|
|
|
|
System_Boundary(goodgo, "GoodGo Microservices") {
|
|
System(order_service, "Order Service", "Manages orders with Saga")
|
|
System(payment_service, "Payment Service", "Processes payments")
|
|
System(inventory_service, "Inventory Service", "Manages stock")
|
|
System(saga_orchestrator, "Saga Orchestrator", "Coordinates distributed transactions")
|
|
System(outbox_processor, "Outbox Processor", "Publishes events reliably")
|
|
}
|
|
|
|
System_Ext(db_order, "Order DB", "PostgreSQL with Outbox table")
|
|
System_Ext(db_payment, "Payment DB", "PostgreSQL with version field")
|
|
System_Ext(db_inventory, "Inventory DB", "PostgreSQL")
|
|
System_Ext(kafka, "Event Bus", "Kafka - Event streaming")
|
|
System_Ext(redis, "Cache", "Redis - Idempotency keys")
|
|
|
|
Rel(user, order_service, "Places order", "HTTPS")
|
|
Rel(order_service, saga_orchestrator, "Starts saga", "Internal")
|
|
Rel(saga_orchestrator, payment_service, "Process payment", "HTTP")
|
|
Rel(saga_orchestrator, inventory_service, "Reserve stock", "HTTP")
|
|
|
|
Rel(order_service, db_order, "Writes + Outbox", "SQL")
|
|
Rel(payment_service, db_payment, "Updates with version", "SQL")
|
|
Rel(inventory_service, db_inventory, "Reads/Writes", "SQL")
|
|
|
|
Rel(outbox_processor, db_order, "Polls outbox", "SQL")
|
|
Rel(outbox_processor, kafka, "Publishes events", "Kafka Protocol")
|
|
Rel(order_service, redis, "Checks idempotency key", "Redis Protocol")
|
|
|
|
UpdateRelStyle(saga_orchestrator, payment_service, $lineColor="red", $textColor="red")
|
|
UpdateRelStyle(saga_orchestrator, inventory_service, $lineColor="red", $textColor="red")
|
|
```
|
|
|
|
**EN**: The GoodGo platform uses a database-per-service architecture where each service owns its data. Data consistency across services is achieved through patterns like Saga (for coordinated workflows), Outbox (for reliable event publishing), Idempotency (for safe retries), and Optimistic Locking (for concurrent updates). These patterns enable eventual consistency while maintaining data integrity.
|
|
|
|
**VI**: Nền tảng GoodGo sử dụng kiến trúc database-per-service nơi mỗi service sở hữu dữ liệu riêng. Tính nhất quán dữ liệu giữa các services đạt được thông qua các patterns như Saga (cho workflows phối hợp), Outbox (cho event publishing đáng tin cậy), Idempotency (cho retries an toàn), và Optimistic Locking (cho concurrent updates). Các patterns này cho phép eventual consistency đồng thời duy trì data integrity.
|
|
|
|
## Saga Pattern / Pattern Saga
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Orchestrator
|
|
participant OrderService
|
|
participant PaymentService
|
|
participant InventoryService
|
|
|
|
Orchestrator->>OrderService: 1. Create Order
|
|
OrderService-->>Orchestrator: Order Created
|
|
|
|
Orchestrator->>PaymentService: 2. Process Payment
|
|
PaymentService-->>Orchestrator: Payment Success
|
|
|
|
Orchestrator->>InventoryService: 3. Reserve Inventory
|
|
|
|
alt Inventory Reserved
|
|
InventoryService-->>Orchestrator: Success
|
|
Orchestrator->>Orchestrator: Complete Saga ✓
|
|
else Inventory Failed
|
|
InventoryService-->>Orchestrator: Failed ✗
|
|
Orchestrator->>PaymentService: Compensate: Refund
|
|
PaymentService-->>Orchestrator: Refunded
|
|
Orchestrator->>OrderService: Compensate: Cancel Order
|
|
OrderService-->>Orchestrator: Cancelled
|
|
end
|
|
```
|
|
|
|
**EN Description**: Saga manages distributed transactions as sequence of local transactions with compensation.
|
|
|
|
**VI Mô tả**: Saga quản lý distributed transactions dưới dạng chuỗi local transactions với compensation.
|
|
|
|
**Implementation**:
|
|
```typescript
|
|
// Saga orchestrator
|
|
class OrderSaga {
|
|
async execute(orderData: OrderData): Promise<void> {
|
|
const sagaContext = {
|
|
orderId: null,
|
|
paymentId: null,
|
|
inventoryId: null
|
|
};
|
|
|
|
try {
|
|
// Step 1: Create order
|
|
sagaContext.orderId = await orderService.create(orderData);
|
|
|
|
// Step 2: Process payment
|
|
sagaContext.paymentId = await paymentService.process(orderData.payment);
|
|
|
|
// Step 3: Reserve inventory
|
|
sagaContext.inventoryId = await inventoryService.reserve(orderData.items);
|
|
|
|
// All success - commit
|
|
await this.completeSaga(sagaContext);
|
|
} catch (error) {
|
|
// Compensate in reverse order
|
|
await this.compensate(sagaContext, error);
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
private async compensate(context: SagaContext, error: Error): Promise<void> {
|
|
if (context.inventoryId) {
|
|
await inventoryService.release(context.inventoryId);
|
|
}
|
|
if (context.paymentId) {
|
|
await paymentService.refund(context.paymentId);
|
|
}
|
|
if (context.orderId) {
|
|
await orderService.cancel(context.orderId);
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Outbox Pattern / Pattern Outbox
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Service
|
|
participant DB as Database
|
|
participant OutboxTable as Outbox Table
|
|
participant Processor as Outbox Processor
|
|
participant Kafka
|
|
|
|
Service->>DB: Begin Transaction
|
|
Service->>DB: Update Business Data
|
|
Service->>OutboxTable: Insert Event
|
|
Service->>DB: Commit Transaction
|
|
|
|
loop Every 5 seconds
|
|
Processor->>OutboxTable: SELECT unpublished events
|
|
OutboxTable-->>Processor: Events
|
|
Processor->>Kafka: Publish Events
|
|
Kafka-->>Processor: Ack
|
|
Processor->>OutboxTable: Mark as published
|
|
end
|
|
```
|
|
|
|
**EN**: Guarantees event publishing by storing events in database within same transaction as business data.
|
|
|
|
**VI**: Đảm bảo event publishing bằng cách lưu events trong database cùng transaction với business data.
|
|
|
|
**Implementation**:
|
|
```typescript
|
|
// Store event in outbox
|
|
async createUser(userData: CreateUserDto): Promise<User> {
|
|
return await prisma.$transaction(async (tx) => {
|
|
// Business operation
|
|
const user = await tx.user.create({ data: userData });
|
|
|
|
// Store event in outbox (same transaction)
|
|
await tx.outbox.create({
|
|
data: {
|
|
aggregateId: user.id,
|
|
aggregateType: 'User',
|
|
eventType: 'user.created.v1',
|
|
payload: JSON.stringify(user),
|
|
createdAt: new Date()
|
|
}
|
|
});
|
|
|
|
return user;
|
|
});
|
|
}
|
|
|
|
// Outbox processor (runs periodically)
|
|
async processOutbox(): Promise<void> {
|
|
const events = await prisma.outbox.findMany({
|
|
where: { publishedAt: null },
|
|
take: 100
|
|
});
|
|
|
|
for (const event of events) {
|
|
try {
|
|
await kafkaProducer.send({
|
|
topic: event.eventType,
|
|
messages: [{ value: event.payload }]
|
|
});
|
|
|
|
await prisma.outbox.update({
|
|
where: { id: event.id },
|
|
data: { publishedAt: new Date() }
|
|
});
|
|
} catch (error) {
|
|
logger.error('Failed to publish event', { event, error });
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Idempotency Pattern / Pattern Idempotency
|
|
|
|
```mermaid
|
|
graph LR
|
|
Request1[Request with<br/>Idempotency Key]
|
|
Request2[Retry with<br/>Same Key]
|
|
|
|
Request1 --> Check{Key Exists?}
|
|
Check -->|No| Process[Process Request]
|
|
Check -->|Yes| Return[Return Cached Result]
|
|
|
|
Process --> Store[Store Result<br/>with Key]
|
|
Store --> Response1[Response]
|
|
|
|
Request2 --> Check
|
|
Return --> Response2[Same Response]
|
|
|
|
style Check fill:#fff3cd
|
|
style Store fill:#d4edda
|
|
```
|
|
|
|
**EN**: Ensures operations can be safely retried without side effects by using idempotency keys.
|
|
|
|
**VI**: Đảm bảo operations có thể retry an toàn mà không có side effects bằng cách sử dụng idempotency keys.
|
|
|
|
**Implementation**:
|
|
```typescript
|
|
// Idempotency middleware
|
|
async function idempotentOperation<T>(
|
|
key: string,
|
|
operation: () => Promise<T>,
|
|
ttl: number = 86400 // 24 hours
|
|
): Promise<T> {
|
|
// Check if already processed
|
|
const cached = await redis.get(`idempotency:${key}`);
|
|
if (cached) {
|
|
return JSON.parse(cached);
|
|
}
|
|
|
|
// Process operation
|
|
const result = await operation();
|
|
|
|
// Store result
|
|
await redis.setex(`idempotency:${key}`, ttl, JSON.stringify(result));
|
|
|
|
return result;
|
|
}
|
|
|
|
// Usage in controller
|
|
async createPayment(req: Request, res: Response): Promise<void> {
|
|
const idempotencyKey = req.headers['idempotency-key'] as string;
|
|
|
|
if (!idempotencyKey) {
|
|
return res.status(400).json({ error: 'Idempotency-Key header required' });
|
|
}
|
|
|
|
const result = await idempotentOperation(
|
|
idempotencyKey,
|
|
() => paymentService.process(req.body)
|
|
);
|
|
|
|
res.json({ success: true, data: result });
|
|
}
|
|
```
|
|
|
|
## Optimistic Locking / Khóa Lạc quan
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant User1
|
|
participant User2
|
|
participant Service
|
|
participant DB
|
|
|
|
User1->>Service: Read (version=1)
|
|
User2->>Service: Read (version=1)
|
|
|
|
User1->>Service: Update (version=1)
|
|
Service->>DB: UPDATE WHERE version=1
|
|
DB-->>Service: Success, version→2
|
|
Service-->>User1: Success
|
|
|
|
User2->>Service: Update (version=1)
|
|
Service->>DB: UPDATE WHERE version=1
|
|
DB-->>Service: No rows updated
|
|
Service-->>User2: Conflict - version mismatch
|
|
User2->>Service: Read (version=2)
|
|
User2->>Service: Update (version=2)
|
|
Service-->>User2: Success
|
|
```
|
|
|
|
**EN**: Prevents lost updates by checking version on update.
|
|
|
|
**VI**: Ngăn chặn lost updates bằng cách kiểm tra version khi update.
|
|
|
|
**Implementation**:
|
|
```prisma
|
|
// Prisma schema
|
|
model User {
|
|
id String @id @default(cuid())
|
|
email String @unique
|
|
name String
|
|
version Int @default(1) // Version field
|
|
}
|
|
```
|
|
|
|
```typescript
|
|
// Update with optimistic locking
|
|
async updateUser(userId: string, data: UpdateUserDto, currentVersion: number): Promise<User> {
|
|
const result = await prisma.user.updateMany({
|
|
where: {
|
|
id: userId,
|
|
version: currentVersion // Check version
|
|
},
|
|
data: {
|
|
...data,
|
|
version: { increment: 1 } // Increment version
|
|
}
|
|
});
|
|
|
|
if (result.count === 0) {
|
|
throw new ConflictError('Version mismatch - data was modified by another user');
|
|
}
|
|
|
|
return await prisma.user.findUnique({ where: { id: userId } });
|
|
}
|
|
```
|
|
|
|
## CQRS Pattern
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph "Write Side"
|
|
Command[Command] --> WriteModel[Write Model<br/>Normalized]
|
|
WriteModel --> Events[Domain Events]
|
|
end
|
|
|
|
subgraph "Read Side"
|
|
Events --> Projection[Event Projection]
|
|
Projection --> ReadModel[Read Model<br/>Denormalized]
|
|
Query[Query] --> ReadModel
|
|
end
|
|
|
|
WriteModel --> DB1[(Write DB)]
|
|
ReadModel --> DB2[(Read DB<br/>Optimized)]
|
|
|
|
style WriteModel fill:#f0e1ff
|
|
style ReadModel fill:#d4edda
|
|
```
|
|
|
|
**EN**: Separates read and write models for optimal performance.
|
|
|
|
**VI**: Tách biệt read và write models để tối ưu hiệu suất.
|
|
|
|
## Performance Characteristics / Đặc điểm Hiệu suất
|
|
|
|
**EN**: Performance metrics and optimization strategies for data consistency patterns.
|
|
|
|
**VI**: Chỉ số hiệu suất và chiến lược tối ưu cho patterns đồng bộ dữ liệu.
|
|
|
|
| Pattern / Pattern | Latency Impact / Tác động Độ trễ | Throughput / Thông lượng | Notes / Ghi chú |
|
|
|-------------------|----------------------------------|--------------------------|-----------------|
|
|
| **Saga Execution / Thực thi Saga** | 500ms - 2s | 100-500 sagas/s | Depends on number of steps and compensation |
|
|
| **Outbox Processing / Xử lý Outbox** | < 100ms | 10,000 events/s | Async processing, minimal user impact |
|
|
| **Idempotency Check / Kiểm tra Idempotency** | < 10ms | 50,000 checks/s | Redis lookup, very fast |
|
|
| **Optimistic Lock Update / Cập nhật Optimistic Lock** | < 50ms | 5,000 updates/s | Single DB operation with version check |
|
|
| **CQRS Projection / CQRS Projection** | 100ms - 1s | 1,000 events/s | Event processing to read model |
|
|
| **Compensation Execution / Thực thi Compensation** | 200ms - 1s | Varies | Rollback operations in saga |
|
|
|
|
### Performance Optimization Strategies / Chiến lược Tối ưu Hiệu suất
|
|
|
|
**Saga Pattern**:
|
|
- Minimize number of steps (< 5 steps ideal)
|
|
- Parallel execution where possible
|
|
- Cache service responses
|
|
- Set appropriate timeouts (30s default)
|
|
|
|
**Outbox Pattern**:
|
|
- Batch process outbox events (100-500 per batch)
|
|
- Index `publishedAt` column for performance
|
|
- Archive processed events periodically
|
|
- Use connection pooling for Kafka
|
|
|
|
**Idempotency**:
|
|
- Use Redis for fast key lookups
|
|
- Set TTL to 24-48 hours
|
|
- Hash long idempotency keys
|
|
- Clean expired keys regularly
|
|
|
|
**Optimistic Locking**:
|
|
- Works best for low-contention scenarios
|
|
- Implement retry with exponential backoff
|
|
- Monitor conflict rates (should be < 5%)
|
|
- Consider pessimistic locking if conflicts > 10%
|
|
|
|
## Security Considerations / Cân nhắc Bảo mật
|
|
|
|
**EN**: Security measures for protecting data consistency operations.
|
|
|
|
**VI**: Biện pháp bảo mật để bảo vệ các operations đồng bộ dữ liệu.
|
|
|
|
### Saga Security / Bảo mật Saga
|
|
|
|
**Compensation Protection / Bảo vệ Compensation**:
|
|
- Validate saga execution permissions at each step
|
|
- Encrypt sensitive data in saga context
|
|
- Log all saga executions for audit
|
|
- Implement timeout to prevent hanging sagas
|
|
|
|
```typescript
|
|
// EN: Secure saga context
|
|
// VI: Saga context bảo mật
|
|
interface SecureSagaContext {
|
|
sagaId: string;
|
|
userId: string; // EN: User who initiated / VI: User khởi tạo
|
|
permissions: string[]; // EN: Required permissions / VI: Quyền yêu cầu
|
|
encryptedData: string; // EN: Encrypted sensitive data / VI: Dữ liệu nhạy cảm đã mã hóa
|
|
auditLog: AuditEntry[]; // EN: Audit trail / VI: Audit trail
|
|
}
|
|
```
|
|
|
|
### Outbox Security / Bảo mật Outbox
|
|
|
|
**Event Payload Encryption / Mã hóa Event Payload**:
|
|
- Encrypt PII (Personally Identifiable Information) before storing in outbox
|
|
- Use AES-256-GCM for event payload encryption
|
|
- Decrypt only when publishing to Kafka
|
|
- Rotate encryption keys quarterly
|
|
|
|
**Access Control / Kiểm soát Truy cập**:
|
|
- Restrict outbox table access to outbox processor only
|
|
- Use database roles and permissions
|
|
- Monitor outbox table access patterns
|
|
|
|
### Idempotency Security / Bảo mật Idempotency
|
|
|
|
**Key Security / Bảo mật Key**:
|
|
- Use cryptographic hashing for idempotency keys (SHA-256)
|
|
- Include user context in key generation
|
|
- Validate key ownership before processing
|
|
- Clear keys on user logout for sensitive operations
|
|
|
|
```typescript
|
|
// EN: Secure idempotency key generation
|
|
// VI: Tạo idempotency key bảo mật
|
|
function generateIdempotencyKey(
|
|
operation: string,
|
|
userId: string,
|
|
data: any
|
|
): string {
|
|
const payload = JSON.stringify({ operation, userId, data });
|
|
return crypto.createHash('sha256').update(payload).digest('hex');
|
|
}
|
|
```
|
|
|
|
### Optimistic Locking Security / Bảo mật Optimistic Lock
|
|
|
|
**Version Tampering Prevention / Ngăn chặn Giả mạo Version**:
|
|
- Validate version field on server-side only
|
|
- Never accept version from client directly
|
|
- Log version conflicts for security monitoring
|
|
- Rate limit update attempts per user
|
|
|
|
## Deployment / Triển khai
|
|
|
|
**EN**: How data consistency patterns are deployed and scaled.
|
|
|
|
**VI**: Cách các patterns đồng bộ dữ liệu được triển khai và mở rộng.
|
|
|
|
```mermaid
|
|
graph TD
|
|
subgraph "Production Deployment"
|
|
subgraph "Order Service Cluster"
|
|
OS1[Order Service\nPod 1]
|
|
OS2[Order Service\nPod 2]
|
|
OS3[Order Service\nPod 3]
|
|
end
|
|
|
|
subgraph "Saga Orchestrator"
|
|
SO1[Saga Orchestrator\nPod 1]
|
|
SO2[Saga Orchestrator\nPod 2]
|
|
end
|
|
|
|
subgraph "Outbox Processor"
|
|
OP1[Outbox Processor\nPod 1]
|
|
OP2[Outbox Processor\nPod 2]
|
|
end
|
|
|
|
OS1 & OS2 & OS3 --> DB[(Order DB\nwith Outbox)]
|
|
OS1 & OS2 & OS3 --> Redis[(Redis\nIdempotency Keys)]
|
|
|
|
SO1 & SO2 --> PS[Payment Service]
|
|
SO1 & SO2 --> IS[Inventory Service]
|
|
|
|
OP1 & OP2 --> DB
|
|
OP1 & OP2 --> Kafka[Kafka Cluster\n5 brokers]
|
|
end
|
|
|
|
style SO1 fill:#e1f5ff
|
|
style SO2 fill:#e1f5ff
|
|
style OP1 fill:#fff4e1
|
|
style OP2 fill:#fff4e1
|
|
style DB fill:#d4edda
|
|
style Kafka fill:#ffe1e1
|
|
```
|
|
|
|
### Deployment Configuration / Cấu hình Triển khai
|
|
|
|
| Component / Thành phần | Replicas | Resources | HA Strategy |
|
|
|------------------------|----------|-----------|-------------|
|
|
| **Saga Orchestrator** | 2-3 | 512Mi RAM, 500m CPU | Leader election with etcd |
|
|
| **Outbox Processor** | 2-5 | 256Mi RAM, 250m CPU | Distributed lock per event batch |
|
|
| **Services with Outbox** | 3+ | Varies | Standard service scaling |
|
|
| **Redis (Idempotency)** | 3 nodes | 1Gi RAM each | Redis Cluster with replication |
|
|
|
|
### Scaling Strategy / Chiến lược Mở rộng
|
|
|
|
**Saga Orchestrator**:
|
|
- Scale based on pending saga count
|
|
- Use queue-based load distribution
|
|
- Monitor saga execution duration
|
|
|
|
**Outbox Processor**:
|
|
- Scale with database sharding (1 processor per shard)
|
|
- Increase batch size before adding replicas
|
|
- Monitor outbox table size and age
|
|
|
|
**Idempotency Store (Redis)**:
|
|
- Scale Redis cluster horizontally
|
|
- Use consistent hashing for key distribution
|
|
- Monitor memory usage (should be < 70%)
|
|
|
|
## Monitoring & Observability / Giám sát & Khả năng quan sát
|
|
|
|
**EN**: Monitoring strategies for data consistency patterns.
|
|
|
|
**VI**: Chiến lược giám sát cho patterns đồng bộ dữ liệu.
|
|
|
|
### Key Metrics / Chỉ số Chính
|
|
|
|
**Saga Metrics**:
|
|
- `saga_executions_total` - Total saga executions (success/failure)
|
|
- `saga_duration_seconds` - Saga execution time histogram
|
|
- `saga_compensations_total` - Total compensation executions
|
|
- `saga_timeout_total` - Sagas that timed out
|
|
- `saga_pending_count` - Sagas currently executing
|
|
|
|
**Outbox Metrics**:
|
|
- `outbox_events_total` - Events written to outbox
|
|
- `outbox_published_total` - Events published to Kafka
|
|
- `outbox_processing_lag_seconds` - Time from write to publish
|
|
- `outbox_table_size` - Outbox table row count
|
|
- `outbox_failed_events_total` - Failed event publications
|
|
|
|
**Idempotency Metrics**:
|
|
- `idempotency_checks_total` - Total idempotency checks
|
|
- `idempotency_hits_total` - Duplicate requests prevented
|
|
- `idempotency_key_ttl_seconds` - Average key TTL
|
|
- `idempotency_redis_errors_total` - Redis failures
|
|
|
|
**Optimistic Lock Metrics**:
|
|
- `optimistic_lock_conflicts_total` - Version conflicts detected
|
|
- `optimistic_lock_retries_total` - Retry attempts after conflict
|
|
- `optimistic_lock_success_rate` - Update success percentage
|
|
|
|
### Alerts / Cảnh báo
|
|
|
|
**Critical Alerts**:
|
|
```yaml
|
|
# Saga timeout rate too high
|
|
alert: HighSagaTimeoutRate
|
|
expr: rate(saga_timeout_total[5m]) > 0.05
|
|
for: 5m
|
|
severity: critical
|
|
|
|
# Outbox processing lag
|
|
alert: OutboxProcessingLag
|
|
expr: outbox_processing_lag_seconds > 300
|
|
for: 10m
|
|
severity: critical
|
|
|
|
# High optimistic lock conflict rate
|
|
alert: HighOptimisticLockConflicts
|
|
expr: rate(optimistic_lock_conflicts_total[5m]) / rate(optimistic_lock_attempts_total[5m]) > 0.1
|
|
for: 5m
|
|
severity: warning
|
|
```
|
|
|
|
### Monitoring Dashboard / Dashboard Giám sát
|
|
|
|
**Grafana Panels**:
|
|
|
|
1. **Saga Orchestration Overview**:
|
|
- Saga execution rate (success/failure)
|
|
- Average saga duration
|
|
- Compensation rate
|
|
- Pending saga count
|
|
|
|
2. **Outbox Processing Health**:
|
|
- Outbox publishing rate
|
|
- Processing lag (P95, P99)
|
|
- Failed events
|
|
- Table size trend
|
|
|
|
3. **Idempotency Effectiveness**:
|
|
- Duplicate prevention rate
|
|
- Redis hit rate
|
|
- Key distribution
|
|
|
|
4. **Data Consistency SLA**:
|
|
- Overall consistency rate (target: 99.9%)
|
|
- Mean time to consistency (MTTC)
|
|
- Conflict resolution success rate
|
|
|
|
### Distributed Tracing / Tracing Phân tán
|
|
|
|
**Trace Saga Execution**:
|
|
```typescript
|
|
// EN: Traced saga step
|
|
// VI: Saga step được trace
|
|
async function executeStepWithTracing(
|
|
step: SagaStep,
|
|
context: SagaContext
|
|
): Promise<void> {
|
|
const tracer = trace.getTracer('saga-orchestrator');
|
|
const span = tracer.startSpan(`saga.step.${step.name}`, {
|
|
attributes: {
|
|
'saga.id': context.sagaId,
|
|
'saga.step': step.name,
|
|
'saga.attempt': context.currentAttempt
|
|
}
|
|
});
|
|
|
|
try {
|
|
await step.execute(context);
|
|
span.setStatus({ code: SpanStatusCode.OK });
|
|
} catch (error) {
|
|
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
|
|
span.recordException(error);
|
|
throw error;
|
|
} finally {
|
|
span.end();
|
|
}
|
|
}
|
|
```
|
|
|
|
## Related Documentation / Tài liệu Liên quan
|
|
|
|
- [Event-Driven Architecture](./event-driven-architecture.md) - Event sourcing and Kafka
|
|
- [System Design](./system-design.md) - Overall architecture
|
|
- [Microservices Communication](./microservices-communication.md) - Service communication patterns
|
|
- [Resilience Patterns](../skills/resilience-patterns.md) - Circuit breaker, retry for saga steps
|
|
- [Caching Patterns](../skills/caching-patterns.md) - Caching for idempotency keys
|
|
- [Database Prisma](../skills/database-prisma.md) - Prisma transactions for outbox pattern
|
|
|
|
---
|
|
|
|
**Last Updated**: 2026-01-07
|
|
**Authors**: GoodGo Architecture Team
|
|
**Reviewers**: To be assigned
|