---
name: data-consistency-patterns
description: Data consistency patterns for distributed microservices including Saga patterns, distributed transactions, eventual consistency, compensation, and idempotency. Use when handling distributed transactions, implementing eventual consistency, or managing data synchronization across services.
---

# Data Consistency Patterns

## When to Use This Skill

Use this skill when:
- Implementing distributed transactions across multiple services
- Handling eventual consistency in microservices
- Implementing Saga patterns for distributed workflows
- Designing compensation strategies for failed transactions
- Implementing idempotent operations
- Managing data synchronization across services
- Handling conflict resolution

## Core Concepts

### ACID vs BASE

**ACID (Traditional):** Atomicity, Consistency, Isolation, Durability

**BASE (Distributed):** Basic Availability, Soft state, Eventual consistency

### Consistency Models

- **Strong Consistency**: All nodes see same data at same time
- **Eventual Consistency**: System becomes consistent over time
- **Weak Consistency**: No guarantees about when consistency occurs

## Key Patterns

### Saga Orchestrator Pattern

#### Saga Orchestration Flow

The following diagram illustrates how a Saga orchestrator executes steps sequentially and handles compensation on failure:

```mermaid
sequenceDiagram
    participant Client
    participant Orchestrator
    participant Step1 as Step 1: Create Order
    participant Step2 as Step 2: Reserve Inventory
    participant Step3 as Step 3: Process Payment

    Client->>Orchestrator: Execute Saga
    Orchestrator->>Step1: Execute Step 1
    Step1-->>Orchestrator: Success (Order Created)
    Orchestrator->>Step2: Execute Step 2
    Step2-->>Orchestrator: Success (Inventory Reserved)
    Orchestrator->>Step3: Execute Step 3
    Step3-->>Orchestrator: Failure (Payment Failed)
    Orchestrator->>Step2: Compensate Step 2
    Step2-->>Orchestrator: Compensation Complete
    Orchestrator->>Step1: Compensate Step 1
    Step1-->>Orchestrator: Compensation Complete
    Orchestrator-->>Client: Saga Failed (Compensated)
```

#### Compensation Flow

When a step fails, the orchestrator compensates all previously completed steps in reverse order:

```mermaid
flowchart TD
    Start([Saga Execution Starts]) --> Step1[Execute Step 1]
    Step1 -->|Success| Step2[Execute Step 2]
    Step1 -->|Failure| Fail1[Saga Failed<br/>No Compensation Needed]
    
    Step2 -->|Success| Step3[Execute Step 3]
    Step2 -->|Failure| Comp1[Compensate Step 1]
    
    Step3 -->|Success| Complete([Saga Completed])
    Step3 -->|Failure| Comp2[Compensate Step 2]
    
    Comp2 --> Comp1
    Comp1 --> Fail2[Saga Failed<br/>All Steps Compensated]
    
    style Start fill:#e1f5ff
    style Complete fill:#d4edda
    style Fail1 fill:#f8d7da
    style Fail2 fill:#f8d7da
    style Comp1 fill:#fff3cd
    style Comp2 fill:#fff3cd
```

#### Eventual Consistency Flow

This diagram shows how data becomes consistent across services over time through event propagation:

```mermaid
flowchart LR
    subgraph ServiceA[Service A: Write Model]
        Write[Write Operation] --> EventStore[Event Store]
        EventStore --> Publish[Publish Event]
    end
    
    subgraph EventBus[Event Bus]
        Publish --> Queue[Event Queue]
    end
    
    subgraph ServiceB[Service B: Read Model]
        Queue --> Consume[Consume Event]
        Consume --> Update[Update Read Model]
        Update --> Consistent[Eventually Consistent]
    end
    
    subgraph ServiceC[Service C: Read Model]
        Queue --> Consume2[Consume Event]
        Consume2 --> Update2[Update Read Model]
        Update2 --> Consistent2[Eventually Consistent]
    end
    
    style Write fill:#e1f5ff
    style Consistent fill:#d4edda
    style Consistent2 fill:#d4edda
    style Queue fill:#fff3cd
```

```typescript
// Centralized orchestrator coordinates steps
const saga = new SagaOrchestrator();
await saga.execute({
  sagaId: 'saga_123',
  steps: [
    { name: 'create-order', execute: createOrder, compensate: cancelOrder },
    { name: 'reserve-inventory', execute: reserveInventory, compensate: releaseInventory },
    { name: 'process-payment', execute: chargePayment, compensate: refundPayment },
  ],
  data: {},
  status: 'pending',
});
```

### Saga Choreography Pattern

In choreography, services react to events without a central coordinator:

```typescript
// Services react to events
eventConsumer.on('order.created', async (event) => {
  await inventoryService.reserve(event.data.items);
  await eventPublisher.publish('inventory.reserved', {...});
});
```

#### Saga Choreography Flow

The following diagram shows how services coordinate through events in a choreography pattern:

```mermaid
sequenceDiagram
    participant OrderService
    participant EventBus
    participant InventoryService
    participant PaymentService
    participant NotificationService

    OrderService->>EventBus: Publish order.created
    EventBus->>InventoryService: order.created event
    InventoryService->>InventoryService: Reserve Inventory
    InventoryService->>EventBus: Publish inventory.reserved
    EventBus->>PaymentService: inventory.reserved event
    PaymentService->>PaymentService: Process Payment
    PaymentService->>EventBus: Publish payment.processed
    EventBus->>NotificationService: payment.processed event
    NotificationService->>NotificationService: Send Confirmation
    
    Note over InventoryService,PaymentService: If payment fails,<br/>compensation events are published
    PaymentService->>EventBus: Publish payment.failed
    EventBus->>InventoryService: payment.failed event
    InventoryService->>InventoryService: Release Inventory
    EventBus->>OrderService: payment.failed event
    OrderService->>OrderService: Cancel Order
```

### Idempotency

Idempotency ensures operations can be safely retried without side effects:

```typescript
// Execute operation with idempotency check
await idempotencyHandler.execute(
  idempotencyKey,
  async () => await userService.create(data)
);
```

#### Idempotency Flow

```mermaid
flowchart TD
    Request[Client Request] --> Check{Idempotency Key<br/>Exists?}
    Check -->|Yes| Return[Return Cached Result]
    Check -->|No| Execute[Execute Operation]
    Execute --> Store[Store Result with Key]
    Store --> Return2[Return Result]
    
    Return --> Client[Client Response]
    Return2 --> Client
    
    style Check fill:#fff3cd
    style Return fill:#d4edda
    style Return2 fill:#d4edda
```

### Optimistic Locking

Optimistic locking prevents lost updates using version fields:

```typescript
// Update with version check
await optimisticLockService.updateWithLock(
  repository,
  id,
  (current) => ({ ...current, name: newName })
);
```

#### Optimistic Locking Flow

```mermaid
sequenceDiagram
    participant Client1
    participant Client2
    participant Service
    participant DB[(Database)]

    Client1->>Service: Read Entity (version=1)
    Service->>DB: SELECT * WHERE id=123
    DB-->>Service: Entity (version=1)
    Service-->>Client1: Entity Data
    
    Client2->>Service: Read Entity (version=1)
    Service->>DB: SELECT * WHERE id=123
    DB-->>Service: Entity (version=1)
    Service-->>Client2: Entity Data
    
    Client1->>Service: Update (version=1)
    Service->>DB: UPDATE WHERE id=123 AND version=1
    DB-->>Service: Success (version=2)
    
    Client2->>Service: Update (version=1)
    Service->>DB: UPDATE WHERE id=123 AND version=1
    DB-->>Service: Conflict (version=2 exists)
    Service-->>Client2: OptimisticLockError
    Note over Client2: Retry with new version
```

### CQRS Pattern

Command Query Responsibility Segregation separates read and write operations for optimized performance and scalability.

#### CQRS Architecture Flow

The following diagram illustrates how CQRS separates write and read paths:

```mermaid
flowchart TB
    subgraph WritePath[Write Path]
        Command[Command] --> WriteModel[Write Model<br/>Normalized]
        WriteModel --> Event[Domain Event]
        Event --> EventStore[(Event Store)]
    end
    
    subgraph ReadPath[Read Path]
        Query[Query] --> ReadModel[Read Model<br/>Denormalized]
        ReadModel --> Response[Query Response]
    end
    
    subgraph Sync[Eventual Consistency]
        EventStore --> EventHandler[Event Handler]
        EventHandler --> Projection[Projection]
        Projection --> ReadModel
    end
    
    style WritePath fill:#e1f5ff
    style ReadPath fill:#d4edda
    style Sync fill:#fff3cd
    style EventStore fill:#f8d7da
```

```typescript
// Write path: Command handler
await commandHandler.handle({
  type: 'CREATE_ORDER',
  data: { userId, items }
});

// Read path: Optimized query
const orders = await readModel.findOrdersByUser(userId);
```

### Outbox Pattern

The Outbox pattern ensures reliable event publishing by storing events in the same database transaction as business data.

#### Outbox Pattern Flow

This diagram shows how the Outbox pattern guarantees event delivery:

```mermaid
sequenceDiagram
    participant Client
    participant Service
    participant DB[(Database)]
    participant OutboxTable[(Outbox Table)]
    participant Processor[Outbox Processor]
    participant EventBus[Event Bus]

    Client->>Service: Business Operation
    Service->>DB: Begin Transaction
    Service->>DB: Update Business Data
    Service->>OutboxTable: Insert Event (same transaction)
    Service->>DB: Commit Transaction
    
    Note over Service,OutboxTable: Event stored atomically<br/>with business data
    
    Processor->>OutboxTable: Poll for Unpublished Events
    OutboxTable-->>Processor: Return Events
    Processor->>EventBus: Publish Event
    EventBus-->>Processor: Publish Confirmed
    Processor->>OutboxTable: Mark as Published
    
    Note over Processor,EventBus: Background processor<br/>ensures delivery
```

```typescript
// Execute business operation and store event in same transaction
await prisma.$transaction(async (tx) => {
  // Business operation
  await tx.order.create({ data: orderData });
  
  // Store event in outbox (same transaction)
  await tx.outboxEvent.create({
    data: {
      eventType: 'order.created',
      payload: orderData,
      status: 'pending'
    }
  });
});

// Background processor publishes events from outbox
```

## Best Practices

1. **Design Compensations**: Every step needs compensation
2. **Idempotent Steps**: Make steps idempotent for retries
3. **Conflict Resolution**: Define resolution strategies
4. **Monitoring**: Track saga execution and consistency lag
5. **Read Models**: Use separate read models for queries (CQRS)

## Resources

- [Saga Pattern](https://microservices.io/patterns/data/saga.html)
- [Event-Driven Architecture](./event-driven-architecture.md)
- [Error Handling Patterns](./error-handling-patterns.md)
- Skill Source: `.cursor/skills/data-consistency-patterns/SKILL.md`