423 lines
16 KiB
Markdown
423 lines
16 KiB
Markdown
---
|
|
name: outbox-pattern
|
|
description: Transactional Outbox Pattern - đảm bảo atomicity khi publish events. Use for reliable messaging, at-least-once delivery, và event consistency.
|
|
compatibility: ".NET 10+, EF Core, MassTransit, PostgreSQL"
|
|
metadata:
|
|
author: Velik Ho
|
|
version: "1.0"
|
|
---
|
|
|
|
# Outbox Pattern / Transactional Outbox
|
|
|
|
Pattern đảm bảo tính nhất quán khi publish events từ microservices.
|
|
|
|
## When to Use This Skill / Khi Nào Sử Dụng
|
|
|
|
Use this skill when:
|
|
- Publishing integration events reliably / Publish integration events tin cậy
|
|
- Ensuring at-least-once delivery / Đảm bảo gửi ít nhất một lần
|
|
- Avoiding dual-write problem / Tránh vấn đề dual-write
|
|
- Distributed transactions not feasible / Không khả thi dùng distributed transactions
|
|
|
|
## Core Concepts / Khái Niệm Cốt Lõi
|
|
|
|
### The Dual-Write Problem / Vấn Đề Dual-Write
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ ❌ DUAL-WRITE PROBLEM │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Service A │
|
|
│ ┌─────────────────────────────────────────────┐ │
|
|
│ │ 1. await _db.SaveChangesAsync(); ✅ │ │
|
|
│ │ 2. await _bus.PublishAsync(event); ❌ FAIL! │ │
|
|
│ └─────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Result: DB updated BUT message NOT sent = INCONSISTENCY │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ ✅ OUTBOX PATTERN │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Service A │
|
|
│ ┌─────────────────────────────────────────────┐ │
|
|
│ │ BEGIN TRANSACTION │ │
|
|
│ │ 1. INSERT INTO Orders ... │ │
|
|
│ │ 2. INSERT INTO Outbox (event) │ │
|
|
│ │ COMMIT │ │
|
|
│ └─────────────────────────────────────────────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌─────────────────────────────────────────────┐ │
|
|
│ │ Outbox Processor (Background) │ │
|
|
│ │ 1. SELECT * FROM Outbox WHERE Sent = false │ │
|
|
│ │ 2. Publish to Message Bus │ │
|
|
│ │ 3. UPDATE Outbox SET Sent = true │ │
|
|
│ └─────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Result: ATOMIC - Both or Neither │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Outbox Flow / Luồng Outbox
|
|
|
|
```
|
|
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
|
│ Command │───▶│ Handler │───▶│ DB + Outbox│───▶│ Processor │
|
|
│ (API) │ │ (Domain) │ │ (Same Tx) │ │ (Background)│
|
|
└─────────────┘ └─────────────┘ └─────────────┘ └──────┬──────┘
|
|
│
|
|
▼
|
|
┌─────────────┐
|
|
│ Message Bus │
|
|
│ (RabbitMQ) │
|
|
└─────────────┘
|
|
```
|
|
|
|
### Key Components / Thành Phần Chính
|
|
|
|
| Component | Purpose | Implementation |
|
|
|-----------|---------|----------------|
|
|
| **Outbox Table** | Store pending events | EF Core entity |
|
|
| **Outbox Processor** | Publish pending events | BackgroundService |
|
|
| **Idempotency** | Prevent duplicates | Event ID tracking |
|
|
|
|
## Key Patterns / Mẫu Chính
|
|
|
|
### Outbox Entity / Entity Outbox
|
|
|
|
```csharp
|
|
/// <summary>
|
|
/// EN: Outbox message entity for transactional messaging.
|
|
/// VI: Entity outbox message cho transactional messaging.
|
|
/// </summary>
|
|
public class OutboxMessage
|
|
{
|
|
public Guid Id { get; set; }
|
|
public string EventType { get; set; } = default!;
|
|
public string Payload { get; set; } = default!;
|
|
public DateTime CreatedAt { get; set; }
|
|
public DateTime? ProcessedAt { get; set; }
|
|
public bool IsProcessed { get; set; }
|
|
public int RetryCount { get; set; }
|
|
public string? Error { get; set; }
|
|
}
|
|
```
|
|
|
|
### DbContext Configuration / Cấu Hình DbContext
|
|
|
|
```csharp
|
|
/// <summary>
|
|
/// EN: DbContext with outbox support.
|
|
/// VI: DbContext với hỗ trợ outbox.
|
|
/// </summary>
|
|
public class OrderDbContext : DbContext
|
|
{
|
|
public DbSet<Order> Orders => Set<Order>();
|
|
public DbSet<OutboxMessage> OutboxMessages => Set<OutboxMessage>();
|
|
|
|
protected override void OnModelCreating(ModelBuilder modelBuilder)
|
|
{
|
|
modelBuilder.Entity<OutboxMessage>(entity =>
|
|
{
|
|
entity.ToTable("OutboxMessages");
|
|
entity.HasKey(e => e.Id);
|
|
|
|
entity.Property(e => e.EventType)
|
|
.HasMaxLength(500)
|
|
.IsRequired();
|
|
|
|
entity.Property(e => e.Payload)
|
|
.HasColumnType("jsonb")
|
|
.IsRequired();
|
|
|
|
entity.HasIndex(e => new { e.IsProcessed, e.CreatedAt })
|
|
.HasFilter("\"IsProcessed\" = false");
|
|
});
|
|
}
|
|
}
|
|
```
|
|
|
|
### Unit of Work with Outbox / Unit of Work với Outbox
|
|
|
|
```csharp
|
|
/// <summary>
|
|
/// EN: Unit of Work that saves domain events to outbox.
|
|
/// VI: Unit of Work lưu domain events vào outbox.
|
|
/// </summary>
|
|
public interface IUnitOfWork
|
|
{
|
|
Task<int> SaveChangesAsync(CancellationToken ct = default);
|
|
}
|
|
|
|
public class UnitOfWork : IUnitOfWork
|
|
{
|
|
private readonly OrderDbContext _context;
|
|
private readonly ILogger<UnitOfWork> _logger;
|
|
|
|
public UnitOfWork(OrderDbContext context, ILogger<UnitOfWork> logger)
|
|
{
|
|
_context = context;
|
|
_logger = logger;
|
|
}
|
|
|
|
public async Task<int> SaveChangesAsync(CancellationToken ct = default)
|
|
{
|
|
// EN: Convert domain events to outbox messages
|
|
// VI: Chuyển domain events thành outbox messages
|
|
var aggregates = _context.ChangeTracker
|
|
.Entries<IAggregateRoot>()
|
|
.Where(e => e.Entity.DomainEvents.Any())
|
|
.Select(e => e.Entity)
|
|
.ToList();
|
|
|
|
foreach (var aggregate in aggregates)
|
|
{
|
|
foreach (var domainEvent in aggregate.DomainEvents)
|
|
{
|
|
var outboxMessage = new OutboxMessage
|
|
{
|
|
Id = Guid.NewGuid(),
|
|
EventType = domainEvent.GetType().AssemblyQualifiedName!,
|
|
Payload = JsonSerializer.Serialize(domainEvent, domainEvent.GetType()),
|
|
CreatedAt = DateTime.UtcNow,
|
|
IsProcessed = false
|
|
};
|
|
|
|
_context.OutboxMessages.Add(outboxMessage);
|
|
}
|
|
|
|
aggregate.ClearDomainEvents();
|
|
}
|
|
|
|
var result = await _context.SaveChangesAsync(ct);
|
|
|
|
_logger.LogDebug("Saved {Count} changes with outbox messages", result);
|
|
|
|
return result;
|
|
}
|
|
}
|
|
```
|
|
|
|
### Outbox Processor / Outbox Processor
|
|
|
|
```csharp
|
|
/// <summary>
|
|
/// EN: Background service that processes outbox messages.
|
|
/// VI: Background service xử lý outbox messages.
|
|
/// </summary>
|
|
public class OutboxProcessor : BackgroundService
|
|
{
|
|
private readonly IServiceScopeFactory _scopeFactory;
|
|
private readonly ILogger<OutboxProcessor> _logger;
|
|
private readonly TimeSpan _pollingInterval = TimeSpan.FromSeconds(5);
|
|
private const int BatchSize = 100;
|
|
private const int MaxRetries = 5;
|
|
|
|
public OutboxProcessor(
|
|
IServiceScopeFactory scopeFactory,
|
|
ILogger<OutboxProcessor> logger)
|
|
{
|
|
_scopeFactory = scopeFactory;
|
|
_logger = logger;
|
|
}
|
|
|
|
protected override async Task ExecuteAsync(CancellationToken ct)
|
|
{
|
|
_logger.LogInformation("Outbox Processor started");
|
|
|
|
while (!ct.IsCancellationRequested)
|
|
{
|
|
try
|
|
{
|
|
await ProcessPendingMessagesAsync(ct);
|
|
}
|
|
catch (Exception ex)
|
|
{
|
|
_logger.LogError(ex, "Error processing outbox messages");
|
|
}
|
|
|
|
await Task.Delay(_pollingInterval, ct);
|
|
}
|
|
}
|
|
|
|
private async Task ProcessPendingMessagesAsync(CancellationToken ct)
|
|
{
|
|
using var scope = _scopeFactory.CreateScope();
|
|
var context = scope.ServiceProvider.GetRequiredService<OrderDbContext>();
|
|
var publishEndpoint = scope.ServiceProvider.GetRequiredService<IPublishEndpoint>();
|
|
|
|
var messages = await context.OutboxMessages
|
|
.Where(m => !m.IsProcessed && m.RetryCount < MaxRetries)
|
|
.OrderBy(m => m.CreatedAt)
|
|
.Take(BatchSize)
|
|
.ToListAsync(ct);
|
|
|
|
foreach (var message in messages)
|
|
{
|
|
try
|
|
{
|
|
var eventType = Type.GetType(message.EventType);
|
|
if (eventType == null)
|
|
{
|
|
_logger.LogWarning("Unknown event type: {EventType}", message.EventType);
|
|
continue;
|
|
}
|
|
|
|
var @event = JsonSerializer.Deserialize(message.Payload, eventType);
|
|
await publishEndpoint.Publish(@event!, eventType, ct);
|
|
|
|
message.IsProcessed = true;
|
|
message.ProcessedAt = DateTime.UtcNow;
|
|
|
|
_logger.LogDebug("Published outbox message {Id}", message.Id);
|
|
}
|
|
catch (Exception ex)
|
|
{
|
|
message.RetryCount++;
|
|
message.Error = ex.Message;
|
|
_logger.LogWarning(ex, "Failed to publish message {Id}, retry {Retry}",
|
|
message.Id, message.RetryCount);
|
|
}
|
|
}
|
|
|
|
await context.SaveChangesAsync(ct);
|
|
}
|
|
}
|
|
```
|
|
|
|
### Command Handler with Outbox / Command Handler với Outbox
|
|
|
|
```csharp
|
|
/// <summary>
|
|
/// EN: Command handler that raises domain events saved to outbox.
|
|
/// VI: Command handler phát domain events được lưu vào outbox.
|
|
/// </summary>
|
|
public class CreateOrderCommandHandler : IRequestHandler<CreateOrderCommand, OrderResult>
|
|
{
|
|
private readonly IOrderRepository _repository;
|
|
private readonly ILogger<CreateOrderCommandHandler> _logger;
|
|
|
|
public CreateOrderCommandHandler(
|
|
IOrderRepository repository,
|
|
ILogger<CreateOrderCommandHandler> logger)
|
|
{
|
|
_repository = repository;
|
|
_logger = logger;
|
|
}
|
|
|
|
public async Task<OrderResult> Handle(CreateOrderCommand request, CancellationToken ct)
|
|
{
|
|
var order = new Order(request.UserId, request.ShippingAddress);
|
|
|
|
foreach (var item in request.Items)
|
|
{
|
|
order.AddItem(item.ProductId, item.Quantity, item.UnitPrice);
|
|
}
|
|
|
|
// EN: Domain events are raised inside aggregate
|
|
// VI: Domain events được phát trong aggregate
|
|
|
|
await _repository.AddAsync(order, ct);
|
|
|
|
// EN: SaveChanges converts domain events to outbox messages
|
|
// VI: SaveChanges chuyển domain events thành outbox messages
|
|
await _repository.UnitOfWork.SaveChangesAsync(ct);
|
|
|
|
_logger.LogInformation("Order created: {OrderId}", order.Id);
|
|
|
|
return new OrderResult(order.Id);
|
|
}
|
|
}
|
|
```
|
|
|
|
## Common Mistakes / Lỗi Thường Gặp
|
|
|
|
### 1. Publishing Before Saving
|
|
|
|
```csharp
|
|
// ❌ BAD: Publish before DB commit
|
|
await _publishEndpoint.Publish(new OrderCreatedEvent(order.Id));
|
|
await _dbContext.Orders.AddAsync(order);
|
|
await _dbContext.SaveChangesAsync();
|
|
// If SaveChanges fails, message is already sent!
|
|
|
|
// ✅ GOOD: Use outbox pattern
|
|
order.AddDomainEvent(new OrderCreatedEvent(order.Id));
|
|
await _dbContext.Orders.AddAsync(order);
|
|
await _dbContext.SaveChangesAsync(); // Saves both order AND outbox in same transaction
|
|
```
|
|
|
|
### 2. No Idempotency in Consumers
|
|
|
|
```csharp
|
|
// ❌ BAD: No idempotency check
|
|
public async Task Consume(ConsumeContext<OrderCreatedEvent> context)
|
|
{
|
|
await _service.ProcessOrder(context.Message.OrderId);
|
|
// May process duplicate messages!
|
|
}
|
|
|
|
// ✅ GOOD: Idempotent consumer
|
|
public async Task Consume(ConsumeContext<OrderCreatedEvent> context)
|
|
{
|
|
var eventId = context.MessageId;
|
|
if (await _processedMessages.ExistsAsync(eventId))
|
|
return; // Already processed
|
|
|
|
await _service.ProcessOrder(context.Message.OrderId);
|
|
await _processedMessages.MarkAsync(eventId);
|
|
}
|
|
```
|
|
|
|
### 3. No Retry Limit
|
|
|
|
```csharp
|
|
// ❌ BAD: Infinite retries
|
|
var messages = await context.OutboxMessages
|
|
.Where(m => !m.IsProcessed) // Will keep retrying failed messages forever
|
|
.ToListAsync();
|
|
|
|
// ✅ GOOD: Retry limit with dead letter
|
|
var messages = await context.OutboxMessages
|
|
.Where(m => !m.IsProcessed && m.RetryCount < MaxRetries)
|
|
.ToListAsync();
|
|
|
|
// Move failed messages to dead letter after max retries
|
|
```
|
|
|
|
## Quick Reference / Tham Chiếu Nhanh
|
|
|
|
### Outbox Table Schema
|
|
|
|
| Column | Type | Purpose |
|
|
|--------|------|---------|
|
|
| Id | UUID | Primary key |
|
|
| EventType | VARCHAR(500) | Full type name for deserialization |
|
|
| Payload | JSONB | Serialized event data |
|
|
| CreatedAt | TIMESTAMP | When event was created |
|
|
| ProcessedAt | TIMESTAMP | When event was published |
|
|
| IsProcessed | BOOLEAN | Processing status |
|
|
| RetryCount | INT | Failed attempts |
|
|
| Error | TEXT | Last error message |
|
|
|
|
### Delivery Guarantees
|
|
|
|
| Pattern | Guarantee | Trade-off |
|
|
|---------|-----------|-----------|
|
|
| Direct publish | Best-effort | May lose messages |
|
|
| Outbox | At-least-once | May duplicate messages |
|
|
| Inbox + Outbox | Exactly-once | More complexity |
|
|
|
|
## Resources / Tài Nguyên
|
|
|
|
- [Detailed Examples](./references/REFERENCE.md) - Full code examples
|
|
- [Event Sourcing](../event-sourcing/SKILL.md) - Event-based persistence
|
|
- [Inter-service Communication](../inter-service-communication/SKILL.md) - MassTransit
|
|
- [SAGA Pattern](../saga-pattern/SKILL.md) - Distributed transactions
|
|
- [Repository Pattern](../repository-pattern/SKILL.md) - Unit of Work
|