pos-system/services/storage-service-net/docs/en/ARCHITECTURE.md

Storage Service Architecture

Detailed architecture documentation for the Storage Service microservice.

Architecture Overview

graph TB
    subgraph "API Layer"
        C[Controllers]
        CMD[Commands]
        Q[Queries]
        B[Behaviors]
    end
    
    subgraph "Domain Layer"
        SF[StorageFile]
        SQ[UserStorageQuota]
        DE[Domain Events]
        RI[Repository Interfaces]
    end
    
    subgraph "Infrastructure Layer"
        SP[Storage Providers]
        IAM[IAM Client]
        R[Repositories]
        CTX[DbContext]
    end
    
    subgraph "External Services"
        MINIO[(MinIO)]
        OSS[(Aliyun OSS)]
        IAMS[IAM Service]
        DB[(PostgreSQL)]
    end
    
    C --> CMD
    C --> Q
    CMD --> B --> SF
    Q --> R
    R --> CTX --> DB
    SF --> DE
    
    CMD --> SP
    SP --> MINIO
    SP --> OSS
    
    C --> IAM --> IAMS
    
    style C fill:#4a90d9,stroke:#2d5986,color:#fff
    style SF fill:#50c878,stroke:#2d8659,color:#fff
    style MINIO fill:#c73b3b,stroke:#922b2b,color:#fff
    style OSS fill:#ff6b35,stroke:#cc5500,color:#fff
    style IAMS fill:#9b59b6,stroke:#7d3c98,color:#fff

Layer Responsibilities

1. Domain Layer (StorageService.Domain)

The heart of the application containing pure business logic.

Component	Purpose
StorageFile	Aggregate root for file metadata and lifecycle
UserStorageQuota	Aggregate root for user storage limits and usage
StorageProvider	Enum: MinIO, AliyunOSS
FileAccessLevel	Enum: Private, Public, Shared
Domain Events	FileUploadedDomainEvent, FileDeletedDomainEvent, UserQuotaUpdatedDomainEvent

2. Infrastructure Layer (StorageService.Infrastructure)

Technical implementations and external integrations:

Component	Purpose
MinioStorageProvider	MinIO S3-compatible storage operations
AliyunOssStorageProvider	Alibaba Cloud OSS operations
StorageProviderFactory	Runtime provider selection based on config
HttpIamServiceClient	Inter-service communication with IAM
FileRepository	EF Core repository for StorageFile
QuotaRepository	EF Core repository for UserStorageQuota

3. API Layer (StorageService.API)

Application entry point and CQRS implementation:

Component	Purpose
FilesController	File CRUD endpoints (legacy proxy upload)
SignedUrlController	Direct upload endpoints (recommended)
QuotaController	User quota endpoints
SignUploadCommand	Generate pre-signed upload URLs
ConfirmUploadCommand	Confirm direct uploads and save metadata
UploadFileCommand	Handle proxy file uploads (legacy)
DeleteFileCommand	Handle file deletions
Query Handlers	Handle read operations

Direct Upload Architecture (Recommended)

For systems with millions of users, Direct Client Upload pattern is recommended over proxy upload.

Upload Patterns Comparison

Aspect	Proxy Upload (Legacy)	Direct Upload (Recommended)
Throughput	~100-500/sec	~10,000+/sec
Memory per request	100MB (file size)	~10KB (metadata only)
Latency (100MB file)	30-60s	10-20s
Backend load	High	Minimal

Direct Upload Flow

sequenceDiagram
    participant Client
    participant Storage_Service as Storage Service
    participant MinIO
    
    rect rgb(200, 230, 200)
        Note over Client,Storage_Service: 1. Request Upload URL (lightweight)
        Client->>Storage_Service: POST /api/v1/storage/sign-upload
        Storage_Service->>Storage_Service: Validate JWT, Check Quota
        Storage_Service-->>Client: Pre-signed PUT URL + ObjectKey
    end
    
    rect rgb(200, 200, 230)
        Note over Client,MinIO: 2. Direct Upload (bypasses backend)
        Client->>MinIO: PUT file binary to Pre-signed URL
        MinIO-->>Client: 200 OK
    end
    
    rect rgb(230, 230, 200)
        Note over Client,Storage_Service: 3. Confirm Upload (lightweight)
        Client->>Storage_Service: POST /api/v1/storage/confirm-upload
        Storage_Service->>MinIO: Verify file exists
        Storage_Service->>Storage_Service: Save metadata, Update quota
        Storage_Service-->>Client: File metadata
    end

Path-based Access Control

Files are organized with access level prefixes:

storage-bucket/
├── public/{userId}/{date}/{fileId}_{filename}   → Publicly accessible
├── private/{userId}/{date}/{fileId}_{filename}  → Requires pre-signed URL
└── shared/{userId}/{date}/{fileId}_{filename}   → Access controlled by rules

Direct Upload Components

Component	Purpose
SignUploadCommand	Validate quota, generate object key with path prefix, create pre-signed PUT URL
SignUploadCommandHandler	Handle sign-upload requests
ConfirmUploadCommand	Verify file exists, save metadata, update quota
ConfirmUploadCommandHandler	Handle confirm-upload with idempotency
SignedUrlController	/sign-upload and /confirm-upload endpoints

Storage Provider Architecture

graph TD
    subgraph "Storage Provider Factory"
        F[StorageProviderFactory]
        C[Configuration]
    end
    
    subgraph "Providers"
        MP[MinioStorageProvider]
        AP[AliyunOssStorageProvider]
    end
    
    subgraph "Storage Backends"
        MINIO[(MinIO)]
        OSS[(Aliyun OSS)]
    end
    
    C --> |STORAGE_PROVIDER=minio| F
    C --> |STORAGE_PROVIDER=aliyun| F
    F --> |GetProvider| MP
    F --> |GetProvider| AP
    MP --> MINIO
    AP --> OSS
    
    style F fill:#4a90d9,stroke:#2d5986,color:#fff
    style MP fill:#c73b3b,stroke:#922b2b,color:#fff
    style AP fill:#ff6b35,stroke:#cc5500,color:#fff

Storage Provider Interface

public interface IStorageProvider
{
    Task<UploadResult> UploadAsync(Stream stream, string objectKey, ...);
    Task<Stream> DownloadAsync(string bucketName, string objectKey);
    Task DeleteAsync(string bucketName, string objectKey);
    Task<bool> ExistsAsync(string bucketName, string objectKey);
    Task<string> GetPreSignedDownloadUrlAsync(string bucketName, string objectKey, int expirationSeconds);
    Task<string> GetPreSignedUploadUrlAsync(string bucketName, string objectKey, int expirationSeconds);
}

Inter-Service Communication

sequenceDiagram
    participant Client
    participant Storage as Storage Service
    participant Cache as In-Memory Cache
    participant IAM as IAM Service
    
    Client->>Storage: Upload File (JWT)
    Storage->>Cache: Check user cache
    alt Cache Hit
        Cache-->>Storage: User info
    else Cache Miss
        Storage->>IAM: GET /api/v1/users/me
        Note over Storage,IAM: Headers: Authorization, X-Service-Name
        IAM-->>Storage: User info
        Storage->>Cache: Store (5 min TTL)
    end
    Storage->>Storage: Validate quota
    Storage->>Storage: Upload to provider
    Storage-->>Client: Upload result

IAM Client Features

Feature	Description
Caching	In-memory cache for user info (5 min TTL)
Health Check	Check IAM availability with caching (1 min TTL)
Polly Resilience	Retry (3x exponential) + Circuit Breaker
Permission Check	HasPermissionAsync, HasRoleAsync

Available IAM Client Methods

// User Operations
Task<IamUserInfo?> ValidateUserAsync(string accessToken);
Task<IamUserInfo?> GetUserByIdAsync(string userId, string accessToken);
Task<bool> UserExistsAsync(string userId, string accessToken);
Task<IReadOnlyList<string>> GetUserRolesAsync(string userId, string accessToken);
Task<IReadOnlyList<string>> GetUserPermissionsAsync(string userId, string accessToken);
Task<bool> HasPermissionAsync(string userId, string permission, string accessToken);
Task<bool> HasRoleAsync(string userId, string role, string accessToken);

// Health Check
Task<IamHealthStatus> CheckHealthAsync();
Task<bool> IsAvailableAsync();

// Cache Management
void InvalidateUserCache(string userId);
void ClearCache();

Database Schema

erDiagram
    storage_files {
        uuid id PK
        varchar file_name
        varchar bucket_name
        varchar object_key
        varchar content_type
        bigint file_size_bytes
        varchar user_id
        varchar tenant_id
        int provider
        int access_level
        timestamp uploaded_at
        timestamp expires_at
        varchar checksum
        boolean is_deleted
        timestamp deleted_at
    }
    
    user_storage_quotas {
        uuid id PK
        varchar user_id UK
        bigint max_storage_bytes
        bigint used_storage_bytes
        int max_file_count
        int current_file_count
        varchar quota_tier
        timestamp last_updated_at
        timestamp created_at
    }

API Endpoints

Files (Legacy - Proxy Upload)

Method	Endpoint	Description
POST	/api/v1/files/upload	Upload file via backend (max 100MB)
GET	/api/v1/files	List user files with pagination
GET	/api/v1/files/{id}	Get file metadata
GET	/api/v1/files/{id}/download-url	Get pre-signed download URL
DELETE	/api/v1/files/{id}	Delete file (soft delete)

Direct Upload (Recommended)

Method	Endpoint	Description
POST	/api/v1/storage/sign-upload	Get pre-signed PUT URL for direct upload
POST	/api/v1/storage/confirm-upload	Confirm upload and save metadata

Quota

Method	Endpoint	Description
GET	/api/v1/quota	Get user storage quota

Health Checks

graph TD
    HC[Health Check Endpoint]
    HC --> |/health/live| L[Liveness]
    HC --> |/health/ready| R[Readiness]
    
    R --> DB[(PostgreSQL)]
    R --> MINIO[(MinIO)]
    R --> IAM[IAM Service]
    
    style HC fill:#3498db,stroke:#2980b9,color:#fff
    style L fill:#2ecc71,stroke:#27ae60,color:#fff
    style R fill:#f39c12,stroke:#d68910,color:#fff

Deployment Architecture

Docker Compose (Local)

services:
  storage-api:
    build: .
    ports: ["5002:8080"]
    depends_on:
      - postgres
      - redis
      - minio
    environment:
      - Storage__Provider=minio
      - Storage__MinIO__Endpoint=minio:9000
  
  minio:
    image: minio/minio:latest
    ports: ["9000:9000", "9001:9001"]

Traefik Integration

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.storage-service.rule=PathPrefix(`/api/v1/files`) || PathPrefix(`/api/v1/quota`)"
  - "traefik.http.services.storage-service.loadbalancer.server.port=8080"

Security Considerations

1. Authentication: JWT Bearer validation via IAM Service
2. Authorization: User ownership check on files
3. Input Validation: File size limits, content type validation
4. Pre-signed URLs: Time-limited access to files
5. Soft Delete: Files are marked deleted, not immediately removed

Performance Optimization

1. Caching: In-memory cache for IAM user info
2. Pre-signed URLs: Direct client-to-storage downloads
3. Streaming Upload: Stream-based file handling
4. Async Operations: All I/O operations are async
5. Connection Pooling: HTTP client with Polly policies

References

• MinIO Documentation
• Aliyun OSS Documentation
• Polly Resilience
• Clean Architecture