pos-system/microservices/docs/vi/architecture/storage-proposal.md

Đề Xuất Kiến Trúc Storage Service

Tài liệu này mô tả kiến trúc cho Storage Service, microservice quản lý lưu trữ file với hỗ trợ multi-provider.

Tổng Quan: Storage Service

Storage Service cung cấp:

• Multi-provider Storage - MinIO (S3-compatible) và Aliyun OSS
• Direct Client Upload - Pre-signed URLs để bypass backend (khuyến nghị)
• Multipart Upload - Hỗ trợ files lớn > 100MB
• Logical Folder Architecture - Folder trong DB, flat storage trong bucket
• Pre-signed URLs - Download/Upload URLs có thời hạn với AWS Signature V4
• User Quota Management - Giới hạn dung lượng và số file
• Inter-service Communication - Xác thực JWT qua IAM Service với caching
• CQRS Pattern - MediatR cho Commands/Queries
• Clean Architecture - Domain, Infrastructure, API layers

Note

Storage Service đã được triển khai với .NET 10, Clean Architecture tại services/storage-service-net/

---

1. Phạm Vi Storage Service

1.1 File Management (Quản Lý File)

A. Upload Operations ✅

• Direct Client Upload (pre-signed PUT URL) - Đã triển khai
• Proxy Upload qua backend - Đã triển khai
• Multipart Upload cho files lớn - Đã triển khai
• Access Level (Private, Public, Shared) - Đã triển khai

B. Download Operations ✅

• Pre-signed Download URLs - Đã triển khai
• Direct URLs cho Public files - Đã triển khai
• AWS Signature V4 security - Đã triển khai

C. File Lifecycle ✅

• Soft delete - Đã triển khai
• File metadata management - Đã triển khai
• User ownership validation - Đã triển khai

D. Folder Management ✅

• ✅ Logical folder hierarchy - Đã triển khai
• ✅ Folder CRUD operations - Đã triển khai
• ✅ Rename folder (O(1)) - Đã triển khai

1.2 Storage Infrastructure ✅

A. Multi-provider Support

• ✅ MinIO (S3-compatible) - Đã triển khai
• ✅ Aliyun OSS - Đã triển khai
• ✅ Runtime provider switching - Đã triển khai

B. Quota Management ✅

• ✅ Per-user storage limits - Đã triển khai
• ✅ File count limits - Đã triển khai
• ✅ Usage tracking - Đã triển khai

1.3 Advanced Features ✅

A. File Sharing ✅

• ✅ Share links với expiration - Đã triển khai
• ✅ Permission-based sharing (View, Download, Edit, Admin) - Đã triển khai
• ✅ Password-protected links - Đã triển khai
• ✅ Max downloads limit - Đã triển khai

B. File Versioning ✅

• ✅ Version history - Đã triển khai
• ✅ Rollback/Restore support - Đã triển khai
• ✅ Download specific version - Đã triển khai

C. Search & Discovery (Planned)

• 🔄 Full-text search trong metadata
• 🔄 Tag-based filtering
• 🔄 Advanced search queries

---

2. Kiến Trúc Module Structure (Thực Tế)

services/storage-service-net/
├── src/
│   ├── StorageService.API/              # Presentation Layer
│   │   ├── Controllers/                  # FilesController, SignedUrlController, QuotaController
│   │   ├── Application/                  # CQRS Commands, Queries, Handlers
│   │   │   ├── Commands/                 # UploadFile, SignUpload, ConfirmUpload, Delete
│   │   │   ├── Queries/                  # GetFile, GetFiles, GetDownloadUrl, GetQuota
│   │   │   └── Validators/               # FluentValidation validators
│   │   └── Program.cs                    # App entry point
│   ├── StorageService.Domain/           # Domain Layer
│   │   ├── AggregatesModel/              # StorageFile, UserStorageQuota
│   │   ├── Events/                       # FileUploadedEvent, FileDeletedEvent
│   │   ├── Exceptions/                   # FileNotFoundException, QuotaExceededException
│   │   └── SeedWork/                     # Entity, IAggregateRoot, IRepository
│   └── StorageService.Infrastructure/   # Infrastructure Layer
│       ├── Providers/                    # MinioStorageProvider, AliyunOssStorageProvider
│       ├── StorageServiceContext.cs      # DbContext
│       ├── Repositories/                 # FileRepository, QuotaRepository
│       └── Services/                     # IamServiceClient
├── tests/
│   ├── StorageService.UnitTests/
│   └── StorageService.FunctionalTests/
├── docs/
├── Dockerfile
└── StorageService.slnx

Sơ Đồ Kiến Trúc Clean Architecture

graph TD
    classDef base fill:#202020,stroke:#505050,color:#fff,stroke-width:1px;
    classDef core fill:#1a237e,stroke:#3949ab,color:#fff,stroke-width:1px;
    classDef newModule fill:#1b5e20,stroke:#43a047,color:#fff,stroke-width:1px;
    classDef database fill:#4a148c,stroke:#7b1fa2,color:#fff,stroke-width:1px;
    classDef storage fill:#b71c1c,stroke:#f44336,color:#fff,stroke-width:1px;

    STORAGE[Storage Service]:::core

    subgraph FileOps [File Operations]
        direction TB
        UPLOAD[Upload Handler]:::newModule
        DOWNLOAD[Download Handler]:::newModule
        MULTIPART[Multipart Upload]:::newModule
    end

    subgraph StorageBackend [Storage Backend]
        direction TB
        MINIO[MinIO Provider]:::storage
        OSS[Aliyun OSS Provider]:::storage
    end

    subgraph DataLayer [Data Layer]
        direction TB
        QUOTA[Quota Service]:::newModule
        FOLDER[Folder Service]:::newModule
    end

    DB[(PostgreSQL)]:::database
    IAMSVC[IAM Service]:::core

    STORAGE --> FileOps
    STORAGE --> DataLayer
    FileOps --> StorageBackend
    DataLayer -.-> DB
    STORAGE --> IAMSVC

---

3. Database Schema

3.1 Core Tables (Đã Triển Khai)

• storage_files: Metadata files (id, file_name, storage_key, content_type, size, access_level, provider)
• user_storage_quotas: Quota per user (max_storage, used_storage, max_files, current_files)

3.2 Extended Tables (Đã Triển Khai)

• folders: Logical folder hierarchy ✅
• file_shares: Share links và permissions ✅
• file_versions: Version history ✅
• multipart_uploads: Tracking upload sessions ✅
• multipart_upload_parts: Individual parts ✅

---

4. API Endpoints (Thực Tế)

4.1 Direct Upload APIs ✅ (Khuyến Nghị)

Method	Endpoint	Mô tả	Auth
POST	/api/v1/storage/sign-upload	Lấy pre-signed PUT URL để upload trực tiếp	✅
POST	/api/v1/storage/confirm-upload	Xác nhận upload và lưu metadata	✅

4.2 Legacy Upload APIs ✅ (Proxy)

Method	Endpoint	Mô tả	Auth
POST	/api/v1/files/upload	Upload file qua backend (tối đa 100MB)	✅

4.3 File Management APIs ✅

Method	Endpoint	Mô tả	Auth
GET	/api/v1/files	Danh sách files với phân trang	✅
GET	/api/v1/files/{id}	Lấy metadata file theo ID	✅
GET	/api/v1/files/{id}/download-url	Lấy pre-signed download URL	✅
DELETE	/api/v1/files/{id}	Xóa file (soft delete)	✅

4.4 Quota APIs ✅

Method	Endpoint	Mô tả	Auth
GET	/api/v1/quota	Lấy quota storage của user hiện tại	✅

4.5 Multipart Upload APIs ✅

Method	Endpoint	Mô tả	Auth
POST	/api/v1/files/multipart/initiate	Khởi tạo upload session	✅
POST	/api/v1/files/multipart/upload-part	Upload 1 chunk	✅
POST	/api/v1/files/multipart/complete	Hoàn thành và merge parts	✅
DELETE	/api/v1/files/multipart/abort	Hủy upload, cleanup	✅
GET	/api/v1/files/multipart/{uploadId}	Kiểm tra tiến độ	✅

4.6 Folder APIs ✅

Method	Endpoint	Mô tả	Auth
POST	/api/v1/storage/folders	Tạo folder mới	✅
GET	/api/v1/storage/folders	List folders (root hoặc children)	✅
GET	/api/v1/storage/folders/{id}	Lấy folder info	✅
PUT	/api/v1/storage/folders/{id}	Đổi tên folder (O(1))	✅
DELETE	/api/v1/storage/folders/{id}	Xóa folder (soft delete)	✅

4.7 File Sharing APIs ✅

Method	Endpoint	Mô tả	Auth
POST	/api/v1/storage/files/{id}/shares	Tạo share link	✅
GET	/api/v1/storage/files/{id}/shares	List shares của file	✅
DELETE	/api/v1/storage/shares/{id}	Thu hồi share link	✅
GET	/api/v1/storage/shares/public/{token}	Truy cập file được chia sẻ (public)	❌

4.8 File Versioning APIs ✅

Method	Endpoint	Mô tả	Auth
GET	/api/v1/storage/files/{id}/versions	Lấy danh sách versions	✅
GET	/api/v1/storage/files/{id}/versions/{verId}/download	Download version cụ thể	✅
POST	/api/v1/storage/files/{id}/versions/{verId}/restore	Khôi phục về version cũ	✅

---

5. Implementation Roadmap

Phase 1: Core Storage ✅ (Completed)

• ✅ Multi-provider storage (MinIO, Aliyun OSS)
• ✅ Proxy upload qua backend
• ✅ Pre-signed download URLs
• ✅ User quota management
• ✅ File CRUD operations
• ✅ Inter-service communication với IAM

Phase 2: Direct Upload ✅ (Completed)

• ✅ Pre-signed PUT URLs cho direct upload
• ✅ Confirm upload flow
• ✅ Access level support (Private, Public, Shared)
• ✅ AWS Signature V4 security

Phase 3: Large File Support ✅ (Completed)

• ✅ Multipart upload initiation
• ✅ Part-by-part upload
• ✅ Upload completion/abort
• ✅ Progress tracking

Phase 4: Logical Folders ✅ (Completed)

• ✅ Folder hierarchy trong database
• ✅ Folder CRUD APIs
• ✅ Rename folder O(1)
• ✅ Soft delete

Phase 5: File Sharing ✅ (Completed)

• ✅ Share link generation
• ✅ Expiring share links
• ✅ Password protection
• ✅ Permission levels (View, Download, Edit, Admin)
• ✅ Max downloads limit

Phase 6: File Versioning ✅ (Completed)

• ✅ Version tracking
• ✅ Restore/Rollback support
• ✅ Download specific versions

Phase 7: Search & Discovery (Planned)

• 🔄 Full-text search trong metadata
• 🔄 Tag-based filtering
• 🔄 Advanced search queries

---

6. Kiến Trúc Đặc Biệt

6.1 Direct Upload Pattern

sequenceDiagram
    participant Client
    participant Storage as Storage Service
    participant MinIO

    Client->>Storage: POST /sign-upload (metadata)
    Storage->>Storage: Validate JWT, Check Quota
    Storage-->>Client: Pre-signed PUT URL + ObjectKey
    Client->>MinIO: PUT file binary (direct)
    MinIO-->>Client: 200 OK
    Client->>Storage: POST /confirm-upload (objectKey)
    Storage->>MinIO: Verify file exists
    Storage->>Storage: Save metadata, Update quota
    Storage-->>Client: File metadata

Lợi ích:

• Backend không xử lý file binary → Giảm 90%+ tải
• Có thể xử lý hàng triệu uploads đồng thời
• Tốc độ upload nhanh hơn

6.2 Logical Folder Architecture

Layer	Trách nhiệm
Database	Quản lý folder hierarchy, paths, permissions
Bucket	Lưu trữ file binary với flat UUID keys

Lợi ích:

• Rename folder: O(1) - chỉ update DB
• Move file: O(1) - chỉ update folder_id
• UUID keys không predictable → Bảo mật cao
• Dễ migrate storage provider

---

7. Lợi Ích

Cho Developers

• ✅ Multi-provider abstraction
• ✅ Clean Architecture
• ✅ CQRS Pattern với MediatR
• ✅ Pre-signed URL handling

Cho End Users

• ✅ Fast direct uploads
• ✅ Large file support (multipart)
• ✅ Clear quota visibility
• ✅ Secure file access

---

Kết Luận

Storage Service là một microservice hoàn chỉnh cho file storage với:

• Direct Upload Pattern cho high scalability
• Multi-provider Support (MinIO, Aliyun OSS)
• Multipart Upload cho large files
• Logical Folder Architecture theo Data Sovereignty principles
• Pre-signed URLs với AWS Signature V4 security

Các tính năng tiếp theo (Folders, Sharing, Versioning) sẽ được triển khai trong các phase tới.