admin/pos-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
pos-system/microservices/services/storage-service-net/SERVICE_DOCS.md
Ho Ngoc Hai 76d75c753b Migrate
2026-05-23 18:37:02 +07:00

548 lines
26 KiB
Markdown
Raw Permalink Blame History

# StorageService - Service Documentation
> Auto-generated from source code audit on 2026-03-13
## Overview
**StorageService** is a file storage microservice providing S3-compatible object storage with support for direct uploads, multipart uploads, file sharing, file versioning, logical folders, and per-user storage quotas. It supports two storage backends: **MinIO** (primary) and **Aliyun OSS**.
- **Port**: 5002 (Development)
- **Database**: `storage_service` (PostgreSQL / Neon)
- **Storage Backend**: MinIO (S3-compatible) at `167.114.174.113:9000`
- **Cache**: Redis at `167.114.174.113:6379`
- **Default Bucket**: `goodgo`
- **Max File Size**: 100MB (configurable), validator limit 500MB
- **Auth**: JWT Bearer via IAM Service
- **Framework**: .NET 10.0, Clean Architecture + CQRS
---
## API Endpoints
### Files (`/api/v1/files`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| POST | `/api/v1/files/upload` | Bearer | Upload a file (max 100MB) |
| GET | `/api/v1/files` | Bearer | List user files (skip, take, search) |
| GET | `/api/v1/files/{fileId}` | Bearer | Get file metadata by ID |
| GET | `/api/v1/files/{fileId}/download-url` | Bearer | Get pre-signed download URL |
| GET | `/api/v1/files/{fileId}/cdn-url` | Bearer | Get CDN URL (public) or fallback pre-signed URL |
| DELETE | `/api/v1/files/{fileId}` | Bearer | Soft delete a file |
### Direct Upload - Pre-Signed URL (`/api/v1/storage`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| POST | `/api/v1/storage/sign-upload` | Bearer | Get pre-signed PUT URL for direct upload to MinIO |
| POST | `/api/v1/storage/confirm-upload` | Bearer | Confirm upload and save metadata after direct upload |
### Multipart Upload (`/api/v1/files/multipart`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| POST | `/api/v1/files/multipart/initiate` | Bearer | Initiate multipart upload session |
| POST | `/api/v1/files/multipart/upload-part` | Bearer | Upload a single part (max 20MB per part) |
| POST | `/api/v1/files/multipart/complete` | Bearer | Complete and merge all parts |
| DELETE | `/api/v1/files/multipart/abort` | Bearer | Abort and cleanup |
| GET | `/api/v1/files/multipart/{uploadId}` | Bearer | Get upload progress |
### File Sharing (`/api/v1/storage`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| POST | `/api/v1/storage/files/{fileId}/shares` | Bearer | Create share link (with optional password, expiry, max downloads) |
| GET | `/api/v1/storage/files/{fileId}/shares` | Bearer | Get all shares for a file |
| DELETE | `/api/v1/storage/shares/{shareId}` | Bearer | Revoke a share link |
| GET | `/api/v1/storage/shares/public/{token}` | Anonymous | Access shared file by token (public endpoint) |
### File Versioning (`/api/v1/storage/files/{fileId}/versions`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| GET | `/api/v1/storage/files/{fileId}/versions` | Bearer | Get all versions of a file |
| GET | `/api/v1/storage/files/{fileId}/versions/{versionId}/download` | Bearer | Get download URL for specific version |
| POST | `/api/v1/storage/files/{fileId}/versions/{versionId}/restore` | Bearer | Restore file to a specific version |
### Folders (`/api/v1/storage/folders`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| POST | `/api/v1/storage/folders` | Bearer | Create folder (root or child) |
| GET | `/api/v1/storage/folders` | Bearer | List folders (root or children of parentId) |
| GET | `/api/v1/storage/folders/{folderId}` | Bearer | Get folder by ID |
| PUT | `/api/v1/storage/folders/{folderId}` | Bearer | Rename folder (O(1) operation) |
| DELETE | `/api/v1/storage/folders/{folderId}` | Bearer | Soft delete folder |
### Quota (`/api/v1/quota`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| GET | `/api/v1/quota` | Bearer | Get current user's storage quota |
### Admin - Files (`/api/v1/admin/files`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| GET | `/api/v1/admin/files` | Admin/SuperAdmin | Get all files with filtering and pagination |
| DELETE | `/api/v1/admin/files/{fileId}` | Admin/SuperAdmin | Delete file (bypasses ownership, requires reason) |
### Admin - Quotas (`/api/v1/admin/quotas`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| GET | `/api/v1/admin/quotas` | Admin/SuperAdmin | Get all users' quotas with filtering |
| GET | `/api/v1/admin/quotas/{userId}` | Admin/SuperAdmin | Get quota for specific user |
| PUT | `/api/v1/admin/quotas/{userId}` | Admin/SuperAdmin | Update user quota limits |
### Admin - Shares (`/api/v1/admin/shares`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| GET | `/api/v1/admin/shares` | Admin/SuperAdmin | Get all file shares with filtering |
| DELETE | `/api/v1/admin/shares/{shareId}` | Admin/SuperAdmin | Revoke share (requires reason) |
### Admin - Statistics (`/api/v1/admin/statistics`)
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| GET | `/api/v1/admin/statistics` | Admin/SuperAdmin | Get aggregated storage statistics |
| GET | `/api/v1/admin/statistics/users-near-limit` | Admin/SuperAdmin | Get users with usage >= 80% |
### Health Checks
| Method | Route | Auth | Description |
|--------|-------|------|-------------|
| GET | `/health` | None | Full health check (incl. PostgreSQL) |
| GET | `/health/live` | None | Liveness probe (app running) |
| GET | `/health/ready` | None | Readiness probe |
---
## Commands
### UploadFileCommand
- **Input**: `Stream FileStream, string FileName, string ContentType, long FileSizeBytes, string UserId, string? TenantId, FileAccessLevel AccessLevel`
- **Logic**: Check file size limit -> Check user quota -> Generate object key (`{accessLevel}/{userId}/{date}/{uniqueId}_{filename}`) -> Upload to storage provider -> Save StorageFile entity -> Update quota -> SaveEntities
- **Validator**: FileName (required, max 255, no `..` `/` `\`), ContentType (required, max 100), FileSizeBytes (> 0, <= 500MB), UserId (required, max 128), TenantId (max 128)
- **Result**: `UploadFileResult(bool Success, Guid? FileId, string? ObjectKey, string? Error)`
### SignUploadCommand
- **Input**: `string UserId, string FileName, string ContentType, long FileSizeBytes, FileAccessLevel AccessLevel, string? TenantId`
- **Logic**: Validate file size -> Check user quota -> Generate object key -> Ensure bucket exists -> Generate pre-signed PUT URL
- **Validator**: Same rules as UploadFileCommand
- **Result**: `SignUploadResult(bool Success, string? UploadUrl, string? ObjectKey, DateTime? ExpiresAt, string? Error)`
### ConfirmUploadCommand
- **Input**: `string UserId, string ObjectKey, string FileName, string ContentType, long FileSizeBytes, FileAccessLevel AccessLevel, string? TenantId`
- **Logic**: Validate object key ownership (userId in path) -> Verify file exists in storage -> Idempotency check (existing file by objectKey) -> Double-check quota -> Create StorageFile entity -> Update quota -> Invalidate caches
- **Validator**: UserId (required, max 128), ObjectKey (required, max 1024), FileName (required, max 255), ContentType (required, max 100), FileSizeBytes (> 0)
- **Result**: `ConfirmUploadResult(bool Success, Guid? FileId, FileDto? Metadata, string? Error)`
### DeleteFileCommand
- **Input**: `Guid FileId, string UserId`
- **Logic**: Get file -> Check ownership -> Delete from storage provider (best-effort) -> Soft delete record -> Update quota -> Invalidate caches
- **Validator**: FileId (required), UserId (required, max 128)
- **Result**: `DeleteFileResult(bool Success, string? Error)`
### CreateFileShareCommand
- **Input**: `Guid FileId, string UserId, SharePermission Permission, string? SharedWith, string? Password, DateTime? ExpiresAt, int? MaxDownloads`
- **Logic**: Verify file exists + ownership -> Create FileShare entity (generates random token, hashes password with PBKDF2) -> Save -> Return share URL
- **Validator**: FileId (required), UserId (required, max 128), SharedWith (max 256), Password (min 4, max 100), MaxDownloads (> 0, <= 10000), ExpiresAt (must be future)
- **Result**: `CreateFileShareResult(bool Success, Guid? ShareId, string? ShareToken, string? ShareUrl, string? Error)`
### RevokeFileShareCommand
- **Input**: `Guid ShareId, string UserId`
- **Logic**: Get share -> Verify ownership -> Revoke (idempotent) -> Save
- **Result**: `RevokeFileShareResult(bool Success, string? Error)`
### InitiateMultipartUploadCommand
- **Input**: `string FileName, long FileSizeBytes, string ContentType, string UserId, int? ChunkSizeBytes, FileAccessLevel AccessLevel, string? TenantId`
- **Logic**: Validate file size -> Check quota -> Determine chunk size (default 5MB, min 5MB, max 100MB) -> Generate object key -> Initiate multipart upload at provider -> Create MultipartUpload entity (expiration: 24h) -> Save
- **Result**: `InitiateMultipartUploadResult(... UploadId, ProviderUploadId, ObjectKey, BucketName, TotalChunks, ChunkSizeBytes, ExpiresAt ...)`
### UploadPartCommand
- **Input**: `Guid UploadId, int PartNumber, Stream DataStream, string UserId`
- **Logic**: Get upload with parts -> Verify ownership -> Check status (InProgress) -> Check expiration -> Upload part to provider -> Add part to domain entity -> Save
- **Result**: `UploadPartResult(bool Success, string? ETag, string? Error)`
### CompleteMultipartUploadCommand
- **Input**: `Guid UploadId, string UserId`
- **Logic**: Get upload with parts -> Verify ownership -> Domain validation (all parts uploaded) -> Complete at provider -> Create StorageFile entity -> Update quota -> Save
- **Result**: `CompleteMultipartUploadResult(bool Success, Guid? FileId, string? ObjectKey, string? Error)`
### AbortMultipartUploadCommand
- **Input**: `Guid UploadId, string UserId`
- **Logic**: Get upload -> Verify ownership -> Abort at provider (best-effort) -> Mark as aborted in DB -> Save
- **Result**: `AbortMultipartUploadResult(bool Success, string? Error)`
### AdminDeleteFileCommand
- **Input**: `Guid FileId, string AdminUserId, string Reason`
- **Logic**: Same as DeleteFileCommand but bypasses ownership check. Logs admin action with reason.
- **Result**: `AdminDeleteFileResult(bool Success, string? DeletedFileName, string? FileOwnerUserId, string? Error)`
### AdminRevokeShareCommand
- **Input**: `Guid ShareId, string AdminUserId, string Reason`
- **Logic**: Get share -> Revoke -> Log admin action with reason -> Save
- **Result**: `AdminRevokeShareResult(bool Success, Guid? FileId, string? ShareOwnerUserId, string? Error)`
### UpdateUserQuotaCommand
- **Input**: `string TargetUserId, long MaxStorageBytes, int MaxFileCount, string? QuotaTier`
- **Logic**: Get or create quota -> Update limits (validates max >= current usage) -> Invalidate cache -> Save
- **Validator**: TargetUserId (required, max 128), MaxStorageBytes (> 0), MaxFileCount (> 0), QuotaTier (max 50)
- **Result**: `UpdateUserQuotaResult(bool Success, QuotaUpdatedDto? Data, string? Error)`
---
## Queries
### GetFileQuery
- **Input**: `Guid FileId, string UserId`
- **Logic**: Cache-aside pattern (10min TTL) -> Get from DB -> Check ownership
- **Result**: `FileDto?`
### GetUserFilesQuery
- **Input**: `string UserId, int Skip, int Take, string? SearchTerm`
- **Logic**: If search term: search by filename; otherwise: list by userId -> Count total -> Map to DTOs
- **Result**: `UserFilesResult(IReadOnlyList<FileDto> Files, int TotalCount)`
### GetUserQuotaQuery
- **Input**: `string UserId`
- **Logic**: Cache-aside pattern (5min TTL) -> GetOrCreate quota -> Map to DTO
- **Result**: `QuotaDto?`
### GetDownloadUrlQuery
- **Input**: `Guid FileId, string UserId, int ExpirationSeconds`
- **Logic**: Get file -> Check access (private files: owner only) -> Mark accessed -> Generate pre-signed download URL
- **Result**: `DownloadUrlResult(bool Success, string? Url, int? ExpiresInSeconds, string? Error)`
### GetFileSharesQuery
- **Input**: `Guid FileId, string UserId`
- **Logic**: Verify file ownership -> Get all shares for file -> Map to DTOs
- **Result**: `IEnumerable<FileShareDto>`
### AccessSharedFileQuery
- **Input**: `string Token, string? Password`
- **Logic**: Get share by token -> Validate (not expired/revoked/limit reached) -> Validate password (PBKDF2) -> Get file -> Generate pre-signed download URL -> Increment download count -> Save
- **Result**: `SharedFileAccessResult(bool Success, string? DownloadUrl, string? FileName, ...)`
### GetMultipartUploadProgressQuery
- **Input**: `Guid UploadId, string UserId`
- **Logic**: Get upload with parts -> Verify ownership -> Calculate progress
- **Result**: `MultipartUploadProgressResult?`
### AdminGetFilesQuery
- **Input**: `int PageNumber, int PageSize, string? UserId, string? AccessLevel, string? ContentType, DateTime? UploadedAfter/Before, string? SortBy, bool Descending`
- **Logic**: IgnoreQueryFilters (includes soft-deleted) -> Apply filters -> Sort -> Paginate
- **Result**: `AdminFilesResult(IReadOnlyList<AdminFileDto> Items, int TotalCount, int PageNumber, int PageSize, int TotalPages)`
### GetAllUsersQuotaQuery
- **Input**: `int PageNumber, int PageSize, string? QuotaTier, double? MinUsagePercentage, string? SortBy, bool Descending`
- **Logic**: Filter by tier, min usage% -> Sort -> Paginate
- **Result**: `AllUsersQuotaResult(IReadOnlyList<AdminQuotaDto> Items, ...)`
### GetStorageStatisticsQuery
- **Input**: (none)
- **Logic**: Aggregate stats: total users, total storage used/allocated, total files, avg usage%, users by tier, users near limit (>80%), users over limit (>=100%)
- **Result**: `StorageStatisticsDto`
### AdminGetSharesQuery
- **Input**: `int PageNumber, int PageSize, string? Status, string? SharedBy`
- **Logic**: Filter by status, sharedBy -> Paginate -> Order by CreatedAt desc
- **Result**: `AdminSharesResult(IReadOnlyList<AdminShareDto> Items, ...)`
---
## Domain Model
### StorageFile (Aggregate Root)
- **Table**: `storage_files`
- **Fields**: Id, FileName, BucketName, ObjectKey, ContentType, FileSizeBytes, UserId, TenantId, Provider (enum->string), AccessLevel (enum->string), UploadedAt, LastAccessedAt, ExpiresAt, Checksum, IsDeleted, DeletedAt
- **Behavior Methods**: `MarkAccessed()`, `UpdateAccessLevel(level)`, `Delete()` (soft), `SetExpiration(dt)`, `UpdateFromVersion(objectKey, size, contentType)`
- **Domain Events**: `FileUploadedDomainEvent` (on create), `FileDeletedDomainEvent` (on delete)
- **Query Filter**: `!IsDeleted` (global)
### UserStorageQuota (Aggregate Root)
- **Table**: `user_storage_quotas`
- **Fields**: Id, UserId, MaxStorageBytes (default 1GB), UsedStorageBytes, MaxFileCount (default 1000), CurrentFileCount, QuotaTier (default "free"), LastUpdatedAt, CreatedAt
- **Computed**: `RemainingStorageBytes`, `RemainingFileCount`, `UsagePercentage`
- **Behavior Methods**: `CanUpload(size)`, `AddUsage(size, count)`, `RemoveUsage(size, count)`, `UpdateLimits(max, maxFiles, tier)`, `RecalculateUsage(totalBytes, totalFiles)`
- **Domain Events**: `UserQuotaUpdatedDomainEvent` (on AddUsage/RemoveUsage)
- **Validation**: `UpdateLimits` throws if new max < current usage
### FileShare (Aggregate Root)
- **Table**: `file_shares`
- **Fields**: Id, FileId (FK -> storage_files), SharedBy, SharedWith, Permission (enum->string), ShareToken (unique, 32 random bytes base64), PasswordHash (PBKDF2-SHA256, 100K iterations), ExpiresAt, MaxDownloads, DownloadCount, Status (enum->string), CreatedAt, RevokedAt
- **Behavior Methods**: `IsValid()` (checks status, expiry, download limit), `ValidatePassword(pwd)`, `IncrementDownloadCount()`, `Revoke()`
- **Enums**: `SharePermission` (View=0, Download=1, Edit=2, Admin=3), `FileShareStatus` (Active=0, Expired=1, Revoked=2, LimitReached=3)
### FileVersion (Entity)
- **Table**: `file_versions`
- **Fields**: Id, FileId (FK -> storage_files), VersionNumber, ObjectKey, SizeBytes, ContentType, Checksum, CreatedAt, CreatedBy, Comment, IsCurrent
- **Behavior Methods**: `MarkAsCurrent()`, `UnmarkAsCurrent()`
### Folder (Aggregate Root)
- **Table**: `folders`
- **Fields**: Id, UserId, ParentId (FK self-referencing, restrict delete), Name, Path (materialized path e.g. `/parent/child/`), Level (0=root), CreatedAt, UpdatedAt, IsDeleted, DeletedAt
- **Behavior Methods**: `CreateRoot(userId, name)` (static), `CreateChild(name)`, `Rename(newName)`, `MoveTo(newParent)`, `Delete()` (soft)
- **Design Note**: Folders are LOGICAL only. Storage uses flat UUID keys. Rename/move is O(1) -- database-only.
- **Query Filter**: `!IsDeleted` (global)
### MultipartUpload (Aggregate Root)
- **Table**: `multipart_uploads`
- **Fields**: Id, UserId, FileName, TotalSizeBytes, ChunkSizeBytes, TotalChunks (computed), UploadedChunks, ProviderUploadId, BucketName, ObjectKey, ContentType, Status (enum->string), CreatedAt, CompletedAt, ExpiresAt
- **Behavior Methods**: `AddPart(partNum, etag, size)`, `Complete()` (validates all parts uploaded), `Abort()`, `MarkFailed()`, `IsExpired()`, `GetProgressPercentage()`
- **Domain Events**: `MultipartUploadInitiatedDomainEvent`, `MultipartUploadCompletedDomainEvent`, `MultipartUploadAbortedDomainEvent`
- **Child Entity**: `MultipartUploadPart` (Id, MultipartUploadId, PartNumber, ETag, SizeBytes, UploadedAt)
- **Enums**: `MultipartUploadStatus` (InProgress=0, Completed=1, Aborted=2, Failed=3, Expired=4)
---
## Database Schema
### Table: `storage_files`
| Column | Type | Constraints |
|--------|------|-------------|
| id | uuid | PK |
| file_name | varchar(255) | NOT NULL |
| bucket_name | varchar(100) | NOT NULL |
| object_key | varchar(500) | NOT NULL, UNIQUE |
| content_type | varchar(100) | NOT NULL |
| file_size_bytes | bigint | NOT NULL |
| user_id | varchar(100) | NOT NULL |
| tenant_id | varchar(100) | nullable |
| provider | varchar (string enum) | NOT NULL |
| access_level | varchar (string enum) | NOT NULL |
| uploaded_at | timestamp | NOT NULL |
| last_accessed_at | timestamp | nullable |
| expires_at | timestamp | nullable |
| checksum | varchar(100) | nullable |
| is_deleted | boolean | NOT NULL, default false |
| deleted_at | timestamp | nullable |
**Indexes**: object_key (unique), user_id, tenant_id, uploaded_at, is_deleted
**Query Filter**: `is_deleted = false`
### Table: `user_storage_quotas`
| Column | Type | Constraints |
|--------|------|-------------|
| id | uuid | PK |
| user_id | varchar(100) | NOT NULL, UNIQUE |
| max_storage_bytes | bigint | NOT NULL |
| used_storage_bytes | bigint | NOT NULL |
| max_file_count | int | NOT NULL |
| current_file_count | int | NOT NULL |
| quota_tier | varchar(50) | nullable |
| created_at | timestamp | NOT NULL |
| last_updated_at | timestamp | NOT NULL |
**Indexes**: user_id (unique)
### Table: `file_shares`
| Column | Type | Constraints |
|--------|------|-------------|
| id | uuid | PK |
| file_id | uuid | NOT NULL, FK -> storage_files (CASCADE) |
| shared_by | varchar(255) | NOT NULL |
| shared_with | varchar(255) | nullable |
| permission | varchar(50) (string enum) | NOT NULL |
| share_token | varchar(255) | NOT NULL, UNIQUE |
| password_hash | varchar(255) | nullable |
| expires_at | timestamp | nullable |
| max_downloads | int | nullable |
| download_count | int | NOT NULL, default 0 |
| status | varchar(50) (string enum) | NOT NULL |
| created_at | timestamp | NOT NULL |
| revoked_at | timestamp | nullable |
**Indexes**: file_id, share_token (unique), shared_with, shared_by
### Table: `file_versions`
| Column | Type | Constraints |
|--------|------|-------------|
| id | uuid | PK |
| file_id | uuid | NOT NULL, FK -> storage_files (CASCADE) |
| version_number | int | NOT NULL |
| object_key | varchar(500) | NOT NULL |
| size_bytes | bigint | NOT NULL |
| content_type | varchar(100) | NOT NULL |
| checksum | varchar(100) | nullable |
| created_at | timestamp | NOT NULL |
| created_by | varchar(255) | NOT NULL |
| comment | varchar(500) | nullable |
| is_current | boolean | default false |
**Indexes**: file_id, (file_id + version_number) unique, (file_id + is_current)
### Table: `folders`
| Column | Type | Constraints |
|--------|------|-------------|
| id | uuid | PK |
| user_id | varchar(255) | NOT NULL |
| parent_id | uuid | nullable, FK self (RESTRICT) |
| name | varchar(255) | NOT NULL |
| path | varchar(1000) | NOT NULL |
| level | int | NOT NULL |
| created_at | timestamp | NOT NULL |
| updated_at | timestamp | NOT NULL |
| is_deleted | boolean | NOT NULL, default false |
| deleted_at | timestamp | nullable |
**Indexes**: user_id, parent_id, (user_id + parent_id + name) unique filtered `is_deleted = false`, path
**Query Filter**: `is_deleted = false`
### Table: `multipart_uploads`
| Column | Type | Constraints |
|--------|------|-------------|
| id | uuid | PK |
| user_id | varchar(255) | NOT NULL |
| file_name | varchar(255) | NOT NULL |
| total_size_bytes | bigint | NOT NULL |
| chunk_size_bytes | int | NOT NULL |
| total_chunks | int | NOT NULL |
| uploaded_chunks | int | NOT NULL |
| upload_id | varchar(255) | NOT NULL (provider upload ID) |
| bucket_name | varchar(255) | NOT NULL |
| object_key | varchar(500) | NOT NULL |
| content_type | varchar(100) | NOT NULL |
| status | varchar(50) (string enum) | NOT NULL |
| created_at | timestamp | NOT NULL |
| completed_at | timestamp | nullable |
| expires_at | timestamp | NOT NULL |
**Indexes**: (user_id + status), upload_id, created_at
### Table: `multipart_upload_parts`
| Column | Type | Constraints |
|--------|------|-------------|
| id | uuid | PK |
| multipart_upload_id | uuid | NOT NULL, FK -> multipart_uploads (CASCADE) |
| part_number | int | NOT NULL |
| etag | varchar(255) | NOT NULL |
| size_bytes | bigint | NOT NULL |
| uploaded_at | timestamp | NOT NULL |
**Indexes**: (multipart_upload_id + part_number) unique
### Migrations
1. `20260112185402_InitialCreate` - storage_files, user_storage_quotas
2. `20260113155939_AddMultipartUploadTables` - multipart_uploads, multipart_upload_parts
3. `20260113170635_AddFileSharing` - file_shares
4. `20260113171238_AddFileVersioning` - file_versions
5. `20260113171612_AddFolders` - folders
---
## Dependencies
### NuGet Packages
**API Layer:**
- MediatR 12.4.1
- FluentValidation 11.11.0
- Microsoft.AspNetCore.Authentication.JwtBearer 10.0.1
- Swashbuckle.AspNetCore 7.2.0
- Asp.Versioning.Mvc 8.1.0
- AspNetCore.HealthChecks.NpgSql 8.0.2
- AspNetCore.HealthChecks.Redis 8.0.1
- Hellang.Middleware.ProblemDetails 6.5.1
- Serilog.AspNetCore 8.0.3
**Domain Layer:**
- MediatR.Contracts 2.0.1
**Infrastructure Layer:**
- Microsoft.EntityFrameworkCore 10.0.0
- Npgsql.EntityFrameworkCore.PostgreSQL 10.0.0
- MediatR 12.4.1
- Dapper 2.1.35
- Polly 8.5.0
- StackExchange.Redis 2.8.16
- Minio 6.0.4
- AWSSDK.S3 3.7.400 (for low-level multipart upload)
- Aliyun.OSS.SDK.NetCore 2.14.1
### External Services
- **IAM Service** (`http://localhost:5001`) - JWT token validation
- **MinIO** (`167.114.174.113:9000`) - Object storage backend
- **Redis** (`167.114.174.113:6379`) - Caching (quota, file metadata, user files)
- **PostgreSQL** (Neon) - Metadata persistence
---
## Configuration
### appsettings.json
| Section | Key | Default | Description |
|---------|-----|---------|-------------|
| ConnectionStrings | DefaultConnection | Neon PostgreSQL | Database connection |
| Storage | Provider | `minio` | Active provider (`minio` or `aliyun`) |
| Storage | DefaultBucket | `goodgo` | Default S3 bucket |
| Storage | PreSignedUrlExpirationSeconds | 3600 | Pre-signed URL TTL |
| Storage | MaxFileSizeBytes | 104857600 (100MB) | Max upload size |
| Storage:MinIO | Endpoint | `167.114.174.113:9000` | MinIO endpoint |
| Storage:MinIO | AccessKey | minioadmin | MinIO access key |
| Storage:MinIO | UseSSL | false | SSL for MinIO |
| Storage:MinIO | Region | us-east-1 | MinIO region |
| Storage:AliyunOSS | Endpoint | (empty) | Aliyun OSS endpoint |
| Redis | Host | `167.114.174.113` | Redis host |
| Redis | Port | 6379 | Redis port |
| Redis | Database | 0 | Redis database index |
| Jwt | Secret | goodgo-iam-service-secret-key-32chars! | JWT secret |
| Jwt | Issuer | goodgo-platform | JWT issuer |
| Jwt | AccessTokenExpiryMinutes | 15 | Token TTL |
| CDN | Enabled | false | CDN integration toggle |
| CDN | BaseUrl | (empty) | CDN base URL |
| IamService | BaseUrl | `http://localhost:5001` | IAM service URL |
| IamService | TimeoutSeconds | 30 | HTTP timeout |
---
## Architecture Notes
### Upload Patterns
1. **Server-Side Upload** (`POST /files/upload`): File goes through the API, uploaded to MinIO by the handler. Simple but doesn't scale for millions of concurrent uploads.
2. **Direct Client Upload** (`POST /storage/sign-upload` + `POST /storage/confirm-upload`): Client gets pre-signed PUT URL, uploads directly to MinIO, then confirms. Bypasses backend for file transfer -- highly scalable.
3. **Multipart Upload** (`POST /files/multipart/initiate` -> upload parts -> `POST /files/multipart/complete`): For large files, uploaded in 5MB chunks. 24-hour expiration.
### Caching Strategy
- **File metadata**: Redis cache-aside, 10min TTL
- **User quota**: Redis cache-aside, 5min TTL
- Cache invalidation on: file delete, upload confirm, quota update
### Security
- All user-facing endpoints require JWT Bearer auth
- File ownership validation on all operations (except admin endpoints)
- Share passwords hashed with PBKDF2-SHA256 (100K iterations, 16-byte salt)
- Share tokens: 32 random bytes, base64url-encoded
- Admin endpoints require `Admin` or `SuperAdmin` role
- Object key ownership validation via path structure (`{access}/{userId}/...`)
### Object Key Structure
```
{accessLevel}/{userId}/{yyyyMMdd}/{8-char-uuid}_{sanitized-filename}
```
- accessLevel: `public`, `private`, or `shared`
- File names sanitized (invalid chars removed, max 100 chars + extension)
### MediatR Pipeline
`LoggingBehavior` -> `ValidatorBehavior` -> `TransactionBehavior` -> Handler
### Resilience
- PostgreSQL: retry on failure (5 retries, 30s max delay)
- IAM HTTP client: Polly retry (3 retries, exponential backoff) + circuit breaker (5 failures, 30s break)
- Storage delete: best-effort (continues with soft delete even if provider delete fails)
- Multipart upload abort: best-effort provider cleanup
### Tests
- **Unit Tests**: StorageFile, FileShare, Folder domain entity tests; UploadFileCommandHandler, DeleteFileCommandHandler, SignUploadCommandHandler, FileShareCommandHandler tests
- **Functional Tests**: FilesApi, FileSharingApi, SignedUrlApi tests with CustomWebApplicationFactory (InMemory DB)
Reference in New Issue View Git Blame Copy Permalink