Files
Ho Ngoc Hai 76d75c753b Migrate
2026-05-23 18:37:02 +07:00

616 lines
19 KiB
Markdown
Raw Permalink Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Storage Service
> A .NET 10 microservice for file storage management supporting MinIO and Aliyun OSS.
## Features
- **Multi-provider Storage**: MinIO (S3-compatible) and Aliyun OSS
- **Runtime Provider Switching**: Switch between MinIO and Aliyun via environment variable
- **Complete File CRUD**: Upload, download, delete, list files
- **Pre-signed URLs**: Secure time-limited download/upload URLs
- **User Quotas**: Storage capacity and file count limits per user
- **Inter-service Communication**: JWT validation via IAM Service with caching
## Quick Start
### Prerequisites
- .NET 10 SDK
- Docker & Docker Compose
- PostgreSQL (or Neon)
- MinIO (or Aliyun OSS account)
### Run with Docker
```bash
cd services/storage-service-net
docker-compose up -d
```
Access: http://localhost:5002/swagger
### Run Locally
```bash
cd services/storage-service-net
# Install dependencies
dotnet restore
# Run migrations (first time)
dotnet ef database update --project src/StorageService.Infrastructure --startup-project src/StorageService.API
# Start the service
dotnet run --project src/StorageService.API
```
## Configuration
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `Storage__Provider` | Provider selection: `minio` or `aliyun` | `minio` |
| `Storage__DefaultBucket` | Default bucket name | `storage` |
| `Storage__MaxFileSizeBytes` | Maximum file size | `104857600` (100MB) |
| `Storage__PreSignedUrlExpirationSeconds` | Pre-signed URL expiration | `3600` |
### MinIO Configuration
| Variable | Description | Default |
|----------|-------------|---------|
| `Storage__MinIO__Endpoint` | MinIO server endpoint | `localhost:9000` |
| `Storage__MinIO__AccessKey` | Access key | - |
| `Storage__MinIO__SecretKey` | Secret key | - |
| `Storage__MinIO__UseSSL` | Enable SSL | `false` |
### Aliyun OSS Configuration
| Variable | Description | Default |
|----------|-------------|---------|
| `Storage__AliyunOSS__Endpoint` | OSS endpoint | - |
| `Storage__AliyunOSS__AccessKeyId` | Access key ID | - |
| `Storage__AliyunOSS__AccessKeySecret` | Access key secret | - |
| `Storage__AliyunOSS__Region` | OSS region | - |
### IAM Service Configuration
| Variable | Description | Default |
|----------|-------------|---------|
| `IamService__BaseUrl` | IAM Service URL | `http://localhost:5001` |
| `IamService__ServiceName` | Service identifier | `storage-service` |
| `IamService__TimeoutSeconds` | Request timeout | `30` |
| `IamService__CacheDurationSeconds` | User info cache TTL | `300` |
| `IamService__HealthCheckCacheDurationSeconds` | Health check cache TTL | `60` |
## API Endpoints
### Files (Legacy - Proxy Upload)
| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/files/upload` | Upload a file via backend (max 100MB) |
| `GET` | `/api/v1/files` | List user's files with pagination |
| `GET` | `/api/v1/files/{id}` | Get file metadata by ID |
| `GET` | `/api/v1/files/{id}/download-url` | Get pre-signed download URL |
| `DELETE` | `/api/v1/files/{id}` | Delete a file (soft delete) |
### Direct Upload (Recommended for Scale)
> **Benefits of Direct Upload:**
> - Backend doesn't handle file binary → 90%+ load reduction
> - Can handle millions of concurrent uploads
> - Faster upload speed (no intermediary)
| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/storage/sign-upload` | Get pre-signed PUT URL for direct upload |
| `POST` | `/api/v1/storage/confirm-upload` | Confirm upload and save metadata |
### Quota
| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/quota` | Get current user's storage quota |
### Folders
> **Important:** Folders are logical (database-only). File storage uses flat UUID keys. Rename/move operations are O(1).
| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/folders` | Create new folder |
| `GET` | `/api/v1/folders` | List root folders |
| `GET` | `/api/v1/folders/{id}` | Get folder by ID with children |
| `PUT` | `/api/v1/folders/{id}` | Rename folder |
| `DELETE` | `/api/v1/folders/{id}` | Delete folder (soft delete) |
### File Versioning
| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/files/{id}/versions` | List all versions of a file |
| `GET` | `/api/v1/files/{id}/versions/{versionNumber}/download` | Get download URL for specific version |
| `POST` | `/api/v1/files/{id}/versions/{versionNumber}/restore` | Restore file to specific version |
### File Sharing
| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/storage/files/{id}/shares` | Create share link with options |
| `GET` | `/api/v1/storage/files/{id}/shares` | Get all shares for a file |
| `DELETE` | `/api/v1/storage/shares/{id}` | Revoke share link |
| `GET` | `/api/v1/storage/shares/public/{token}` | Access shared file (public, no auth) |
**Share Options:**
- `permission`: View, Download, Edit, Admin
- `password`: Optional password protection
- `expiresAt`: Expiration date/time
- `maxDownloads`: Maximum download count
### Admin API (Requires Role: Admin)
> **Note:** These endpoints require `Admin` or `SuperAdmin` role.
#### Quota Management
| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/admin/quotas` | Get all users' quotas |
| `GET` | `/api/v1/admin/quotas/{userId}` | Get quota for specific user |
| `PUT` | `/api/v1/admin/quotas/{userId}` | Update quota limits |
#### Files Management
| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/admin/files` | Get all files with filtering |
| `DELETE` | `/api/v1/admin/files/{id}` | Delete file for policy violation |
#### Shares Management
| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/admin/shares` | Get all shares |
| `DELETE` | `/api/v1/admin/shares/{id}` | Revoke share for violation |
#### Statistics
| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/admin/statistics` | Dashboard aggregated statistics |
| `GET` | `/api/v1/admin/statistics/users-near-limit` | Users near quota limit (>80%) |
## Direct Upload Example (Recommended)
### Step 1: Get Pre-signed URL
```bash
curl -X POST "http://localhost:5002/api/v1/storage/sign-upload" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"fileName": "document.pdf",
"fileSizeBytes": 1048576,
"contentType": "application/pdf",
"accessLevel": "private"
}'
```
Response:
```json
{
"success": true,
"data": {
"uploadUrl": "http://minio:9000/storage/private/user123/20260113/abc12345_document.pdf?X-Amz-...",
"objectKey": "private/user123/20260113/abc12345_document.pdf",
"expiresAt": "2026-01-13T21:49:33Z"
}
}
```
### Step 2: Upload Directly to MinIO
```bash
curl -X PUT "${uploadUrl}" \
-H "Content-Type: application/pdf" \
--data-binary @document.pdf
```
### Step 3: Confirm Upload
```bash
curl -X POST "http://localhost:5002/api/v1/storage/confirm-upload" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"objectKey": "private/user123/20260113/abc12345_document.pdf",
"fileName": "document.pdf",
"fileSizeBytes": 1048576,
"contentType": "application/pdf",
"accessLevel": "private"
}'
```
Response:
```json
{
"success": true,
"data": {
"fileId": "550e8400-e29b-41d4-a716-446655440000",
"metadata": {
"id": "550e8400-e29b-41d4-a716-446655440000",
"fileName": "document.pdf",
"contentType": "application/pdf",
"fileSizeBytes": 1048576,
"provider": "MinIO",
"accessLevel": "Private",
"uploadedAt": "2026-01-13T20:49:33Z"
}
}
}
```
## Pre-signed URLs & Access Levels
The Storage Service generates different types of download URLs based on the file's `accessLevel`.
### Access Level Overview
| Access Level | Storage Path Prefix | Download URL Type | Use Case |
|--------------|---------------------|-------------------|----------|
| **Public** | `public/` | Direct URL (no signature) | Public assets, images, CDN |
| **Private** | `private/` | Pre-signed URL (AWS Signature V4) | User files, documents |
| **Shared** | `shared/` | Pre-signed URL (AWS Signature V4) | Shared files with link |
### Public Files
Public files can be accessed directly without authentication:
```
http://minio:9000/storage/public/2026/01/13/abc123.png
```
**Characteristics:**
- No signature required
- URL never expires
- Anyone with the URL can access
- Best for: avatars, thumbnails, public downloads
### Private/Shared Files (Pre-signed URLs)
Private and shared files require a **pre-signed URL** with AWS Signature Version 4:
```
http://minio:9000/storage/private/2026/01/13/xyz789.pdf
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=minioadmin%2F20260113%2Fus-east-1%2Fs3%2Faws4_request
&X-Amz-Date=20260113T180024Z
&X-Amz-Expires=3600
&X-Amz-SignedHeaders=host
&X-Amz-Signature=2ce827d357d105fc1cf88240dee407e5ea72...
```
**Signature Parameters Explained:**
| Parameter | Description | Example |
|-----------|-------------|---------|
| `X-Amz-Algorithm` | Signing algorithm | `AWS4-HMAC-SHA256` |
| `X-Amz-Credential` | Access key + date/region/service | `minioadmin/20260113/us-east-1/s3/aws4_request` |
| `X-Amz-Date` | Signature creation timestamp | `20260113T180024Z` |
| `X-Amz-Expires` | URL validity in seconds | `3600` (1 hour) |
| `X-Amz-SignedHeaders` | Headers included in signature | `host` |
| `X-Amz-Signature` | The cryptographic signature | `2ce827d3...` |
**Security Benefits:**
- URL expires after configured time (default: 1 hour)
- Signature prevents URL tampering
- No MinIO credentials exposed to client
- Each URL is unique and single-use in practice
### Get Download URL API
```bash
curl -X GET "http://localhost:5002/api/v1/files/{id}/download-url" \
-H "Authorization: Bearer YOUR_TOKEN"
```
**Response for Public file:**
```json
{
"success": true,
"data": {
"downloadUrl": "http://minio:9000/storage/public/2026/01/13/abc123.png",
"expiresAt": null
}
}
```
**Response for Private file:**
```json
{
"success": true,
"data": {
"downloadUrl": "http://minio:9000/storage/private/2026/01/13/xyz789.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...",
"expiresAt": "2026-01-13T19:00:24Z"
}
}
```
### Configuration
| Variable | Description | Default |
|----------|-------------|---------|
| `Storage__PreSignedUrlExpirationSeconds` | Pre-signed URL validity period | `3600` (1 hour) |
> **Note:** For very large files or slow connections, consider increasing the expiration time.
## Logical Folder Architecture (Data Sovereignty)
> ⚠️ **IMPORTANT**: Following the **Data Sovereignty** principle in microservices, folders are a **logical concept in the Database**, NOT dependent on bucket structure.
### Design Principles
Storage Service owns its own data model:
- **Database** manages logical folder structure (hierarchy, paths, permissions)
- **Bucket** only stores file binaries with UUID keys (flat structure)
### ❌ Anti-pattern (DON'T DO THIS)
```
Bucket structure based on user paths:
goodgo/
├── users/john/documents/report.pdf ← BAD
├── users/mary/images/photo.jpg ← BAD
└── users/bob/work/presentation.pptx ← BAD
PROBLEMS:
- Renaming folder = Moving millions of files (slow + risky)
- Predictable paths = Vulnerable to attacks
- Doesn't scale with millions of users
```
### ✅ Correct Architecture (Logical Separation)
#### 1⃣ Database: Logical Structure
```sql
-- Folders table: Manages folder tree
CREATE TABLE folders (
id UUID PRIMARY KEY,
user_id VARCHAR(255) NOT NULL,
parent_id UUID REFERENCES folders(id),
name VARCHAR(255) NOT NULL,
path VARCHAR(1000) NOT NULL, -- Example: /docs/work/2024
created_at TIMESTAMP
);
-- Files table: Links to logical folder
CREATE TABLE files (
id UUID PRIMARY KEY,
user_id VARCHAR(255) NOT NULL,
folder_id UUID REFERENCES folders(id), -- Logical folder
file_name VARCHAR(255) NOT NULL,
storage_key VARCHAR(500) UNIQUE, -- Physical key (UUID)
size_bytes BIGINT,
content_type VARCHAR(100),
access_level VARCHAR(20),
created_at TIMESTAMP
);
```
#### 2⃣ Bucket: Flat Physical Storage
Files are stored with **UUID-based keys** following pattern:
```
{prefix}/{year}/{month}/{day}/{uuid}.{ext}
Example:
goodgo/
├── private/2026/01/13/d290f1ee6c544b0190e6d701748f0851.pdf
├── private/2026/01/13/a7f3b2c19d8e4f01bcd5e92f7a4d8b63.jpg
├── public/2026/01/14/f9e4c1a2b7d36e5f8a0c4d9e2b1f7a8c.png
└── shared/2026/01/15/c3d8f2e1a9b47c6e5d0f8a3b2e1c7d9f.docx
NO user/folder structure in bucket!
All are flat UUID keys.
```
#### 3⃣ Storage Key Generation
```csharp
// Automatically generate physical key (NOT based on folder)
public string GenerateStorageKey(FileAccessLevel access, string fileName)
{
var now = DateTime.UtcNow;
var prefix = access == FileAccessLevel.Public ? "public" : "private";
var uuid = Guid.NewGuid().ToString("N");
var ext = Path.GetExtension(fileName);
// Pattern: {prefix}/{year}/{month}/{day}/{uuid}{ext}
return $"{prefix}/{now:yyyy}/{now:MM}/{now:dd}/{uuid}{ext}";
}
```
### Benefits
| Operation | Database (Logic) | Bucket (Physical) | Performance |
|-----------|------------------|-------------------|-------------|
| **Create folder** | INSERT 1 row | Nothing | Instant |
| **Rename folder** | UPDATE 1 row | Nothing | O(1) - Instant |
| **Move file** | UPDATE File.folder_id | Nothing | O(1) - Instant |
| **Delete folder** | Cascade delete | Queue cleanup | Background job |
**Compared to Anti-pattern:**
- Rename folder: **O(1)** vs O(n) - 1000x faster
- Security: UUID keys are unpredictable
- Migration: Easy to switch storage provider
### Workflow Examples
**Upload file to folder:**
```bash
# 1. Create folder (Database only)
POST /api/v1/folders
{
"name": "documents",
"parentId": null
}
# 2. Upload file to folder
POST /api/v1/storage/sign-upload
{
"fileName": "report.pdf",
"folderId": "folder-uuid-123", # Logical folder
"fileSizeBytes": 1048576
}
# Response: Physical key DOES NOT contain folder name
{
"uploadUrl": "...",
"objectKey": "private/2026/01/13/d290f1ee6c544b0190e6d701748f0851.pdf"
}
```
**List files in folder:**
```bash
GET /api/v1/folders/{folderId}/files
# Database query: SELECT * FROM files WHERE folder_id = {folderId}
# Client sees: /documents/report.pdf (logical path)
# Bucket stores: private/2026/01/13/{uuid}.pdf (physical key)
```
> 📚 **Technical Details:** See [ARCHITECTURE.md](./ARCHITECTURE.md) for deeper understanding of this pattern.
## Legacy Upload Example (Via Backend)
```bash
curl -X POST "http://localhost:5002/api/v1/files/upload" \
-H "Authorization: Bearer YOUR_TOKEN" \
-F "file=@document.pdf"
```
## Inter-Service Communication
The service communicates with IAM Service for user validation. This section explains the configuration required for proper inter-service communication in Docker environments.
### Docker Network Configuration
When running in Docker Compose, services communicate via Docker network using container names:
```yaml
# docker-compose.yml
storage-service:
environment:
# IMPORTANT: Use container name for inter-service communication
- IamService__BaseUrl=http://iam-service-net:8080
- IamService__ServiceName=storage-service
```
> **Note**: The URL `http://iam-service-net:8080` uses:
> - `iam-service-net` = Docker container name (not `localhost`)
> - `8080` = Internal container port (not exposed port `5001`)
### JWT Token Issuer Configuration
To ensure JWT tokens work across services, IAM Service must use a fixed issuer URI:
```yaml
# IAM Service docker-compose.yml
iam-service-net:
environment:
- IdentityServer__IssuerUri=http://iam-service
```
This ensures tokens issued by IAM Service have a consistent `iss` claim regardless of how the client accesses it.
### Communication Flow
```
┌─────────────────┐ JWT Token (iss: http://iam-service) ┌─────────────────┐
│ Storage │ ──────────────────────────────────────────► │ IAM Service │
│ Service │ ◄────────────────────────────────────────── │ │
│ :8080 │ User info + Roles/Permissions │ :8080 │
└─────────────────┘ └─────────────────┘
│ Docker Network: goodgo-network │
└──────────────────────────────────────────────────────────────┘
```
### Headers & Caching
- **Headers**: `Authorization: Bearer <token>`, `X-Service-Name: storage-service`
- **User Info Cache**: 5 minutes (300 seconds)
- **Health Check Cache**: 1 minute (60 seconds)
- **Resilience**: Polly retry (3x) + circuit breaker
### Available Methods (IIamServiceClient)
| Method | Description |
|--------|-------------|
| `ValidateUserAsync` | Validate JWT token and get user info |
| `GetUserByIdAsync` | Get user by ID |
| `GetUserRolesAsync` | Get user's assigned roles |
| `GetUserPermissionsAsync` | Get user's permissions |
| `HasPermissionAsync` | Check if user has specific permission |
| `HasRoleAsync` | Check if user has specific role |
| `CheckHealthAsync` | Check IAM Service health status |
### Troubleshooting
| Issue | Cause | Solution |
|-------|-------|----------|
| `401 Unauthorized` | JWT issuer mismatch | Set `IdentityServer__IssuerUri` in IAM Service |
| `Connection refused` | Wrong URL or container not running | Use container name, not `localhost` |
| `Name resolution failed` | Container not in same network | Ensure both services use `goodgo-network` |
| `Timeout` | IAM Service slow/unhealthy | Check IAM health: `curl http://iam-service-net:8080/health` |
## Database Migrations
```bash
# Create new migration
dotnet ef migrations add MigrationName \
--project src/StorageService.Infrastructure \
--startup-project src/StorageService.API
# Apply migrations
dotnet ef database update \
--project src/StorageService.Infrastructure \
--startup-project src/StorageService.API
```
## Testing
```bash
# Run all tests
dotnet test
# Run with coverage
dotnet test --collect:"XPlat Code Coverage"
```
## Project Structure
```
services/storage-service-net/
├── src/
│ ├── StorageService.API/ # Controllers, Commands, Queries
│ ├── StorageService.Domain/ # Entities, Repository interfaces
│ └── StorageService.Infrastructure/# Providers, DbContext, Repositories
├── tests/
│ ├── StorageService.UnitTests/
│ └── StorageService.FunctionalTests/
├── docs/
│ ├── en/ # English documentation
│ └── vi/ # Vietnamese documentation
├── docker-compose.yml
├── Dockerfile
└── README.md
```
## License
MIT