docs(api): Update README and architecture documentation for file handling and download URLs
- Expanded the API documentation to include detailed sections on file management endpoints, including upload, retrieval, sharing, and deletion. - Added a comprehensive overview of pre-signed URLs and access levels, clarifying the differences between public, private, and shared file access. - Introduced a new architecture section detailing the download URL generation flow and security considerations for pre-signed URLs. - Enhanced the README with examples and explanations to improve developer understanding of file access and management processes.
This commit is contained in:
Ho Ngoc Hai
@@ -64,14 +64,39 @@ dotnet run --project src/StorageService.API
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Files
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| `POST` | `/api/v1/files/upload` | Upload file |
|
||||
| `GET` | `/api/v1/files` | List user files |
|
||||
| `GET` | `/api/v1/files/{id}` | Get file by ID |
|
||||
| `GET` | `/api/v1/files/{id}/download-url` | Get pre-signed download URL |
|
||||
| `DELETE` | `/api/v1/files/{id}` | Delete file |
|
||||
| `GET` | `/api/v1/quota` | Get user quota |
|
||||
| `POST` | `/api/v1/files/upload` | Upload file via backend |
|
||||
| `GET` | `/api/v1/files` | List user files with pagination |
|
||||
| `GET` | `/api/v1/files/{id}` | Get file metadata by ID |
|
||||
| `GET` | `/api/v1/files/{id}/download-url` | Get download URL (presigned for private) |
|
||||
| `DELETE` | `/api/v1/files/{id}` | Soft delete file |
|
||||
|
||||
### Direct Upload (Recommended)
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| `POST` | `/api/v1/storage/sign-upload` | Get pre-signed PUT URL |
|
||||
| `POST` | `/api/v1/storage/confirm-upload` | Confirm upload and save metadata |
|
||||
|
||||
### File Sharing
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| `POST` | `/api/v1/files/{id}/share` | Create share link for file |
|
||||
| `GET` | `/api/v1/files/shared/{token}` | Access file via share token |
|
||||
| `DELETE` | `/api/v1/files/{id}/share` | Revoke share link |
|
||||
|
||||
### Quota
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| `GET` | `/api/v1/quota` | Get current user quota |
|
||||
| `GET` | `/api/v1/quota/me` | Get quota for authenticated user |
|
||||
|
||||
> **Note:** For private/shared files, `download-url` returns a pre-signed URL with AWS Signature V4. See [docs/en/README.md](docs/en/README.md) for details.
|
||||
|
||||
## Database Migrations / Migration Database
|
||||
|
||||
|
||||
@@ -400,6 +400,88 @@ WHERE id = '{file-id}';
|
||||
| **ConfirmUploadCommandHandler** | Handle confirm-upload with idempotency |
|
||||
| **SignedUrlController** | `/sign-upload` and `/confirm-upload` endpoints |
|
||||
|
||||
## Download URL Architecture
|
||||
|
||||
The Storage Service generates different download URLs based on file access levels.
|
||||
|
||||
### URL Generation Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant API as Storage Service
|
||||
participant MinIO
|
||||
|
||||
Client->>API: GET /api/v1/files/{id}/download-url
|
||||
API->>API: Validate ownership
|
||||
|
||||
alt accessLevel = Public
|
||||
API-->>Client: Direct URL (no signature)
|
||||
else accessLevel = Private/Shared
|
||||
API->>MinIO: GetPreSignedDownloadUrl()
|
||||
MinIO-->>API: Pre-signed URL (AWS Signature V4)
|
||||
API-->>Client: Pre-signed URL with expiration
|
||||
end
|
||||
```
|
||||
|
||||
### Access Level URL Types
|
||||
|
||||
| Access Level | Storage Prefix | URL Type | Expiration |
|
||||
|--------------|----------------|----------|------------|
|
||||
| **Public** | `public/` | Direct URL | Never |
|
||||
| **Private** | `private/` | Pre-signed URL | Configurable (default: 1 hour) |
|
||||
| **Shared** | `shared/` | Pre-signed URL | Configurable (default: 1 hour) |
|
||||
|
||||
### Pre-signed URL Structure
|
||||
|
||||
For private/shared files, URLs include AWS Signature Version 4 parameters:
|
||||
|
||||
```
|
||||
http://minio:9000/bucket/private/2026/01/13/{uuid}.pdf
|
||||
?X-Amz-Algorithm=AWS4-HMAC-SHA256
|
||||
&X-Amz-Credential={accessKey}%2F{date}%2F{region}%2Fs3%2Faws4_request
|
||||
&X-Amz-Date={timestamp}
|
||||
&X-Amz-Expires={seconds}
|
||||
&X-Amz-SignedHeaders=host
|
||||
&X-Amz-Signature={signature}
|
||||
```
|
||||
|
||||
| Parameter | Purpose |
|
||||
|-----------|---------|
|
||||
| `X-Amz-Algorithm` | Signing algorithm (always AWS4-HMAC-SHA256) |
|
||||
| `X-Amz-Credential` | Access key + scope (date/region/service) |
|
||||
| `X-Amz-Date` | Timestamp when signature was created |
|
||||
| `X-Amz-Expires` | URL validity in seconds |
|
||||
| `X-Amz-SignedHeaders` | Headers included in signature calculation |
|
||||
| `X-Amz-Signature` | Cryptographic signature to verify URL integrity |
|
||||
|
||||
### Security Considerations
|
||||
|
||||
1. **Time-Limited Access**: Pre-signed URLs expire after configured time
|
||||
2. **Tamper-Proof**: Any URL modification invalidates the signature
|
||||
3. **Credential Protection**: MinIO access keys never exposed to client
|
||||
4. **Unique URLs**: Each request generates a new signature
|
||||
|
||||
### GetDownloadUrl Implementation
|
||||
|
||||
```csharp
|
||||
public async Task<string> GetDownloadUrlAsync(StorageFile file)
|
||||
{
|
||||
if (file.AccessLevel == FileAccessLevel.Public)
|
||||
{
|
||||
// Direct URL for public files
|
||||
return $"{_settings.PublicEndpoint}/{file.BucketName}/{file.ObjectKey}";
|
||||
}
|
||||
|
||||
// Pre-signed URL for private/shared files
|
||||
return await _storageProvider.GetPreSignedDownloadUrlAsync(
|
||||
file.BucketName,
|
||||
file.ObjectKey,
|
||||
_settings.PreSignedUrlExpirationSeconds
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Multipart Upload Architecture (Large Files)
|
||||
|
||||
For files larger than 100MB, use Multipart Upload to upload in chunks.
|
||||
|
||||
@@ -183,6 +183,100 @@ Response:
|
||||
}
|
||||
```
|
||||
|
||||
## Pre-signed URLs & Access Levels
|
||||
|
||||
The Storage Service generates different types of download URLs based on the file's `accessLevel`.
|
||||
|
||||
### Access Level Overview
|
||||
|
||||
| Access Level | Storage Path Prefix | Download URL Type | Use Case |
|
||||
|--------------|---------------------|-------------------|----------|
|
||||
| **Public** | `public/` | Direct URL (no signature) | Public assets, images, CDN |
|
||||
| **Private** | `private/` | Pre-signed URL (AWS Signature V4) | User files, documents |
|
||||
| **Shared** | `shared/` | Pre-signed URL (AWS Signature V4) | Shared files with link |
|
||||
|
||||
### Public Files
|
||||
|
||||
Public files can be accessed directly without authentication:
|
||||
|
||||
```
|
||||
http://minio:9000/storage/public/2026/01/13/abc123.png
|
||||
```
|
||||
|
||||
**Characteristics:**
|
||||
- No signature required
|
||||
- URL never expires
|
||||
- Anyone with the URL can access
|
||||
- Best for: avatars, thumbnails, public downloads
|
||||
|
||||
### Private/Shared Files (Pre-signed URLs)
|
||||
|
||||
Private and shared files require a **pre-signed URL** with AWS Signature Version 4:
|
||||
|
||||
```
|
||||
http://minio:9000/storage/private/2026/01/13/xyz789.pdf
|
||||
?X-Amz-Algorithm=AWS4-HMAC-SHA256
|
||||
&X-Amz-Credential=minioadmin%2F20260113%2Fus-east-1%2Fs3%2Faws4_request
|
||||
&X-Amz-Date=20260113T180024Z
|
||||
&X-Amz-Expires=3600
|
||||
&X-Amz-SignedHeaders=host
|
||||
&X-Amz-Signature=2ce827d357d105fc1cf88240dee407e5ea72...
|
||||
```
|
||||
|
||||
**Signature Parameters Explained:**
|
||||
|
||||
| Parameter | Description | Example |
|
||||
|-----------|-------------|---------|
|
||||
| `X-Amz-Algorithm` | Signing algorithm | `AWS4-HMAC-SHA256` |
|
||||
| `X-Amz-Credential` | Access key + date/region/service | `minioadmin/20260113/us-east-1/s3/aws4_request` |
|
||||
| `X-Amz-Date` | Signature creation timestamp | `20260113T180024Z` |
|
||||
| `X-Amz-Expires` | URL validity in seconds | `3600` (1 hour) |
|
||||
| `X-Amz-SignedHeaders` | Headers included in signature | `host` |
|
||||
| `X-Amz-Signature` | The cryptographic signature | `2ce827d3...` |
|
||||
|
||||
**Security Benefits:**
|
||||
- URL expires after configured time (default: 1 hour)
|
||||
- Signature prevents URL tampering
|
||||
- No MinIO credentials exposed to client
|
||||
- Each URL is unique and single-use in practice
|
||||
|
||||
### Get Download URL API
|
||||
|
||||
```bash
|
||||
curl -X GET "http://localhost:5002/api/v1/files/{id}/download-url" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN"
|
||||
```
|
||||
|
||||
**Response for Public file:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"downloadUrl": "http://minio:9000/storage/public/2026/01/13/abc123.png",
|
||||
"expiresAt": null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response for Private file:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"downloadUrl": "http://minio:9000/storage/private/2026/01/13/xyz789.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...",
|
||||
"expiresAt": "2026-01-13T19:00:24Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `Storage__PreSignedUrlExpirationSeconds` | Pre-signed URL validity period | `3600` (1 hour) |
|
||||
|
||||
> **Note:** For very large files or slow connections, consider increasing the expiration time.
|
||||
|
||||
## Logical Folder Architecture (Data Sovereignty)
|
||||
|
||||
> ⚠️ **IMPORTANT**: Following the **Data Sovereignty** principle in microservices, folders are a **logical concept in the Database**, NOT dependent on bucket structure.
|
||||
|
||||
@@ -400,6 +400,88 @@ WHERE id = '{file-id}';
|
||||
| **ConfirmUploadCommandHandler** | Xử lý confirm-upload với idempotency |
|
||||
| **SignedUrlController** | Endpoints `/sign-upload` và `/confirm-upload` |
|
||||
|
||||
## Kiến Trúc Download URL
|
||||
|
||||
Storage Service tạo các loại download URL khác nhau dựa trên access level của file.
|
||||
|
||||
### Luồng Tạo URL
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant API as Storage Service
|
||||
participant MinIO
|
||||
|
||||
Client->>API: GET /api/v1/files/{id}/download-url
|
||||
API->>API: Validate ownership
|
||||
|
||||
alt accessLevel = Public
|
||||
API-->>Client: Direct URL (không có signature)
|
||||
else accessLevel = Private/Shared
|
||||
API->>MinIO: GetPreSignedDownloadUrl()
|
||||
MinIO-->>API: Pre-signed URL (AWS Signature V4)
|
||||
API-->>Client: Pre-signed URL với thời hạn
|
||||
end
|
||||
```
|
||||
|
||||
### Loại URL Theo Access Level
|
||||
|
||||
| Access Level | Storage Prefix | Loại URL | Hết hạn |
|
||||
|--------------|----------------|----------|------------|
|
||||
| **Public** | `public/` | Direct URL | Không bao giờ |
|
||||
| **Private** | `private/` | Pre-signed URL | Có thể cấu hình (mặc định: 1 giờ) |
|
||||
| **Shared** | `shared/` | Pre-signed URL | Có thể cấu hình (mặc định: 1 giờ) |
|
||||
|
||||
### Cấu Trúc Pre-signed URL
|
||||
|
||||
Với files private/shared, URLs bao gồm các tham số AWS Signature Version 4:
|
||||
|
||||
```
|
||||
http://minio:9000/bucket/private/2026/01/13/{uuid}.pdf
|
||||
?X-Amz-Algorithm=AWS4-HMAC-SHA256
|
||||
&X-Amz-Credential={accessKey}%2F{date}%2F{region}%2Fs3%2Faws4_request
|
||||
&X-Amz-Date={timestamp}
|
||||
&X-Amz-Expires={seconds}
|
||||
&X-Amz-SignedHeaders=host
|
||||
&X-Amz-Signature={signature}
|
||||
```
|
||||
|
||||
| Tham số | Mục đích |
|
||||
|---------|----------|
|
||||
| `X-Amz-Algorithm` | Thuật toán ký (luôn là AWS4-HMAC-SHA256) |
|
||||
| `X-Amz-Credential` | Access key + scope (date/region/service) |
|
||||
| `X-Amz-Date` | Thời điểm tạo signature |
|
||||
| `X-Amz-Expires` | Thời hạn URL tính bằng giây |
|
||||
| `X-Amz-SignedHeaders` | Headers được bao gồm trong tính toán signature |
|
||||
| `X-Amz-Signature` | Chữ ký mã hóa để xác minh tính toàn vẹn URL |
|
||||
|
||||
### Cân Nhắc Bảo Mật
|
||||
|
||||
1. **Truy Cập Có Thời Hạn**: Pre-signed URLs hết hạn sau thời gian cấu hình
|
||||
2. **Chống Chỉnh Sửa**: Bất kỳ sửa đổi URL nào đều làm signature không hợp lệ
|
||||
3. **Bảo Vệ Credentials**: Access keys MinIO không bao giờ lộ cho client
|
||||
4. **URLs Duy Nhất**: Mỗi request tạo ra signature mới
|
||||
|
||||
### Implementation GetDownloadUrl
|
||||
|
||||
```csharp
|
||||
public async Task<string> GetDownloadUrlAsync(StorageFile file)
|
||||
{
|
||||
if (file.AccessLevel == FileAccessLevel.Public)
|
||||
{
|
||||
// Direct URL cho files public
|
||||
return $"{_settings.PublicEndpoint}/{file.BucketName}/{file.ObjectKey}";
|
||||
}
|
||||
|
||||
// Pre-signed URL cho files private/shared
|
||||
return await _storageProvider.GetPreSignedDownloadUrlAsync(
|
||||
file.BucketName,
|
||||
file.ObjectKey,
|
||||
_settings.PreSignedUrlExpirationSeconds
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Kiến Trúc Multipart Upload (File Lớn)
|
||||
|
||||
Cho files lớn hơn 100MB, sử dụng Multipart Upload để upload theo chunks.
|
||||
|
||||
@@ -184,6 +184,100 @@ Response:
|
||||
}
|
||||
```
|
||||
|
||||
## Pre-signed URLs & Access Levels
|
||||
|
||||
Storage Service tạo các loại download URL khác nhau dựa trên `accessLevel` của file.
|
||||
|
||||
### Tổng Quan Access Level
|
||||
|
||||
| Access Level | Storage Path Prefix | Loại Download URL | Use Case |
|
||||
|--------------|---------------------|-------------------|----------|
|
||||
| **Public** | `public/` | Direct URL (không có signature) | Assets công khai, hình ảnh, CDN |
|
||||
| **Private** | `private/` | Pre-signed URL (AWS Signature V4) | Files user, documents |
|
||||
| **Shared** | `shared/` | Pre-signed URL (AWS Signature V4) | Files chia sẻ qua link |
|
||||
|
||||
### Files Public
|
||||
|
||||
Files public có thể truy cập trực tiếp không cần authentication:
|
||||
|
||||
```
|
||||
http://minio:9000/storage/public/2026/01/13/abc123.png
|
||||
```
|
||||
|
||||
**Đặc điểm:**
|
||||
- Không cần signature
|
||||
- URL không bao giờ hết hạn
|
||||
- Bất kỳ ai có URL đều có thể truy cập
|
||||
- Phù hợp cho: avatars, thumbnails, downloads công khai
|
||||
|
||||
### Files Private/Shared (Pre-signed URLs)
|
||||
|
||||
Files private và shared yêu cầu **pre-signed URL** với AWS Signature Version 4:
|
||||
|
||||
```
|
||||
http://minio:9000/storage/private/2026/01/13/xyz789.pdf
|
||||
?X-Amz-Algorithm=AWS4-HMAC-SHA256
|
||||
&X-Amz-Credential=minioadmin%2F20260113%2Fus-east-1%2Fs3%2Faws4_request
|
||||
&X-Amz-Date=20260113T180024Z
|
||||
&X-Amz-Expires=3600
|
||||
&X-Amz-SignedHeaders=host
|
||||
&X-Amz-Signature=2ce827d357d105fc1cf88240dee407e5ea72...
|
||||
```
|
||||
|
||||
**Giải Thích Các Tham Số Signature:**
|
||||
|
||||
| Tham số | Mô tả | Ví dụ |
|
||||
|---------|-------|---------|
|
||||
| `X-Amz-Algorithm` | Thuật toán ký | `AWS4-HMAC-SHA256` |
|
||||
| `X-Amz-Credential` | Access key + date/region/service | `minioadmin/20260113/us-east-1/s3/aws4_request` |
|
||||
| `X-Amz-Date` | Thời điểm tạo signature | `20260113T180024Z` |
|
||||
| `X-Amz-Expires` | Thời hạn URL (giây) | `3600` (1 giờ) |
|
||||
| `X-Amz-SignedHeaders` | Headers được ký | `host` |
|
||||
| `X-Amz-Signature` | Chữ ký mã hóa | `2ce827d3...` |
|
||||
|
||||
**Lợi Ích Bảo Mật:**
|
||||
- URL hết hạn sau thời gian cấu hình (mặc định: 1 giờ)
|
||||
- Signature ngăn chỉnh sửa URL
|
||||
- Không lộ credentials MinIO cho client
|
||||
- Mỗi URL là duy nhất và sử dụng một lần trong thực tế
|
||||
|
||||
### API Lấy Download URL
|
||||
|
||||
```bash
|
||||
curl -X GET "http://localhost:5002/api/v1/files/{id}/download-url" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN"
|
||||
```
|
||||
|
||||
**Response cho file Public:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"downloadUrl": "http://minio:9000/storage/public/2026/01/13/abc123.png",
|
||||
"expiresAt": null
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response cho file Private:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"downloadUrl": "http://minio:9000/storage/private/2026/01/13/xyz789.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...",
|
||||
"expiresAt": "2026-01-13T19:00:24Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cấu Hình
|
||||
|
||||
| Biến | Mô tả | Mặc định |
|
||||
|------|-------|----------|
|
||||
| `Storage__PreSignedUrlExpirationSeconds` | Thời hạn pre-signed URL | `3600` (1 giờ) |
|
||||
|
||||
> **Lưu ý:** Với files rất lớn hoặc kết nối chậm, hãy cân nhắc tăng thời hạn.
|
||||
|
||||
## Kiến Trúc Logical Folder (Data Sovereignty)
|
||||
|
||||
> ⚠️ **QUAN TRỌNG**: Theo nguyên tắc **Data Sovereignty** trong microservices, folder là **logical concept trong Database**, KHÔNG phụ thuộc vào bucket structure.
|
||||
|
||||
Reference in New Issue
Block a user