pos-system/microservices/services/storage-service-net/docs/en/README.md

Storage Service

A .NET 10 microservice for file storage management supporting MinIO and Aliyun OSS.

Features

• Multi-provider Storage: MinIO (S3-compatible) and Aliyun OSS
• Runtime Provider Switching: Switch between MinIO and Aliyun via environment variable
• Complete File CRUD: Upload, download, delete, list files
• Pre-signed URLs: Secure time-limited download/upload URLs
• User Quotas: Storage capacity and file count limits per user
• Inter-service Communication: JWT validation via IAM Service with caching

Quick Start

Prerequisites

• .NET 10 SDK
• Docker & Docker Compose
• PostgreSQL (or Neon)
• MinIO (or Aliyun OSS account)

Run with Docker

cd services/storage-service-net
docker-compose up -d

Access: http://localhost:5002/swagger

Run Locally

cd services/storage-service-net

# Install dependencies
dotnet restore

# Run migrations (first time)
dotnet ef database update --project src/StorageService.Infrastructure --startup-project src/StorageService.API

# Start the service
dotnet run --project src/StorageService.API

Configuration

Environment Variables

Variable	Description	Default
Storage__Provider	Provider selection: minio or aliyun	minio
Storage__DefaultBucket	Default bucket name	storage
Storage__MaxFileSizeBytes	Maximum file size	104857600 (100MB)
Storage__PreSignedUrlExpirationSeconds	Pre-signed URL expiration	3600

MinIO Configuration

Variable	Description	Default
Storage__MinIO__Endpoint	MinIO server endpoint	localhost:9000
Storage__MinIO__AccessKey	Access key	-
Storage__MinIO__SecretKey	Secret key	-
Storage__MinIO__UseSSL	Enable SSL	false

Aliyun OSS Configuration

Variable	Description	Default
Storage__AliyunOSS__Endpoint	OSS endpoint	-
Storage__AliyunOSS__AccessKeyId	Access key ID	-
Storage__AliyunOSS__AccessKeySecret	Access key secret	-
Storage__AliyunOSS__Region	OSS region	-

IAM Service Configuration

Variable	Description	Default
IamService__BaseUrl	IAM Service URL	http://localhost:5001
IamService__ServiceName	Service identifier	storage-service
IamService__TimeoutSeconds	Request timeout	30
IamService__CacheDurationSeconds	User info cache TTL	300
IamService__HealthCheckCacheDurationSeconds	Health check cache TTL	60

API Endpoints

Files (Legacy - Proxy Upload)

Method	Endpoint	Description
POST	/api/v1/files/upload	Upload a file via backend (max 100MB)
GET	/api/v1/files	List user's files with pagination
GET	/api/v1/files/{id}	Get file metadata by ID
GET	/api/v1/files/{id}/download-url	Get pre-signed download URL
DELETE	/api/v1/files/{id}	Delete a file (soft delete)

Direct Upload (Recommended for Scale)

Benefits of Direct Upload:

• Backend doesn't handle file binary → 90%+ load reduction
• Can handle millions of concurrent uploads
• Faster upload speed (no intermediary)

Method	Endpoint	Description
POST	/api/v1/storage/sign-upload	Get pre-signed PUT URL for direct upload
POST	/api/v1/storage/confirm-upload	Confirm upload and save metadata

Quota

Method	Endpoint	Description
GET	/api/v1/quota	Get current user's storage quota

Folders

Important: Folders are logical (database-only). File storage uses flat UUID keys. Rename/move operations are O(1).

Method	Endpoint	Description
POST	/api/v1/folders	Create new folder
GET	/api/v1/folders	List root folders
GET	/api/v1/folders/{id}	Get folder by ID with children
PUT	/api/v1/folders/{id}	Rename folder
DELETE	/api/v1/folders/{id}	Delete folder (soft delete)

File Versioning

Method	Endpoint	Description
GET	/api/v1/files/{id}/versions	List all versions of a file
GET	/api/v1/files/{id}/versions/{versionNumber}/download	Get download URL for specific version
POST	/api/v1/files/{id}/versions/{versionNumber}/restore	Restore file to specific version

File Sharing

Method	Endpoint	Description
POST	/api/v1/storage/files/{id}/shares	Create share link with options
GET	/api/v1/storage/files/{id}/shares	Get all shares for a file
DELETE	/api/v1/storage/shares/{id}	Revoke share link
GET	/api/v1/storage/shares/public/{token}	Access shared file (public, no auth)

Share Options:

• permission: View, Download, Edit, Admin
• password: Optional password protection
• expiresAt: Expiration date/time
• maxDownloads: Maximum download count

Admin API (Requires Role: Admin)

Note: These endpoints require Admin or SuperAdmin role.

Quota Management

Method	Endpoint	Description
GET	/api/v1/admin/quotas	Get all users' quotas
GET	/api/v1/admin/quotas/{userId}	Get quota for specific user
PUT	/api/v1/admin/quotas/{userId}	Update quota limits

Files Management

Method	Endpoint	Description
GET	/api/v1/admin/files	Get all files with filtering
DELETE	/api/v1/admin/files/{id}	Delete file for policy violation

Shares Management

Method	Endpoint	Description
GET	/api/v1/admin/shares	Get all shares
DELETE	/api/v1/admin/shares/{id}	Revoke share for violation

Statistics

Method	Endpoint	Description
GET	/api/v1/admin/statistics	Dashboard aggregated statistics
GET	/api/v1/admin/statistics/users-near-limit	Users near quota limit (>80%)

Direct Upload Example (Recommended)

Step 1: Get Pre-signed URL

curl -X POST "http://localhost:5002/api/v1/storage/sign-upload" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "fileName": "document.pdf",
    "fileSizeBytes": 1048576,
    "contentType": "application/pdf",
    "accessLevel": "private"
  }'

Response:

{
  "success": true,
  "data": {
    "uploadUrl": "http://minio:9000/storage/private/user123/20260113/abc12345_document.pdf?X-Amz-...",
    "objectKey": "private/user123/20260113/abc12345_document.pdf",
    "expiresAt": "2026-01-13T21:49:33Z"
  }
}

Step 2: Upload Directly to MinIO

curl -X PUT "${uploadUrl}" \
  -H "Content-Type: application/pdf" \
  --data-binary @document.pdf

Step 3: Confirm Upload

curl -X POST "http://localhost:5002/api/v1/storage/confirm-upload" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "objectKey": "private/user123/20260113/abc12345_document.pdf",
    "fileName": "document.pdf",
    "fileSizeBytes": 1048576,
    "contentType": "application/pdf",
    "accessLevel": "private"
  }'

Response:

{
  "success": true,
  "data": {
    "fileId": "550e8400-e29b-41d4-a716-446655440000",
    "metadata": {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "fileName": "document.pdf",
      "contentType": "application/pdf",
      "fileSizeBytes": 1048576,
      "provider": "MinIO",
      "accessLevel": "Private",
      "uploadedAt": "2026-01-13T20:49:33Z"
    }
  }
}

Pre-signed URLs & Access Levels

The Storage Service generates different types of download URLs based on the file's accessLevel.

Access Level Overview

Access Level	Storage Path Prefix	Download URL Type	Use Case
Public	public/	Direct URL (no signature)	Public assets, images, CDN
Private	private/	Pre-signed URL (AWS Signature V4)	User files, documents
Shared	shared/	Pre-signed URL (AWS Signature V4)	Shared files with link

Public Files

Public files can be accessed directly without authentication:

http://minio:9000/storage/public/2026/01/13/abc123.png

Characteristics:

• No signature required
• URL never expires
• Anyone with the URL can access
• Best for: avatars, thumbnails, public downloads

Private/Shared Files (Pre-signed URLs)

Private and shared files require a pre-signed URL with AWS Signature Version 4:

http://minio:9000/storage/private/2026/01/13/xyz789.pdf
  ?X-Amz-Algorithm=AWS4-HMAC-SHA256
  &X-Amz-Credential=minioadmin%2F20260113%2Fus-east-1%2Fs3%2Faws4_request
  &X-Amz-Date=20260113T180024Z
  &X-Amz-Expires=3600
  &X-Amz-SignedHeaders=host
  &X-Amz-Signature=2ce827d357d105fc1cf88240dee407e5ea72...

Signature Parameters Explained:

Parameter	Description	Example
X-Amz-Algorithm	Signing algorithm	AWS4-HMAC-SHA256
X-Amz-Credential	Access key + date/region/service	minioadmin/20260113/us-east-1/s3/aws4_request
X-Amz-Date	Signature creation timestamp	20260113T180024Z
X-Amz-Expires	URL validity in seconds	3600 (1 hour)
X-Amz-SignedHeaders	Headers included in signature	host
X-Amz-Signature	The cryptographic signature	2ce827d3...

Security Benefits:

• URL expires after configured time (default: 1 hour)
• Signature prevents URL tampering
• No MinIO credentials exposed to client
• Each URL is unique and single-use in practice

Get Download URL API

curl -X GET "http://localhost:5002/api/v1/files/{id}/download-url" \
  -H "Authorization: Bearer YOUR_TOKEN"

Response for Public file:

{
  "success": true,
  "data": {
    "downloadUrl": "http://minio:9000/storage/public/2026/01/13/abc123.png",
    "expiresAt": null
  }
}

Response for Private file:

{
  "success": true,
  "data": {
    "downloadUrl": "http://minio:9000/storage/private/2026/01/13/xyz789.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...",
    "expiresAt": "2026-01-13T19:00:24Z"
  }
}

Configuration

Variable	Description	Default
Storage__PreSignedUrlExpirationSeconds	Pre-signed URL validity period	3600 (1 hour)

Note: For very large files or slow connections, consider increasing the expiration time.

Logical Folder Architecture (Data Sovereignty)

⚠️ IMPORTANT: Following the Data Sovereignty principle in microservices, folders are a logical concept in the Database, NOT dependent on bucket structure.

Design Principles

Storage Service owns its own data model:

• Database manages logical folder structure (hierarchy, paths, permissions)
• Bucket only stores file binaries with UUID keys (flat structure)

❌ Anti-pattern (DON'T DO THIS)

Bucket structure based on user paths:
goodgo/
├── users/john/documents/report.pdf     ← BAD
├── users/mary/images/photo.jpg         ← BAD
└── users/bob/work/presentation.pptx    ← BAD

PROBLEMS:
- Renaming folder = Moving millions of files (slow + risky)
- Predictable paths = Vulnerable to attacks
- Doesn't scale with millions of users

✅ Correct Architecture (Logical Separation)

1️⃣ Database: Logical Structure

-- Folders table: Manages folder tree
CREATE TABLE folders (
    id UUID PRIMARY KEY,
    user_id VARCHAR(255) NOT NULL,
    parent_id UUID REFERENCES folders(id),
    name VARCHAR(255) NOT NULL,
    path VARCHAR(1000) NOT NULL,    -- Example: /docs/work/2024
    created_at TIMESTAMP
);

-- Files table: Links to logical folder
CREATE TABLE files (
    id UUID PRIMARY KEY,
    user_id VARCHAR(255) NOT NULL,
    folder_id UUID REFERENCES folders(id),  -- Logical folder
    file_name VARCHAR(255) NOT NULL,
    storage_key VARCHAR(500) UNIQUE,        -- Physical key (UUID)
    size_bytes BIGINT,
    content_type VARCHAR(100),
    access_level VARCHAR(20),
    created_at TIMESTAMP
);

2️⃣ Bucket: Flat Physical Storage

Files are stored with UUID-based keys following pattern:

{prefix}/{year}/{month}/{day}/{uuid}.{ext}

Example:
goodgo/
├── private/2026/01/13/d290f1ee6c544b0190e6d701748f0851.pdf
├── private/2026/01/13/a7f3b2c19d8e4f01bcd5e92f7a4d8b63.jpg
├── public/2026/01/14/f9e4c1a2b7d36e5f8a0c4d9e2b1f7a8c.png
└── shared/2026/01/15/c3d8f2e1a9b47c6e5d0f8a3b2e1c7d9f.docx

NO user/folder structure in bucket!
All are flat UUID keys.

3️⃣ Storage Key Generation

// Automatically generate physical key (NOT based on folder)
public string GenerateStorageKey(FileAccessLevel access, string fileName)
{
    var now = DateTime.UtcNow;
    var prefix = access == FileAccessLevel.Public ? "public" : "private";
    var uuid = Guid.NewGuid().ToString("N");
    var ext = Path.GetExtension(fileName);
    
    // Pattern: {prefix}/{year}/{month}/{day}/{uuid}{ext}
    return $"{prefix}/{now:yyyy}/{now:MM}/{now:dd}/{uuid}{ext}";
}

Benefits

Operation	Database (Logic)	Bucket (Physical)	Performance
Create folder	INSERT 1 row	Nothing	Instant
Rename folder	UPDATE 1 row	Nothing	O(1) - Instant
Move file	UPDATE File.folder_id	Nothing	O(1) - Instant
Delete folder	Cascade delete	Queue cleanup	Background job

Compared to Anti-pattern:

• Rename folder: O(1) vs O(n) - 1000x faster
• Security: UUID keys are unpredictable
• Migration: Easy to switch storage provider

Workflow Examples

Upload file to folder:

# 1. Create folder (Database only)
POST /api/v1/folders
{
  "name": "documents",
  "parentId": null
}

# 2. Upload file to folder
POST /api/v1/storage/sign-upload
{
  "fileName": "report.pdf",
  "folderId": "folder-uuid-123",    # Logical folder
  "fileSizeBytes": 1048576
}

# Response: Physical key DOES NOT contain folder name
{
  "uploadUrl": "...",
  "objectKey": "private/2026/01/13/d290f1ee6c544b0190e6d701748f0851.pdf"
}

List files in folder:

GET /api/v1/folders/{folderId}/files

# Database query: SELECT * FROM files WHERE folder_id = {folderId}
# Client sees: /documents/report.pdf (logical path)
# Bucket stores: private/2026/01/13/{uuid}.pdf (physical key)

📚 Technical Details: See ARCHITECTURE.md for deeper understanding of this pattern.

Legacy Upload Example (Via Backend)

curl -X POST "http://localhost:5002/api/v1/files/upload" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -F "file=@document.pdf"

Inter-Service Communication

The service communicates with IAM Service for user validation. This section explains the configuration required for proper inter-service communication in Docker environments.

Docker Network Configuration

When running in Docker Compose, services communicate via Docker network using container names:

# docker-compose.yml
storage-service:
  environment:
    # IMPORTANT: Use container name for inter-service communication
    - IamService__BaseUrl=http://iam-service-net:8080
    - IamService__ServiceName=storage-service

Note

: The URL http://iam-service-net:8080 uses:

• iam-service-net = Docker container name (not localhost)
• 8080 = Internal container port (not exposed port 5001)

JWT Token Issuer Configuration

To ensure JWT tokens work across services, IAM Service must use a fixed issuer URI:

# IAM Service docker-compose.yml
iam-service-net:
  environment:
    - IdentityServer__IssuerUri=http://iam-service

This ensures tokens issued by IAM Service have a consistent iss claim regardless of how the client accesses it.

Communication Flow

┌─────────────────┐     JWT Token (iss: http://iam-service)     ┌─────────────────┐
│  Storage        │ ──────────────────────────────────────────► │  IAM Service    │
│  Service        │ ◄────────────────────────────────────────── │                 │
│  :8080          │        User info + Roles/Permissions        │  :8080          │
└─────────────────┘                                             └─────────────────┘
        │                      Docker Network: goodgo-network          │
        └──────────────────────────────────────────────────────────────┘

Headers & Caching

• Headers: Authorization: Bearer <token>, X-Service-Name: storage-service
• User Info Cache: 5 minutes (300 seconds)
• Health Check Cache: 1 minute (60 seconds)
• Resilience: Polly retry (3x) + circuit breaker

Available Methods (IIamServiceClient)

Method	Description
ValidateUserAsync	Validate JWT token and get user info
GetUserByIdAsync	Get user by ID
GetUserRolesAsync	Get user's assigned roles
GetUserPermissionsAsync	Get user's permissions
HasPermissionAsync	Check if user has specific permission
HasRoleAsync	Check if user has specific role
CheckHealthAsync	Check IAM Service health status

Troubleshooting

Issue	Cause	Solution
401 Unauthorized	JWT issuer mismatch	Set IdentityServer__IssuerUri in IAM Service
Connection refused	Wrong URL or container not running	Use container name, not localhost
Name resolution failed	Container not in same network	Ensure both services use goodgo-network
Timeout	IAM Service slow/unhealthy	Check IAM health: curl http://iam-service-net:8080/health

Database Migrations

# Create new migration
dotnet ef migrations add MigrationName \
  --project src/StorageService.Infrastructure \
  --startup-project src/StorageService.API

# Apply migrations
dotnet ef database update \
  --project src/StorageService.Infrastructure \
  --startup-project src/StorageService.API

Testing

# Run all tests
dotnet test

# Run with coverage
dotnet test --collect:"XPlat Code Coverage"

Project Structure

services/storage-service-net/
├── src/
│   ├── StorageService.API/           # Controllers, Commands, Queries
│   ├── StorageService.Domain/        # Entities, Repository interfaces
│   └── StorageService.Infrastructure/# Providers, DbContext, Repositories
├── tests/
│   ├── StorageService.UnitTests/
│   └── StorageService.FunctionalTests/
├── docs/
│   ├── en/                           # English documentation
│   └── vi/                           # Vietnamese documentation
├── docker-compose.yml
├── Dockerfile
└── README.md

License

MIT