# Storage Service

> A .NET 10 microservice for file storage management supporting MinIO and Aliyun OSS.

## Features

- **Multi-provider Storage**: MinIO (S3-compatible) and Aliyun OSS
- **Runtime Provider Switching**: Switch between MinIO and Aliyun via environment variable
- **Complete File CRUD**: Upload, download, delete, list files
- **Pre-signed URLs**: Secure time-limited download/upload URLs
- **User Quotas**: Storage capacity and file count limits per user
- **Inter-service Communication**: JWT validation via IAM Service with caching

## Quick Start

### Prerequisites

- .NET 10 SDK
- Docker & Docker Compose
- PostgreSQL (or Neon)
- MinIO (or Aliyun OSS account)

### Run with Docker

```bash
cd services/storage-service-net
docker-compose up -d
```

Access: http://localhost:5002/swagger

### Run Locally

```bash
cd services/storage-service-net

# Install dependencies
dotnet restore

# Run migrations (first time)
dotnet ef database update --project src/StorageService.Infrastructure --startup-project src/StorageService.API

# Start the service
dotnet run --project src/StorageService.API
```

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `Storage__Provider` | Provider selection: `minio` or `aliyun` | `minio` |
| `Storage__DefaultBucket` | Default bucket name | `storage` |
| `Storage__MaxFileSizeBytes` | Maximum file size | `104857600` (100MB) |
| `Storage__PreSignedUrlExpirationSeconds` | Pre-signed URL expiration | `3600` |

### MinIO Configuration

| Variable | Description | Default |
|----------|-------------|---------|
| `Storage__MinIO__Endpoint` | MinIO server endpoint | `localhost:9000` |
| `Storage__MinIO__AccessKey` | Access key | - |
| `Storage__MinIO__SecretKey` | Secret key | - |
| `Storage__MinIO__UseSSL` | Enable SSL | `false` |

### Aliyun OSS Configuration

| Variable | Description | Default |
|----------|-------------|---------|
| `Storage__AliyunOSS__Endpoint` | OSS endpoint | - |
| `Storage__AliyunOSS__AccessKeyId` | Access key ID | - |
| `Storage__AliyunOSS__AccessKeySecret` | Access key secret | - |
| `Storage__AliyunOSS__Region` | OSS region | - |

### IAM Service Configuration

| Variable | Description | Default |
|----------|-------------|---------|
| `IamService__BaseUrl` | IAM Service URL | `http://localhost:5001` |
| `IamService__ServiceName` | Service identifier | `storage-service` |
| `IamService__TimeoutSeconds` | Request timeout | `30` |
| `IamService__CacheDurationSeconds` | User info cache TTL | `300` |
| `IamService__HealthCheckCacheDurationSeconds` | Health check cache TTL | `60` |

## API Endpoints

### Files (Legacy - Proxy Upload)

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/files/upload` | Upload a file via backend (max 100MB) |
| `GET` | `/api/v1/files` | List user's files with pagination |
| `GET` | `/api/v1/files/{id}` | Get file metadata by ID |
| `GET` | `/api/v1/files/{id}/download-url` | Get pre-signed download URL |
| `DELETE` | `/api/v1/files/{id}` | Delete a file (soft delete) |

### Direct Upload (Recommended for Scale)

> **Benefits of Direct Upload:**
> - Backend doesn't handle file binary → 90%+ load reduction
> - Can handle millions of concurrent uploads
> - Faster upload speed (no intermediary)

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/storage/sign-upload` | Get pre-signed PUT URL for direct upload |
| `POST` | `/api/v1/storage/confirm-upload` | Confirm upload and save metadata |

### Quota

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/quota` | Get current user's storage quota |

### Folders

> **Important:** Folders are logical (database-only). File storage uses flat UUID keys. Rename/move operations are O(1).

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/folders` | Create new folder |
| `GET` | `/api/v1/folders` | List root folders |
| `GET` | `/api/v1/folders/{id}` | Get folder by ID with children |
| `PUT` | `/api/v1/folders/{id}` | Rename folder |
| `DELETE` | `/api/v1/folders/{id}` | Delete folder (soft delete) |

### File Versioning

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/files/{id}/versions` | List all versions of a file |
| `GET` | `/api/v1/files/{id}/versions/{versionNumber}/download` | Get download URL for specific version |
| `POST` | `/api/v1/files/{id}/versions/{versionNumber}/restore` | Restore file to specific version |

### File Sharing

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/api/v1/storage/files/{id}/shares` | Create share link with options |
| `GET` | `/api/v1/storage/files/{id}/shares` | Get all shares for a file |
| `DELETE` | `/api/v1/storage/shares/{id}` | Revoke share link |
| `GET` | `/api/v1/storage/shares/public/{token}` | Access shared file (public, no auth) |

**Share Options:**
- `permission`: View, Download, Edit, Admin
- `password`: Optional password protection
- `expiresAt`: Expiration date/time
- `maxDownloads`: Maximum download count

### Admin API (Requires Role: Admin)

> **Note:** These endpoints require `Admin` or `SuperAdmin` role.

#### Quota Management

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/admin/quotas` | Get all users' quotas |
| `GET` | `/api/v1/admin/quotas/{userId}` | Get quota for specific user |
| `PUT` | `/api/v1/admin/quotas/{userId}` | Update quota limits |

#### Files Management

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/admin/files` | Get all files with filtering |
| `DELETE` | `/api/v1/admin/files/{id}` | Delete file for policy violation |

#### Shares Management

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/admin/shares` | Get all shares |
| `DELETE` | `/api/v1/admin/shares/{id}` | Revoke share for violation |

#### Statistics

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/v1/admin/statistics` | Dashboard aggregated statistics |
| `GET` | `/api/v1/admin/statistics/users-near-limit` | Users near quota limit (>80%) |

## Direct Upload Example (Recommended)

### Step 1: Get Pre-signed URL

```bash
curl -X POST "http://localhost:5002/api/v1/storage/sign-upload" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "fileName": "document.pdf",
    "fileSizeBytes": 1048576,
    "contentType": "application/pdf",
    "accessLevel": "private"
  }'
```

Response:
```json
{
  "success": true,
  "data": {
    "uploadUrl": "http://minio:9000/storage/private/user123/20260113/abc12345_document.pdf?X-Amz-...",
    "objectKey": "private/user123/20260113/abc12345_document.pdf",
    "expiresAt": "2026-01-13T21:49:33Z"
  }
}
```

### Step 2: Upload Directly to MinIO

```bash
curl -X PUT "${uploadUrl}" \
  -H "Content-Type: application/pdf" \
  --data-binary @document.pdf
```

### Step 3: Confirm Upload

```bash
curl -X POST "http://localhost:5002/api/v1/storage/confirm-upload" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "objectKey": "private/user123/20260113/abc12345_document.pdf",
    "fileName": "document.pdf",
    "fileSizeBytes": 1048576,
    "contentType": "application/pdf",
    "accessLevel": "private"
  }'
```

Response:
```json
{
  "success": true,
  "data": {
    "fileId": "550e8400-e29b-41d4-a716-446655440000",
    "metadata": {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "fileName": "document.pdf",
      "contentType": "application/pdf",
      "fileSizeBytes": 1048576,
      "provider": "MinIO",
      "accessLevel": "Private",
      "uploadedAt": "2026-01-13T20:49:33Z"
    }
  }
}
```

## Pre-signed URLs & Access Levels

The Storage Service generates different types of download URLs based on the file's `accessLevel`.

### Access Level Overview

| Access Level | Storage Path Prefix | Download URL Type | Use Case |
|--------------|---------------------|-------------------|----------|
| **Public** | `public/` | Direct URL (no signature) | Public assets, images, CDN |
| **Private** | `private/` | Pre-signed URL (AWS Signature V4) | User files, documents |
| **Shared** | `shared/` | Pre-signed URL (AWS Signature V4) | Shared files with link |

### Public Files

Public files can be accessed directly without authentication:

```
http://minio:9000/storage/public/2026/01/13/abc123.png
```

**Characteristics:**
- No signature required
- URL never expires
- Anyone with the URL can access
- Best for: avatars, thumbnails, public downloads

### Private/Shared Files (Pre-signed URLs)

Private and shared files require a **pre-signed URL** with AWS Signature Version 4:

```
http://minio:9000/storage/private/2026/01/13/xyz789.pdf
  ?X-Amz-Algorithm=AWS4-HMAC-SHA256
  &X-Amz-Credential=minioadmin%2F20260113%2Fus-east-1%2Fs3%2Faws4_request
  &X-Amz-Date=20260113T180024Z
  &X-Amz-Expires=3600
  &X-Amz-SignedHeaders=host
  &X-Amz-Signature=2ce827d357d105fc1cf88240dee407e5ea72...
```

**Signature Parameters Explained:**

| Parameter | Description | Example |
|-----------|-------------|---------|
| `X-Amz-Algorithm` | Signing algorithm | `AWS4-HMAC-SHA256` |
| `X-Amz-Credential` | Access key + date/region/service | `minioadmin/20260113/us-east-1/s3/aws4_request` |
| `X-Amz-Date` | Signature creation timestamp | `20260113T180024Z` |
| `X-Amz-Expires` | URL validity in seconds | `3600` (1 hour) |
| `X-Amz-SignedHeaders` | Headers included in signature | `host` |
| `X-Amz-Signature` | The cryptographic signature | `2ce827d3...` |

**Security Benefits:**
- URL expires after configured time (default: 1 hour)
- Signature prevents URL tampering
- No MinIO credentials exposed to client
- Each URL is unique and single-use in practice

### Get Download URL API

```bash
curl -X GET "http://localhost:5002/api/v1/files/{id}/download-url" \
  -H "Authorization: Bearer YOUR_TOKEN"
```

**Response for Public file:**
```json
{
  "success": true,
  "data": {
    "downloadUrl": "http://minio:9000/storage/public/2026/01/13/abc123.png",
    "expiresAt": null
  }
}
```

**Response for Private file:**
```json
{
  "success": true,
  "data": {
    "downloadUrl": "http://minio:9000/storage/private/2026/01/13/xyz789.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...",
    "expiresAt": "2026-01-13T19:00:24Z"
  }
}
```

### Configuration

| Variable | Description | Default |
|----------|-------------|---------|
| `Storage__PreSignedUrlExpirationSeconds` | Pre-signed URL validity period | `3600` (1 hour) |

> **Note:** For very large files or slow connections, consider increasing the expiration time.

## Logical Folder Architecture (Data Sovereignty)

> ⚠️ **IMPORTANT**: Following the **Data Sovereignty** principle in microservices, folders are a **logical concept in the Database**, NOT dependent on bucket structure.

### Design Principles

Storage Service owns its own data model:
- **Database** manages logical folder structure (hierarchy, paths, permissions)
- **Bucket** only stores file binaries with UUID keys (flat structure)

### ❌ Anti-pattern (DON'T DO THIS)

```
Bucket structure based on user paths:
goodgo/
├── users/john/documents/report.pdf     ← BAD
├── users/mary/images/photo.jpg         ← BAD
└── users/bob/work/presentation.pptx    ← BAD

PROBLEMS:
- Renaming folder = Moving millions of files (slow + risky)
- Predictable paths = Vulnerable to attacks
- Doesn't scale with millions of users
```

### ✅ Correct Architecture (Logical Separation)

#### 1️⃣ Database: Logical Structure

```sql
-- Folders table: Manages folder tree
CREATE TABLE folders (
    id UUID PRIMARY KEY,
    user_id VARCHAR(255) NOT NULL,
    parent_id UUID REFERENCES folders(id),
    name VARCHAR(255) NOT NULL,
    path VARCHAR(1000) NOT NULL,    -- Example: /docs/work/2024
    created_at TIMESTAMP
);

-- Files table: Links to logical folder
CREATE TABLE files (
    id UUID PRIMARY KEY,
    user_id VARCHAR(255) NOT NULL,
    folder_id UUID REFERENCES folders(id),  -- Logical folder
    file_name VARCHAR(255) NOT NULL,
    storage_key VARCHAR(500) UNIQUE,        -- Physical key (UUID)
    size_bytes BIGINT,
    content_type VARCHAR(100),
    access_level VARCHAR(20),
    created_at TIMESTAMP
);
```

#### 2️⃣ Bucket: Flat Physical Storage

Files are stored with **UUID-based keys** following pattern:

```
{prefix}/{year}/{month}/{day}/{uuid}.{ext}

Example:
goodgo/
├── private/2026/01/13/d290f1ee6c544b0190e6d701748f0851.pdf
├── private/2026/01/13/a7f3b2c19d8e4f01bcd5e92f7a4d8b63.jpg
├── public/2026/01/14/f9e4c1a2b7d36e5f8a0c4d9e2b1f7a8c.png
└── shared/2026/01/15/c3d8f2e1a9b47c6e5d0f8a3b2e1c7d9f.docx

NO user/folder structure in bucket!
All are flat UUID keys.
```

#### 3️⃣ Storage Key Generation

```csharp
// Automatically generate physical key (NOT based on folder)
public string GenerateStorageKey(FileAccessLevel access, string fileName)
{
    var now = DateTime.UtcNow;
    var prefix = access == FileAccessLevel.Public ? "public" : "private";
    var uuid = Guid.NewGuid().ToString("N");
    var ext = Path.GetExtension(fileName);
    
    // Pattern: {prefix}/{year}/{month}/{day}/{uuid}{ext}
    return $"{prefix}/{now:yyyy}/{now:MM}/{now:dd}/{uuid}{ext}";
}
```

### Benefits

| Operation | Database (Logic) | Bucket (Physical) | Performance |
|-----------|------------------|-------------------|-------------|
| **Create folder** | INSERT 1 row | Nothing | Instant |
| **Rename folder** | UPDATE 1 row | Nothing | O(1) - Instant |
| **Move file** | UPDATE File.folder_id | Nothing | O(1) - Instant |
| **Delete folder** | Cascade delete | Queue cleanup | Background job |

**Compared to Anti-pattern:**
- Rename folder: **O(1)** vs O(n) - 1000x faster
- Security: UUID keys are unpredictable
- Migration: Easy to switch storage provider

### Workflow Examples

**Upload file to folder:**
```bash
# 1. Create folder (Database only)
POST /api/v1/folders
{
  "name": "documents",
  "parentId": null
}

# 2. Upload file to folder
POST /api/v1/storage/sign-upload
{
  "fileName": "report.pdf",
  "folderId": "folder-uuid-123",    # Logical folder
  "fileSizeBytes": 1048576
}

# Response: Physical key DOES NOT contain folder name
{
  "uploadUrl": "...",
  "objectKey": "private/2026/01/13/d290f1ee6c544b0190e6d701748f0851.pdf"
}
```

**List files in folder:**
```bash
GET /api/v1/folders/{folderId}/files

# Database query: SELECT * FROM files WHERE folder_id = {folderId}
# Client sees: /documents/report.pdf (logical path)
# Bucket stores: private/2026/01/13/{uuid}.pdf (physical key)
```

> 📚 **Technical Details:** See [ARCHITECTURE.md](./ARCHITECTURE.md) for deeper understanding of this pattern.

## Legacy Upload Example (Via Backend)

```bash
curl -X POST "http://localhost:5002/api/v1/files/upload" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -F "file=@document.pdf"
```


## Inter-Service Communication

The service communicates with IAM Service for user validation. This section explains the configuration required for proper inter-service communication in Docker environments.

### Docker Network Configuration

When running in Docker Compose, services communicate via Docker network using container names:

```yaml
# docker-compose.yml
storage-service:
  environment:
    # IMPORTANT: Use container name for inter-service communication
    - IamService__BaseUrl=http://iam-service-net:8080
    - IamService__ServiceName=storage-service
```

> **Note**: The URL `http://iam-service-net:8080` uses:
> - `iam-service-net` = Docker container name (not `localhost`)
> - `8080` = Internal container port (not exposed port `5001`)

### JWT Token Issuer Configuration

To ensure JWT tokens work across services, IAM Service must use a fixed issuer URI:

```yaml
# IAM Service docker-compose.yml
iam-service-net:
  environment:
    - IdentityServer__IssuerUri=http://iam-service
```

This ensures tokens issued by IAM Service have a consistent `iss` claim regardless of how the client accesses it.

### Communication Flow

```
┌─────────────────┐     JWT Token (iss: http://iam-service)     ┌─────────────────┐
│  Storage        │ ──────────────────────────────────────────► │  IAM Service    │
│  Service        │ ◄────────────────────────────────────────── │                 │
│  :8080          │        User info + Roles/Permissions        │  :8080          │
└─────────────────┘                                             └─────────────────┘
        │                      Docker Network: goodgo-network          │
        └──────────────────────────────────────────────────────────────┘
```

### Headers & Caching

- **Headers**: `Authorization: Bearer <token>`, `X-Service-Name: storage-service`
- **User Info Cache**: 5 minutes (300 seconds)
- **Health Check Cache**: 1 minute (60 seconds)
- **Resilience**: Polly retry (3x) + circuit breaker

### Available Methods (IIamServiceClient)

| Method | Description |
|--------|-------------|
| `ValidateUserAsync` | Validate JWT token and get user info |
| `GetUserByIdAsync` | Get user by ID |
| `GetUserRolesAsync` | Get user's assigned roles |
| `GetUserPermissionsAsync` | Get user's permissions |
| `HasPermissionAsync` | Check if user has specific permission |
| `HasRoleAsync` | Check if user has specific role |
| `CheckHealthAsync` | Check IAM Service health status |

### Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| `401 Unauthorized` | JWT issuer mismatch | Set `IdentityServer__IssuerUri` in IAM Service |
| `Connection refused` | Wrong URL or container not running | Use container name, not `localhost` |
| `Name resolution failed` | Container not in same network | Ensure both services use `goodgo-network` |
| `Timeout` | IAM Service slow/unhealthy | Check IAM health: `curl http://iam-service-net:8080/health` |

## Database Migrations

```bash
# Create new migration
dotnet ef migrations add MigrationName \
  --project src/StorageService.Infrastructure \
  --startup-project src/StorageService.API

# Apply migrations
dotnet ef database update \
  --project src/StorageService.Infrastructure \
  --startup-project src/StorageService.API
```

## Testing

```bash
# Run all tests
dotnet test

# Run with coverage
dotnet test --collect:"XPlat Code Coverage"
```

## Project Structure

```
services/storage-service-net/
├── src/
│   ├── StorageService.API/           # Controllers, Commands, Queries
│   ├── StorageService.Domain/        # Entities, Repository interfaces
│   └── StorageService.Infrastructure/# Providers, DbContext, Repositories
├── tests/
│   ├── StorageService.UnitTests/
│   └── StorageService.FunctionalTests/
├── docs/
│   ├── en/                           # English documentation
│   └── vi/                           # Vietnamese documentation
├── docker-compose.yml
├── Dockerfile
└── README.md
```

## License

MIT