pos-system/docs/vi/skills/cicd-advanced-patterns.md

CI/CD Patterns Nâng Cao (CI/CD Advanced Patterns)

Advanced CI/CD patterns for GoodGo microservices including blue-green deployments, canary releases, automated rollback, deployment verification, and progressive delivery.

Các patterns CI/CD nâng cao cho GoodGo microservices bao gồm blue-green deployments, canary releases, automated rollback, deployment verification, và progressive delivery.

Tổng Quan

Advanced CI/CD patterns enable safe, zero-downtime deployments with blue-green, canary releases, automated rollbacks, and deployment verification.

Các patterns CI/CD nâng cao cho phép deployments an toàn, zero-downtime với blue-green, canary releases, automated rollbacks, và deployment verification.

Khi Nào Sử Dụng

Use this skill when implementing advanced deployment strategies, automated rollbacks, or progressive delivery.

Sử dụng skill này khi:

• Implement blue-green deployments / Triển khai blue-green deployments
• Setting up canary releases / Thiết lập canary releases
• Implementing automated rollback mechanisms / Triển khai cơ chế rollback tự động
• Creating deployment verification pipelines / Tạo pipelines xác minh deployment
• Implementing progressive delivery / Triển khai progressive delivery
• Setting up deployment gates / Thiết lập deployment gates
• Implementing smoke tests / Triển khai smoke tests
• Managing deployment strategies in Kubernetes / Quản lý chiến lược deployment trong Kubernetes

Khái Niệm Cốt Lõi

Deployment Strategies / Chiến Lược Deployment

1. Rolling Update: Gradual replacement (default K8s) / Thay thế dần dần (mặc định K8s)
2. Blue-Green: Two identical environments, switch traffic / Hai môi trường giống hệt, chuyển traffic
3. Canary: Gradual rollout to subset of users / Rollout dần dần tới subset users
4. Recreate: Stop old, start new (downtime) / Dừng cũ, khởi động mới (có downtime)

Deployment Verification / Xác Minh Deployment

• Smoke tests / Kiểm tra smoke
• Health checks / Kiểm tra sức khỏe
• Performance tests / Kiểm tra hiệu suất
• Rollback triggers / Kích hoạt rollback

Blue-Green Deployment

Blue-green deployment maintains two identical production environments (blue and green). At any time, only one environment serves live traffic. The new version is deployed to the idle environment, verified, and then traffic is switched.

Blue-green deployment duy trì hai môi trường production giống hệt nhau (blue và green). Tại bất kỳ thời điểm nào, chỉ một môi trường phục vụ traffic trực tiếp. Phiên bản mới được triển khai vào môi trường không hoạt động, được xác minh, sau đó traffic được chuyển đổi.

flowchart TD
    Start([Deployment Triggered]) --> DeployGreen[Deploy to Green Environment]
    DeployGreen --> WaitRollout[Wait for Rollout Complete]
    WaitRollout --> RunSmokeTests[Run Smoke Tests]
    RunSmokeTests --> TestsPassed{Tests Passed?}
    
    TestsPassed -->|Yes| SwitchTraffic[Switch Service Selector to Green]
    TestsPassed -->|No| RollbackToBlue[Rollback: Keep Blue Active]
    
    SwitchTraffic --> MonitorHealth[Monitor Health Metrics]
    MonitorHealth --> HealthOK{Health OK?}
    
    HealthOK -->|Yes| Complete([Deployment Complete])
    HealthOK -->|No| AutoRollback[Auto Rollback to Blue]
    
    AutoRollback --> Complete
    RollbackToBlue --> Fail([Deployment Failed])
    
    style Start fill:#e1f5ff
    style Complete fill:#d4edda
    style Fail fill:#f8d7da
    style TestsPassed fill:#fff3cd
    style HealthOK fill:#fff3cd

Kubernetes Implementation / Triển Khai Kubernetes

# deployments/production/kubernetes/user-service-blue.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-blue
  labels:
    app: user-service
    version: blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
      version: blue
  template:
    metadata:
      labels:
        app: user-service
        version: blue
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.0.0
        ports:
        - containerPort: 5000

---
# deployments/production/kubernetes/user-service-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-green
  labels:
    app: user-service
    version: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
      version: green
  template:
    metadata:
      labels:
        app: user-service
        version: green
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.1.0
        ports:
        - containerPort: 5000

---
# Service selector switches between blue/green
# EN: Switch service selector between blue/green
# VI: Service selector chuyển đổi giữa blue/green
apiVersion: v1
kind: Service
metadata:
  name: user-service
spec:
  selector:
    app: user-service
    version: blue  # EN: Switch to green after verification / VI: Chuyển sang green sau khi xác minh
  ports:
  - port: 80
    targetPort: 5000

Canary Deployment

Canary deployment gradually rolls out changes to a small subset of users before making them available to everyone. This allows for real-world testing with minimal risk.

Canary deployment triển khai thay đổi dần dần tới một subset nhỏ users trước khi làm cho chúng có sẵn cho tất cả mọi người. Điều này cho phép kiểm tra trong môi trường thực tế với rủi ro tối thiểu.

flowchart TD
    Start([Canary Deployment Started]) --> DeployCanary[Deploy Canary Version<br/>1 Replica]
    DeployCanary --> Route10[Route 10% Traffic to Canary]
    Route10 --> Wait10[Wait 5-10 minutes]
    Wait10 --> Check10{Health & Metrics OK?}
    
    Check10 -->|No| RollbackCanary[Rollback: Route 0% to Canary]
    Check10 -->|Yes| Route25[Route 25% Traffic to Canary]
    
    Route25 --> Wait25[Wait 5-10 minutes]
    Wait25 --> Check25{Health & Metrics OK?}
    
    Check25 -->|No| RollbackCanary
    Check25 -->|Yes| Route50[Route 50% Traffic to Canary]
    
    Route50 --> Wait50[Wait 5-10 minutes]
    Wait50 --> Check50{Health & Metrics OK?}
    
    Check50 -->|No| RollbackCanary
    Check50 -->|Yes| Route75[Route 75% Traffic to Canary]
    
    Route75 --> Wait75[Wait 5-10 minutes]
    Wait75 --> Check75{Health & Metrics OK?}
    
    Check75 -->|No| RollbackCanary
    Check75 -->|Yes| Route100[Route 100% Traffic to Canary]
    
    Route100 --> PromoteCanary[Promote Canary to Stable]
    PromoteCanary --> Complete([Canary Complete])
    
    RollbackCanary --> Fail([Canary Failed])
    
    style Start fill:#e1f5ff
    style Complete fill:#d4edda
    style Fail fill:#f8d7da
    style Check10 fill:#fff3cd
    style Check25 fill:#fff3cd
    style Check50 fill:#fff3cd
    style Check75 fill:#fff3cd

Kubernetes Canary with Service Mesh / Canary Kubernetes với Service Mesh

# deployments/production/kubernetes/user-service-canary.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-canary
  labels:
    app: user-service
    version: canary
spec:
  replicas: 1  # EN: Start with 1 replica (10% traffic) / VI: Bắt đầu với 1 replica (10% traffic)
  selector:
    matchLabels:
      app: user-service
      version: canary
  template:
    metadata:
      labels:
        app: user-service
        version: canary
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.1.0

---
# VirtualService splits traffic
# EN: VirtualService splits traffic
# VI: VirtualService chia traffic
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - match:
        - headers:
            canary:
              exact: "true"
      route:
        - destination:
            host: user-service
            subset: canary
          weight: 100
    - route:
        - destination:
            host: user-service
            subset: stable
          weight: 90
        - destination:
            host: user-service
            subset: canary
          weight: 10  # EN: 10% traffic to canary / VI: 10% traffic tới canary

Automated Rollback / Rollback Tự Động

Automated rollback mechanisms detect deployment failures and automatically revert to the previous stable version, minimizing downtime and impact.

Cơ chế rollback tự động phát hiện lỗi deployment và tự động quay lại phiên bản ổn định trước đó, giảm thiểu downtime và tác động.

flowchart TD
    Start([Deployment Completed]) --> RunSmokeTests[Run Smoke Tests]
    RunSmokeTests --> SmokePassed{Smoke Tests Pass?}
    
    SmokePassed -->|No| GetPreviousRev[Get Previous Revision]
    GetPreviousRev --> RollbackDeploy[Rollback Deployment]
    RollbackDeploy --> VerifyRollback[Verify Rollback Success]
    VerifyRollback --> RollbackComplete([Rollback Complete])
    
    SmokePassed -->|Yes| MonitorHealth[Monitor Health Metrics]
    MonitorHealth --> HealthOK{Health OK?}
    
    HealthOK -->|Yes| MonitorErrors[Monitor Error Rates]
    HealthOK -->|No| GetPreviousRev
    
    MonitorErrors --> ErrorRateOK{Error Rate < Threshold?}
    
    ErrorRateOK -->|Yes| MonitorPerformance[Monitor Performance]
    ErrorRateOK -->|No| GetPreviousRev
    
    MonitorPerformance --> PerfOK{Performance OK?}
    
    PerfOK -->|Yes| DeploymentSuccess([Deployment Successful])
    PerfOK -->|No| GetPreviousRev
    
    style Start fill:#e1f5ff
    style DeploymentSuccess fill:#d4edda
    style RollbackComplete fill:#f8d7da
    style SmokePassed fill:#fff3cd
    style HealthOK fill:#fff3cd
    style ErrorRateOK fill:#fff3cd
    style PerfOK fill:#fff3cd

Rollback Script / Script Rollback

#!/bin/bash
# scripts/deployment/rollback.sh
# EN: Automated rollback to previous version
# VI: Rollback tự động về version trước

SERVICE_NAME=$1
NAMESPACE=${2:-production}

# EN: Get previous deployment revision
# VI: Lấy revision deployment trước
PREVIOUS_REVISION=$(kubectl rollout history deployment/$SERVICE_NAME -n $NAMESPACE --no-headers | tail -1 | awk '{print $1}')

if [ -z "$PREVIOUS_REVISION" ]; then
  echo "No previous revision found"
  exit 1
fi

echo "Rolling back to revision $PREVIOUS_REVISION"

# EN: Rollback deployment
# VI: Rollback deployment
kubectl rollout undo deployment/$SERVICE_NAME -n $NAMESPACE --to-revision=$PREVIOUS_REVISION

# EN: Wait for rollout
# VI: Đợi rollout
kubectl rollout status deployment/$SERVICE_NAME -n $NAMESPACE

echo "Rollback complete"

Automated Rollback on Failure / Rollback Tự Động Khi Lỗi

# .github/workflows/deploy-production.yml
name: Deploy Production

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy to Kubernetes
        run: |
          kubectl apply -f deployments/production/kubernetes/
          kubectl rollout status deployment/user-service
      
      - name: Run Smoke Tests
        run: ./scripts/deployment/smoke-tests.sh user-service
      
      - name: Rollback on Failure
        if: failure()
        run: ./scripts/deployment/rollback.sh user-service production

Deployment Verification / Xác Minh Deployment

Smoke Tests / Kiểm Tra Smoke

// scripts/deployment/smoke-tests.ts
// EN: Smoke tests for deployment verification
// VI: Smoke tests để xác minh deployment
import axios from 'axios';

const SERVICE_URL = process.env.SERVICE_URL || 'http://localhost';

async function runSmokeTests(): Promise<boolean> {
  try {
    // EN: Health check
    // VI: Health check
    const healthResponse = await axios.get(`${SERVICE_URL}/health`);
    if (healthResponse.status !== 200) {
      console.error('Health check failed');
      return false;
    }

    // EN: Basic functionality test
    // VI: Test chức năng cơ bản
    const testResponse = await axios.get(`${SERVICE_URL}/api/v1/users`, {
      timeout: 5000,
    });
    
    if (testResponse.status !== 200) {
      console.error('Functionality test failed');
      return false;
    }

    console.log('Smoke tests passed');
    return true;
  } catch (error) {
    console.error('Smoke tests failed', error);
    return false;
  }
}

runSmokeTests().then((success) => {
  process.exit(success ? 0 : 1);
});

Health Check Script / Script Kiểm Tra Sức Khỏe

#!/bin/bash
# scripts/deployment/health-checks.sh
# EN: Comprehensive health checks
# VI: Health checks toàn diện

SERVICE_NAME=$1
NAMESPACE=${2:-production}

echo "Running health checks for $SERVICE_NAME"

# EN: Check pods are ready
# VI: Kiểm tra pods đã ready
READY_PODS=$(kubectl get pods -n $NAMESPACE -l app=$SERVICE_NAME --field-selector=status.phase=Running --no-headers | wc -l)

if [ $READY_PODS -eq 0 ]; then
  echo "No ready pods found"
  exit 1
fi

# EN: Check service endpoints
# VI: Kiểm tra service endpoints
ENDPOINTS=$(kubectl get endpoints $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.subsets[0].addresses[*].ip}' | wc -w)

if [ $ENDPOINTS -eq 0 ]; then
  echo "No service endpoints found"
  exit 1
fi

# EN: Check health endpoint
# VI: Kiểm tra health endpoint
SERVICE_URL=$(kubectl get service $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

if [ -z "$SERVICE_URL" ]; then
  SERVICE_URL="http://$SERVICE_NAME.$NAMESPACE.svc.cluster.local"
fi

HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" $SERVICE_URL/health)

if [ $HTTP_CODE -ne 200 ]; then
  echo "Health endpoint returned $HTTP_CODE"
  exit 1
fi

echo "Health checks passed"

Deployment Gates / Cổng Deployment

Deployment gates add checkpoints in the CI/CD pipeline that must pass before proceeding to the next stage.

Deployment gates thêm các điểm kiểm tra trong CI/CD pipeline phải vượt qua trước khi tiến tới giai đoạn tiếp theo.

# .github/workflows/deploy-with-gates.yml
name: Deploy with Gates

jobs:
  deploy:
    steps:
      - name: Deploy
        run: kubectl apply -f deployments/
      
      - name: Wait for Rollout
        run: kubectl rollout status deployment/service
      
      - name: Smoke Tests Gate
        id: smoke-tests
        run: ./scripts/deployment/smoke-tests.sh
      
      - name: Performance Tests Gate
        if: steps.smoke-tests.outcome == 'success'
        run: ./scripts/deployment/performance-tests.sh
      
      - name: Manual Approval Gate
        if: steps.smoke-tests.outcome == 'success'
        uses: trstringer/manual-approval@v1
        with:
          secret: ${{ secrets.GITHUB_TOKEN }}
          approvers: team-leads
          minimum-approvals: 1
          issue-title: "Approve deployment"

Best Practices / Thực Hành Tốt

1. Blue-Green: Use for zero-downtime deployments / Sử dụng cho zero-downtime deployments
2. Canary: Use for gradual rollouts with monitoring / Sử dụng cho rollouts dần dần với monitoring
3. Automated Rollback: Always have rollback plan / Luôn có kế hoạch rollback
4. Smoke Tests: Run immediately after deployment / Chạy ngay sau deployment
5. Health Checks: Monitor health continuously / Giám sát sức khỏe liên tục
6. Gates: Use deployment gates for critical deployments / Sử dụng deployment gates cho deployments quan trọng

Common Mistakes / Lỗi Thường Gặp

1. No Rollback Plan: Can't recover from failed deployment / Không thể phục hồi từ deployment thất bại

   # ✅ Always have rollback command ready
   # ✅ Luôn có lệnh rollback sẵn sàng
   kubectl rollout undo deployment/service
   

2. Skipping Smoke Tests: Catching issues too late / Phát hiện vấn đề quá muộn

   # ✅ Run smoke tests immediately after deploy
   # ✅ Chạy smoke tests ngay sau khi deploy
   - name: Smoke Tests
     run: ./scripts/smoke-tests.sh
   

3. 100% Traffic Switch: All-or-nothing failures / Lỗi tất cả hoặc không gì cả

   # ❌ BAD: Immediate full switch
   # ❌ XẤU: Chuyển đổi toàn bộ ngay lập tức
   # ✅ GOOD: Gradual rollout (10% → 50% → 100%)
   # ✅ TỐT: Rollout dần dần (10% → 50% → 100%)
   

4. No Health Monitoring: Missing deployment issues / Bỏ lỡ vấn đề deployment

   # ✅ Monitor health after deployment
   # ✅ Giám sát sức khỏe sau deployment
   - name: Monitor Health
     run: kubectl rollout status deployment/service --timeout=5m
   

Quick Reference / Tham Khảo Nhanh

Strategy	Risk	Downtime	Resource Cost
Blue-Green	Low	Zero	2x (temporary)
Canary	Low	Zero	+10-20%
Rolling	Medium	Zero	1x
Recreate	High	Yes	1x

Deployment Commands / Lệnh Deployment:

# Apply deployment
kubectl apply -f kubernetes/

# Check rollout status
kubectl rollout status deployment/service

# Rollback
kubectl rollout undo deployment/service

# Canary traffic split (Istio)
kubectl apply -f virtualservice-canary.yaml

GitHub Actions Triggers / Kích Hoạt GitHub Actions:

on:
  push:
    branches: [main]      # Deploy to prod
    tags: ['v*']          # Release
  pull_request:
    branches: [main]      # PR checks

Deployment Gates / Cổng Deployment:

Build → Test → Security Scan → Deploy Staging
→ Smoke Tests → Manual Approval → Deploy Prod

Tài Nguyên

• Kubernetes Deployment
• Istio Traffic Management
• Deployment Kubernetes - K8s deployment patterns / Patterns deployment K8s
• Testing Patterns - Testing strategies / Chiến lược testing
• Project Rules - GoodGo coding standards / Tiêu chuẩn coding GoodGo
• Skill Source: .cursor/skills/cicd-advanced-patterns/SKILL.md