pos-system/.agent/rules/cicd-advanced-patterns.md

trigger

trigger
always_on

CI/CD Advanced Patterns

When to Use This Skill

Use this skill when:

• Implementing blue-green deployments
• Setting up canary releases
• Implementing automated rollback mechanisms
• Creating deployment verification pipelines
• Implementing progressive delivery
• Setting up deployment gates
• Implementing smoke tests
• Managing deployment strategies in Kubernetes

Core Concepts

Deployment Strategies

1. Rolling Update: Gradual replacement (default K8s)
2. Blue-Green: Two identical environments, switch traffic
3. Canary: Gradual rollout to subset of users
4. Recreate: Stop old, start new (downtime)

Deployment Verification

• Smoke tests
• Health checks
• Performance tests
• Rollback triggers

Blue-Green Deployment

Kubernetes Implementation

# deployments/production/kubernetes/user-service-blue.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-blue
  labels:
    app: user-service
    version: blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
      version: blue
  template:
    metadata:
      labels:
        app: user-service
        version: blue
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.0.0
        ports:
        - containerPort: 5000

---
# deployments/production/kubernetes/user-service-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-green
  labels:
    app: user-service
    version: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
      version: green
  template:
    metadata:
      labels:
        app: user-service
        version: green
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.1.0
        ports:
        - containerPort: 5000

---
# Service selector switches between blue/green
apiVersion: v1
kind: Service
metadata:
  name: user-service
spec:
  selector:
    app: user-service
    version: blue  # EN: Switch to green after verification / VI: Chuyển sang green sau khi xác minh
  ports:
  - port: 80
    targetPort: 5000

Blue-Green Switch Script

#!/bin/bash
# scripts/deployment/blue-green-switch.sh
# EN: Switch between blue and green deployments
# VI: Chuyển đổi giữa blue và green deployments

SERVICE_NAME=$1
CURRENT_VERSION=$(kubectl get service $SERVICE_NAME -o jsonpath='{.spec.selector.version}')
NEW_VERSION=$([ "$CURRENT_VERSION" = "blue" ] && echo "green" || echo "blue")

echo "Switching from $CURRENT_VERSION to $NEW_VERSION"

# EN: Update service selector
# VI: Cập nhật service selector
kubectl patch service $SERVICE_NAME -p "{\"spec\":{\"selector\":{\"version\":\"$NEW_VERSION\"}}}"

# EN: Wait for rollout
# VI: Đợi rollout
kubectl rollout status deployment/$SERVICE_NAME-$NEW_VERSION

# EN: Run smoke tests
# VI: Chạy smoke tests
./scripts/deployment/smoke-tests.sh $SERVICE_NAME

if [ $? -ne 0 ]; then
  echo "Smoke tests failed, rolling back"
  kubectl patch service $SERVICE_NAME -p "{\"spec\":{\"selector\":{\"version\":\"$CURRENT_VERSION\"}}}"
  exit 1
fi

echo "Successfully switched to $NEW_VERSION"

Canary Deployment

Kubernetes Canary with Service Mesh

# deployments/production/kubernetes/user-service-canary.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-canary
  labels:
    app: user-service
    version: canary
spec:
  replicas: 1  # EN: Start with 1 replica (10% traffic) / VI: Bắt đầu với 1 replica (10% traffic)
  selector:
    matchLabels:
      app: user-service
      version: canary
  template:
    metadata:
      labels:
        app: user-service
        version: canary
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.1.0

---
# VirtualService splits traffic
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - match:
        - headers:
            canary:
              exact: "true"
      route:
        - destination:
            host: user-service
            subset: canary
          weight: 100
    - route:
        - destination:
            host: user-service
            subset: stable
          weight: 90
        - destination:
            host: user-service
            subset: canary
          weight: 10  # EN: 10% traffic to canary / VI: 10% traffic tới canary

Progressive Canary Rollout

#!/bin/bash
# scripts/deployment/canary-rollout.sh
# EN: Progressive canary rollout
# VI: Progressive canary rollout

SERVICE_NAME=$1
CANARY_PERCENTAGES=(10 25 50 75 100)

for PERCENTAGE in "${CANARY_PERCENTAGES[@]}"; do
  echo "Rolling out to $PERCENTAGE%"
  
  # EN: Update VirtualService weight
  # VI: Cập nhật VirtualService weight
  kubectl patch virtualservice $SERVICE_NAME -p "{\"spec\":{\"http\":[{\"route\":[{\"destination\":{\"subset\":\"canary\"},\"weight\":$PERCENTAGE},{\"destination\":{\"subset\":\"stable\"},\"weight\":$((100-PERCENTAGE))}]}]}}"
  
  # EN: Wait for traffic to stabilize
  # VI: Đợi traffic ổn định
  sleep 60
  
  # EN: Run health checks
  # VI: Chạy health checks
  ./scripts/deployment/health-checks.sh $SERVICE_NAME
  
  if [ $? -ne 0 ]; then
    echo "Health checks failed at $PERCENTAGE%, rolling back"
    kubectl patch virtualservice $SERVICE_NAME -p "{\"spec\":{\"http\":[{\"route\":[{\"destination\":{\"subset\":\"canary\"},\"weight\":0},{\"destination\":{\"subset\":\"stable\"},\"weight\":100}]}]}}"
    exit 1
  fi
  
  echo "Successfully rolled out to $PERCENTAGE%"
done

echo "Canary rollout complete"

Automated Rollback

Rollback Script

#!/bin/bash
# scripts/deployment/rollback.sh
# EN: Automated rollback to previous version
# VI: Rollback tự động về version trước

SERVICE_NAME=$1
NAMESPACE=${2:-production}

# EN: Get previous deployment revision
# VI: Lấy revision deployment trước
PREVIOUS_REVISION=$(kubectl rollout history deployment/$SERVICE_NAME -n $NAMESPACE --no-headers | tail -1 | awk '{print $1}')

if [ -z "$PREVIOUS_REVISION" ]; then
  echo "No previous revision found"
  exit 1
fi

echo "Rolling back to revision $PREVIOUS_REVISION"

# EN: Rollback deployment
# VI: Rollback deployment
kubectl rollout undo deployment/$SERVICE_NAME -n $NAMESPACE --to-revision=$PREVIOUS_REVISION

# EN: Wait for rollout
# VI: Đợi rollout
kubectl rollout status deployment/$SERVICE_NAME -n $NAMESPACE

echo "Rollback complete"

Automated Rollback on Failure

# .github/workflows/deploy-production.yml
name: Deploy Production

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy to Kubernetes
        run: |
          kubectl apply -f deployments/production/kubernetes/
          kubectl rollout status deployment/user-service
      
      - name: Run Smoke Tests
        run: ./scripts/deployment/smoke-tests.sh user-service
      
      - name: Rollback on Failure
        if: failure()
        run: ./scripts/deployment/rollback.sh user-service production

Deployment Verification

Smoke Tests

// scripts/deployment/smoke-tests.ts
// EN: Smoke tests for deployment verification
// VI: Smoke tests để xác minh deployment
import axios from 'axios';

const SERVICE_URL = process.env.SERVICE_URL || 'http://localhost';

async function runSmokeTests(): Promise<boolean> {
  try {
    // EN: Health check
    // VI: Health check
    const healthResponse = await axios.get(`${SERVICE_URL}/health`);
    if (healthResponse.status !== 200) {
      console.error('Health check failed');
      return false;
    }

    // EN: Basic functionality test
    // VI: Test chức năng cơ bản
    const testResponse = await axios.get(`${SERVICE_URL}/api/v1/users`, {
      timeout: 5000,
    });
    
    if (testResponse.status !== 200) {
      console.error('Functionality test failed');
      return false;
    }

    console.log('Smoke tests passed');
    return true;
  } catch (error) {
    console.error('Smoke tests failed', error);
    return false;
  }
}

runSmokeTests().then((success) => {
  process.exit(success ? 0 : 1);
});

Health Check Script

#!/bin/bash
# scripts/deployment/health-checks.sh
# EN: Comprehensive health checks
# VI: Health checks toàn diện

SERVICE_NAME=$1
NAMESPACE=${2:-production}

echo "Running health checks for $SERVICE_NAME"

# EN: Check pods are ready
# VI: Kiểm tra pods đã ready
READY_PODS=$(kubectl get pods -n $NAMESPACE -l app=$SERVICE_NAME --field-selector=status.phase=Running --no-headers | wc -l)

if [ $READY_PODS -eq 0 ]; then
  echo "No ready pods found"
  exit 1
fi

# EN: Check service endpoints
# VI: Kiểm tra service endpoints
ENDPOINTS=$(kubectl get endpoints $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.subsets[0].addresses[*].ip}' | wc -w)

if [ $ENDPOINTS -eq 0 ]; then
  echo "No service endpoints found"
  exit 1
fi

# EN: Check health endpoint
# VI: Kiểm tra health endpoint
SERVICE_URL=$(kubectl get service $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

if [ -z "$SERVICE_URL" ]; then
  SERVICE_URL="http://$SERVICE_NAME.$NAMESPACE.svc.cluster.local"
fi

HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" $SERVICE_URL/health)

if [ $HTTP_CODE -ne 200 ]; then
  echo "Health endpoint returned $HTTP_CODE"
  exit 1
fi

echo "Health checks passed"

Deployment Gates

# .github/workflows/deploy-with-gates.yml
name: Deploy with Gates

jobs:
  deploy:
    steps:
      - name: Deploy
        run: kubectl apply -f deployments/
      
      - name: Wait for Rollout
        run: kubectl rollout status deployment/service
      
      - name: Smoke Tests Gate
        id: smoke-tests
        run: ./scripts/deployment/smoke-tests.sh
      
      - name: Performance Tests Gate
        if: steps.smoke-tests.outcome == 'success'
        run: ./scripts/deployment/performance-tests.sh
      
      - name: Manual Approval Gate
        if: steps.smoke-tests.outcome == 'success'
        uses: trstringer/manual-approval@v1
        with:
          secret: ${{ secrets.GITHUB_TOKEN }}
          approvers: team-leads
          minimum-approvals: 1
          issue-title: "Approve deployment"

Best Practices

1. Blue-Green: Use for zero-downtime deployments
2. Canary: Use for gradual rollouts with monitoring
3. Automated Rollback: Always have rollback plan
4. Smoke Tests: Run immediately after deployment
5. Health Checks: Monitor health continuously
6. Gates: Use deployment gates for critical deployments

Common Mistakes

1. No Rollback Plan: Can't recover from failed deployment

   # ✅ Always have rollback command ready
   kubectl rollout undo deployment/service
   

2. Skipping Smoke Tests: Catching issues too late

   # ✅ Run smoke tests immediately after deploy
   - name: Smoke Tests
     run: ./scripts/smoke-tests.sh
   

3. 100% Traffic Switch: All-or-nothing failures

   # ❌ BAD: Immediate full switch
   # ✅ GOOD: Gradual rollout (10% → 50% → 100%)
   

4. No Health Monitoring: Missing deployment issues

   # ✅ Monitor health after deployment
   - name: Monitor Health
     run: kubectl rollout status deployment/service --timeout=5m
   

Quick Reference

Strategy	Risk	Downtime	Resource Cost
Blue-Green	Low	Zero	2x (temporary)
Canary	Low	Zero	+10-20%
Rolling	Medium	Zero	1x
Recreate	High	Yes	1x

Deployment Commands:

# Apply deployment
kubectl apply -f kubernetes/

# Check rollout status
kubectl rollout status deployment/servi