---
name: cicd-advanced-patterns
description: Advanced CI/CD patterns for GoodGo services. Use for blue-green deployments, canary releases, automated rollbacks, or progressive delivery.
dependencies: "github-actions, docker, kubernetes"
---

# CI/CD Advanced Patterns

## When to Use This Skill

Use this skill when:
- Implementing blue-green deployments
- Setting up canary releases
- Implementing automated rollback mechanisms
- Creating deployment verification pipelines
- Implementing progressive delivery
- Setting up deployment gates
- Implementing smoke tests
- Managing deployment strategies in Kubernetes

## Core Concepts

### Deployment Strategies

1. **Rolling Update**: Gradual replacement (default K8s)
2. **Blue-Green**: Two identical environments, switch traffic
3. **Canary**: Gradual rollout to subset of users
4. **Recreate**: Stop old, start new (downtime)

### Deployment Verification

- Smoke tests
- Health checks
- Performance tests
- Rollback triggers

## Blue-Green Deployment

### Kubernetes Implementation

```yaml
# deployments/production/kubernetes/user-service-blue.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-blue
  labels:
    app: user-service
    version: blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
      version: blue
  template:
    metadata:
      labels:
        app: user-service
        version: blue
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.0.0
        ports:
        - containerPort: 5000

---
# deployments/production/kubernetes/user-service-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-green
  labels:
    app: user-service
    version: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
      version: green
  template:
    metadata:
      labels:
        app: user-service
        version: green
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.1.0
        ports:
        - containerPort: 5000

---
# Service selector switches between blue/green
apiVersion: v1
kind: Service
metadata:
  name: user-service
spec:
  selector:
    app: user-service
    version: blue  # EN: Switch to green after verification / VI: Chuyển sang green sau khi xác minh
  ports:
  - port: 80
    targetPort: 5000
```

### Blue-Green Switch Script

```bash
#!/bin/bash
# scripts/deployment/blue-green-switch.sh
# EN: Switch between blue and green deployments
# VI: Chuyển đổi giữa blue và green deployments

SERVICE_NAME=$1
CURRENT_VERSION=$(kubectl get service $SERVICE_NAME -o jsonpath='{.spec.selector.version}')
NEW_VERSION=$([ "$CURRENT_VERSION" = "blue" ] && echo "green" || echo "blue")

echo "Switching from $CURRENT_VERSION to $NEW_VERSION"

# EN: Update service selector
# VI: Cập nhật service selector
kubectl patch service $SERVICE_NAME -p "{\"spec\":{\"selector\":{\"version\":\"$NEW_VERSION\"}}}"

# EN: Wait for rollout
# VI: Đợi rollout
kubectl rollout status deployment/$SERVICE_NAME-$NEW_VERSION

# EN: Run smoke tests
# VI: Chạy smoke tests
./scripts/deployment/smoke-tests.sh $SERVICE_NAME

if [ $? -ne 0 ]; then
  echo "Smoke tests failed, rolling back"
  kubectl patch service $SERVICE_NAME -p "{\"spec\":{\"selector\":{\"version\":\"$CURRENT_VERSION\"}}}"
  exit 1
fi

echo "Successfully switched to $NEW_VERSION"
```

## Canary Deployment

### Kubernetes Canary with Service Mesh

```yaml
# deployments/production/kubernetes/user-service-canary.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service-canary
  labels:
    app: user-service
    version: canary
spec:
  replicas: 1  # EN: Start with 1 replica (10% traffic) / VI: Bắt đầu với 1 replica (10% traffic)
  selector:
    matchLabels:
      app: user-service
      version: canary
  template:
    metadata:
      labels:
        app: user-service
        version: canary
    spec:
      containers:
      - name: user-service
        image: goodgo/user-service:v1.1.0

---
# VirtualService splits traffic
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - match:
        - headers:
            canary:
              exact: "true"
      route:
        - destination:
            host: user-service
            subset: canary
          weight: 100
    - route:
        - destination:
            host: user-service
            subset: stable
          weight: 90
        - destination:
            host: user-service
            subset: canary
          weight: 10  # EN: 10% traffic to canary / VI: 10% traffic tới canary
```

### Progressive Canary Rollout

```bash
#!/bin/bash
# scripts/deployment/canary-rollout.sh
# EN: Progressive canary rollout
# VI: Progressive canary rollout

SERVICE_NAME=$1
CANARY_PERCENTAGES=(10 25 50 75 100)

for PERCENTAGE in "${CANARY_PERCENTAGES[@]}"; do
  echo "Rolling out to $PERCENTAGE%"
  
  # EN: Update VirtualService weight
  # VI: Cập nhật VirtualService weight
  kubectl patch virtualservice $SERVICE_NAME -p "{\"spec\":{\"http\":[{\"route\":[{\"destination\":{\"subset\":\"canary\"},\"weight\":$PERCENTAGE},{\"destination\":{\"subset\":\"stable\"},\"weight\":$((100-PERCENTAGE))}]}]}}"
  
  # EN: Wait for traffic to stabilize
  # VI: Đợi traffic ổn định
  sleep 60
  
  # EN: Run health checks
  # VI: Chạy health checks
  ./scripts/deployment/health-checks.sh $SERVICE_NAME
  
  if [ $? -ne 0 ]; then
    echo "Health checks failed at $PERCENTAGE%, rolling back"
    kubectl patch virtualservice $SERVICE_NAME -p "{\"spec\":{\"http\":[{\"route\":[{\"destination\":{\"subset\":\"canary\"},\"weight\":0},{\"destination\":{\"subset\":\"stable\"},\"weight\":100}]}]}}"
    exit 1
  fi
  
  echo "Successfully rolled out to $PERCENTAGE%"
done

echo "Canary rollout complete"
```

## Automated Rollback

### Rollback Script

```bash
#!/bin/bash
# scripts/deployment/rollback.sh
# EN: Automated rollback to previous version
# VI: Rollback tự động về version trước

SERVICE_NAME=$1
NAMESPACE=${2:-production}

# EN: Get previous deployment revision
# VI: Lấy revision deployment trước
PREVIOUS_REVISION=$(kubectl rollout history deployment/$SERVICE_NAME -n $NAMESPACE --no-headers | tail -1 | awk '{print $1}')

if [ -z "$PREVIOUS_REVISION" ]; then
  echo "No previous revision found"
  exit 1
fi

echo "Rolling back to revision $PREVIOUS_REVISION"

# EN: Rollback deployment
# VI: Rollback deployment
kubectl rollout undo deployment/$SERVICE_NAME -n $NAMESPACE --to-revision=$PREVIOUS_REVISION

# EN: Wait for rollout
# VI: Đợi rollout
kubectl rollout status deployment/$SERVICE_NAME -n $NAMESPACE

echo "Rollback complete"
```

### Automated Rollback on Failure

```yaml
# .github/workflows/deploy-production.yml
name: Deploy Production

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy to Kubernetes
        run: |
          kubectl apply -f deployments/production/kubernetes/
          kubectl rollout status deployment/user-service
      
      - name: Run Smoke Tests
        run: ./scripts/deployment/smoke-tests.sh user-service
      
      - name: Rollback on Failure
        if: failure()
        run: ./scripts/deployment/rollback.sh user-service production
```

## Deployment Verification

### Smoke Tests

```typescript
// scripts/deployment/smoke-tests.ts
// EN: Smoke tests for deployment verification
// VI: Smoke tests để xác minh deployment
import axios from 'axios';

const SERVICE_URL = process.env.SERVICE_URL || 'http://localhost';

async function runSmokeTests(): Promise<boolean> {
  try {
    // EN: Health check
    // VI: Health check
    const healthResponse = await axios.get(`${SERVICE_URL}/health`);
    if (healthResponse.status !== 200) {
      console.error('Health check failed');
      return false;
    }

    // EN: Basic functionality test
    // VI: Test chức năng cơ bản
    const testResponse = await axios.get(`${SERVICE_URL}/api/v1/users`, {
      timeout: 5000,
    });
    
    if (testResponse.status !== 200) {
      console.error('Functionality test failed');
      return false;
    }

    console.log('Smoke tests passed');
    return true;
  } catch (error) {
    console.error('Smoke tests failed', error);
    return false;
  }
}

runSmokeTests().then((success) => {
  process.exit(success ? 0 : 1);
});
```

### Health Check Script

```bash
#!/bin/bash
# scripts/deployment/health-checks.sh
# EN: Comprehensive health checks
# VI: Health checks toàn diện

SERVICE_NAME=$1
NAMESPACE=${2:-production}

echo "Running health checks for $SERVICE_NAME"

# EN: Check pods are ready
# VI: Kiểm tra pods đã ready
READY_PODS=$(kubectl get pods -n $NAMESPACE -l app=$SERVICE_NAME --field-selector=status.phase=Running --no-headers | wc -l)

if [ $READY_PODS -eq 0 ]; then
  echo "No ready pods found"
  exit 1
fi

# EN: Check service endpoints
# VI: Kiểm tra service endpoints
ENDPOINTS=$(kubectl get endpoints $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.subsets[0].addresses[*].ip}' | wc -w)

if [ $ENDPOINTS -eq 0 ]; then
  echo "No service endpoints found"
  exit 1
fi

# EN: Check health endpoint
# VI: Kiểm tra health endpoint
SERVICE_URL=$(kubectl get service $SERVICE_NAME -n $NAMESPACE -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

if [ -z "$SERVICE_URL" ]; then
  SERVICE_URL="http://$SERVICE_NAME.$NAMESPACE.svc.cluster.local"
fi

HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" $SERVICE_URL/health)

if [ $HTTP_CODE -ne 200 ]; then
  echo "Health endpoint returned $HTTP_CODE"
  exit 1
fi

echo "Health checks passed"
```

## Deployment Gates

```yaml
# .github/workflows/deploy-with-gates.yml
name: Deploy with Gates

jobs:
  deploy:
    steps:
      - name: Deploy
        run: kubectl apply -f deployments/
      
      - name: Wait for Rollout
        run: kubectl rollout status deployment/service
      
      - name: Smoke Tests Gate
        id: smoke-tests
        run: ./scripts/deployment/smoke-tests.sh
      
      - name: Performance Tests Gate
        if: steps.smoke-tests.outcome == 'success'
        run: ./scripts/deployment/performance-tests.sh
      
      - name: Manual Approval Gate
        if: steps.smoke-tests.outcome == 'success'
        uses: trstringer/manual-approval@v1
        with:
          secret: ${{ secrets.GITHUB_TOKEN }}
          approvers: team-leads
          minimum-approvals: 1
          issue-title: "Approve deployment"
```

## Best Practices

1. **Blue-Green**: Use for zero-downtime deployments
2. **Canary**: Use for gradual rollouts with monitoring
3. **Automated Rollback**: Always have rollback plan
4. **Smoke Tests**: Run immediately after deployment
5. **Health Checks**: Monitor health continuously
6. **Gates**: Use deployment gates for critical deployments

## Common Mistakes

1. **No Rollback Plan**: Can't recover from failed deployment
   ```yaml
   # ✅ Always have rollback command ready
   kubectl rollout undo deployment/service
   ```

2. **Skipping Smoke Tests**: Catching issues too late
   ```yaml
   # ✅ Run smoke tests immediately after deploy
   - name: Smoke Tests
     run: ./scripts/smoke-tests.sh
   ```

3. **100% Traffic Switch**: All-or-nothing failures
   ```yaml
   # ❌ BAD: Immediate full switch
   # ✅ GOOD: Gradual rollout (10% → 50% → 100%)
   ```

4. **No Health Monitoring**: Missing deployment issues
   ```yaml
   # ✅ Monitor health after deployment
   - name: Monitor Health
     run: kubectl rollout status deployment/service --timeout=5m
   ```

## Quick Reference

| Strategy | Risk | Downtime | Resource Cost |
|----------|------|----------|---------------|
| **Blue-Green** | Low | Zero | 2x (temporary) |
| **Canary** | Low | Zero | +10-20% |
| **Rolling** | Medium | Zero | 1x |
| **Recreate** | High | Yes | 1x |

**Deployment Commands:**
```bash
# Apply deployment
kubectl apply -f kubernetes/

# Check rollout status
kubectl rollout status deployment/service

# Rollback
kubectl rollout undo deployment/service

# Canary traffic split (Istio)
kubectl apply -f virtualservice-canary.yaml
```

**GitHub Actions Triggers:**
```yaml
on:
  push:
    branches: [main]      # Deploy to prod
    tags: ['v*']          # Release
  pull_request:
    branches: [main]      # PR checks
```

**Deployment Gates:**
```
Build → Test → Security Scan → Deploy Staging
→ Smoke Tests → Manual Approval → Deploy Prod
```

## Resources

- [Kubernetes Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
- [Istio Traffic Management](https://istio.io/latest/docs/tasks/traffic-management/)
- [Deployment Kubernetes](../deployment-kubernetes/SKILL.md) - K8s deployment patterns
- [Testing Patterns](../testing-patterns/SKILL.md) - Testing strategies
- [Project Rules](../project-rules/SKILL.md) - GoodGo coding standards
