CI_CD_PIPELINE_SETUP.md

# CI/CD Pipeline Setup Guide

**Updated**: December 2024  
**Pipeline**: Enhanced HF Spaces Multi-Track Deployment  
**Status**: Production-Ready with Comprehensive Testing

---

## 🎯 **Pipeline Overview**

The enhanced CD pipeline supports:
- **Multi-Track HF Spaces Deployment**: 8 spaces across Track 1 (MCP Tools) and Track 3 (Agent Demo)
- **Staging & Production Environments**: Full deployment lifecycle
- **Comprehensive Testing**: 516 tests + integration validation
- **Automatic Rollback**: Failed deployment recovery
- **Legacy Cloud Support**: Optional Kubernetes deployment

---

## 🔐 **Required GitHub Secrets**

### **Production Environment Secrets**
```bash
# Hugging Face Configuration
HF_TOKEN                    # Your HF write token (hf_xxx...)
HF_USERNAME                 # Your HF username (e.g., "BasalGanglia")

# GitHub Configuration (automatically available)
GITHUB_TOKEN               # GitHub Actions token (auto-provided)

# Optional: Cloud Deployment
KUBECONFIG_PRODUCTION      # Base64 encoded kubeconfig for production
```

### **Staging Environment Secrets**
```bash
# Staging HF Configuration
HF_TOKEN_STAGING           # Staging HF token (can be same as production)
HF_USERNAME_STAGING        # Staging HF username (e.g., "BasalGanglia-staging")

# Optional: Cloud Staging
KUBECONFIG_STAGING         # Base64 encoded kubeconfig for staging
```

### **Optional: Third-Party Integrations**
```bash
# Codecov (for test coverage)
CODECOV_TOKEN              # Codecov upload token

# External API Keys (if needed for testing)
OPENAI_API_KEY            # For LLM testing
NEO4J_PASSWORD            # For KG testing
```

---

## ⚙️ **Required GitHub Variables**

```bash
# Cloud Deployment Controls
ENABLE_CLOUD_STAGING       # "true" to enable cloud staging deployment
ENABLE_CLOUD_PRODUCTION    # "true" to enable cloud production deployment
```

---

## 🏗️ **GitHub Environments Setup**

The pipeline uses repository-level secrets (no environments needed), but you can optionally create environments for additional protection:

### **Optional: Create Staging Environment**
1. Go to **Settings** → **Environments**
2. Click **New environment** → Name: `staging`
3. Add protection rules if desired:
   - Required reviewers
   - Wait timer
   - Deployment branches

### **Optional: Create Production Environment**
1. Go to **Settings** → **Environments**
2. Click **New environment** → Name: `production`
3. Add protection rules:
   - ✅ Required reviewers (recommended)
   - ✅ Wait timer: 5 minutes
   - ✅ Deployment branches: `main` only

---

## 🚀 **Setup Commands**

### **1. Set Required Secrets**

```bash
# Navigate to your repository
cd your-repo

# Set HF production secrets
gh secret set HF_TOKEN --body "hf_your_production_token_here"
gh secret set HF_USERNAME --body "your-hf-username"

# Set HF staging secrets
gh secret set HF_TOKEN_STAGING --body "hf_your_staging_token_here"
gh secret set HF_USERNAME_STAGING --body "your-hf-staging-username"

# Optional: Set cloud deployment variables
gh variable set ENABLE_CLOUD_STAGING --body "false"
gh variable set ENABLE_CLOUD_PRODUCTION --body "false"
```

### **2. Verify Secrets Setup**

```bash
# List all secrets (values hidden)
gh secret list

# Expected output:
# HF_TOKEN                 Updated YYYY-MM-DD
# HF_USERNAME              Updated YYYY-MM-DD
# HF_TOKEN_STAGING         Updated YYYY-MM-DD
# HF_USERNAME_STAGING      Updated YYYY-MM-DD
```

### **3. Test Pipeline Setup**

```bash
# Create a test branch to trigger staging deployment
git checkout -b test/pipeline-setup
git commit --allow-empty -m "test: trigger staging deployment"
git push origin test/pipeline-setup

# Create PR to test staging pipeline
gh pr create --title "Test: Pipeline Setup" --body "Testing new CD pipeline"

# Monitor pipeline
gh run list --limit 1
gh run watch $(gh run list --limit 1 --json databaseId -q '.[0].databaseId')
```

---

## 📊 **Pipeline Jobs Overview**

### **Core Jobs (Always Run)**
1. **`build`**: Docker image build and push
2. **`test`**: Comprehensive test suite (516 tests)

### **Staging Jobs (PR + develop branch)**
3. **`deploy-hf-staging`**: Deploy all 8 HF Spaces to staging
4. **`deploy-cloud-staging`**: Optional cloud staging deployment

### **Production Jobs (main branch + tags)**
5. **`deploy-hf-production`**: Deploy all 8 HF Spaces to production
6. **`deploy-cloud-production`**: Optional cloud production deployment

### **Rollback Jobs (On Failure)**
7. **`rollback-hf`**: Rollback HF Spaces deployment
8. **`rollback-cloud`**: Rollback cloud deployment

---

## 🎯 **Multi-Track Deployment Strategy**

### **Track 3: Main Platform**
- **Space**: `{username}/kgraph-mcp-agent-platform`
- **Tags**: `agent-demo-track`, `gradio-4.0`, `mcp-hackathon`
- **File**: Uses main `app.py` or `app_hf.py`

### **Track 1: MCP Tools (7 spaces)**
1. **Summarizer**: `{username}/mcp-summarizer-tool`
2. **Sentiment**: `{username}/mcp-sentiment-analyzer`
3. **Code Analyzer**: `{username}/mcp-code-analyzer`
4. **File Processor**: `{username}/mcp-file-processor`
5. **Image Tool**: `{username}/mcp-image-tool`
6. **Math Tool**: `{username}/mcp-math-tool`
7. **Web Scraper**: `{username}/mcp-web-scraper`

All Track 1 tools get:
- **Tags**: `mcp-server-track`, `gradio-4.0`, `mcp-hackathon`
- **Endpoints**: `/gradio_api/mcp/sse` for MCP protocol

---

## 🔧 **Pipeline Customization**

### **Environment-Specific Configuration**

```python
# update_tools_for_hf.py supports:
python update_tools_for_hf.py --environment staging --username "username-staging"
python update_tools_for_hf.py --environment production --username "username"
```

### **Deployment Script Configuration**

```bash
# deploy_all_mcp_tools.sh supports:
./deploy_all_mcp_tools.sh staging    # Uses staging config
./deploy_all_mcp_tools.sh production # Uses production config
```

### **Testing Configuration**

```bash
# test_hf_integration.py supports:
python test_hf_integration.py --environment staging --username "username-staging"
python test_hf_integration.py --environment production --username "username"
```

---

## 📈 **Deployment Monitoring**

### **Pipeline Status Monitoring**

```bash
# Watch current pipeline run
gh run watch

# View pipeline logs
gh run view --log

# List recent runs
gh run list --limit 10

# View specific job logs
gh run view [RUN_ID] --job [JOB_NAME]
```

### **HF Spaces Health Checks**

```bash
# Test main platform
curl -f https://huggingface.co/spaces/{username}/kgraph-mcp-agent-platform

# Test MCP tools
curl -f https://huggingface.co/spaces/{username}/mcp-summarizer-tool
curl -f https://huggingface.co/spaces/{username}/mcp-sentiment-analyzer
# ... etc for all 7 tools
```

### **Automated Notifications**

The pipeline automatically:
- ✅ Posts deployment summaries to PRs
- 🚨 Creates GitHub issues on rollback
- 📊 Uploads test coverage to Codecov
- 🔄 Reports deployment status

---

## 🚨 **Troubleshooting**

### **Common Issues**

#### **1. HF Token Authentication Errors**
```bash
# Verify token has write permissions
huggingface-cli whoami

# Test token manually
export HF_TOKEN="your_token_here"
huggingface-cli upload --repo-type space --repo-id "test/test-space" --help
```

#### **2. Space Creation Failures**
```bash
# Pre-create spaces if needed
huggingface-cli repo create --type space "username/space-name"

# Check space permissions
huggingface-cli repo info "username/space-name"
```

#### **3. Deployment Script Permissions**
```bash
# Fix script permissions locally
chmod +x deploy_all_mcp_tools.sh
git add deploy_all_mcp_tools.sh
git commit -m "fix: deployment script permissions"
```

#### **4. Test Failures**
```bash
# Run tests locally first
pytest tests/ -v --tb=short

# Check requirements
pip install -r requirements.txt -r requirements-dev.txt

# Validate test configuration
python -m pytest --collect-only tests/
```

---

## ✅ **Production Readiness Checklist**

### **Pre-Deployment**
- [ ] All secrets configured correctly
- [ ] HF tokens have write permissions
- [ ] Test suite passes (516 tests)
- [ ] Requirements files up to date
- [ ] Deployment scripts executable

### **Post-Deployment**
- [ ] All 8 HF Spaces deployed successfully
- [ ] Main platform accessible and functional
- [ ] MCP tools respond to health checks
- [ ] Integration tests pass
- [ ] Performance metrics within targets (<2s)

### **Monitoring Setup**
- [ ] GitHub notifications enabled
- [ ] Codecov integration working
- [ ] Error tracking configured
- [ ] Rollback procedures tested

---

## 🎯 **Quick Start Commands**

```bash
# 1. Clone and setup
git clone https://github.com/your-org/kgraph-mcp-hackathon
cd kgraph-mcp-hackathon

# 2. Configure secrets
gh secret set HF_TOKEN --body "hf_your_token"
gh secret set HF_USERNAME --body "your-username"

# 3. Test deployment
git checkout -b test/deployment
git commit --allow-empty -m "test: trigger deployment"
git push origin test/deployment
gh pr create --title "Test Deployment" --body "Testing CD pipeline"

# 4. Monitor results
gh run watch
```

**Result**: 8 HF Spaces deployed automatically with comprehensive testing and monitoring!

---

**Status**: ✅ **PRODUCTION READY**  
**Pipeline**: 🚀 **Enhanced Multi-Track HF Deployment**  
**Coverage**: 🎯 **Track 1 (MCP Tools) + Track 3 (Agent Demo)**