src/system_tests/profiling/CHANGELOG.md

Profiling System Changelog

2025-09-29 - Major Reorganization

Added

• ✅ Organized all profiling-related files into system_tests/profiling/ directory
• ✅ YAML configuration support using Dynaconf
• ✅ Retry mechanism with exponential backoff for Langfuse data fetching
• ✅ Main entry point script: run_experiment.sh
• ✅ HTTP server script: serve.sh for viewing results
• ✅ Interactive charts and visualizations in comparison HTML
• ✅ Environment variable configuration per experiment run
• ✅ Provider comparison support (OpenAI vs Azure vs WatsonX)
• ✅ Comprehensive documentation with examples

Interactive Visualizations

• 📊 Bar charts for execution time, cost, tokens, and LLM calls
• 📊 Variability chart showing Min/Avg/Max execution times with:
  • Range calculation (Max - Min)
  • Standard deviation for statistical analysis
  • Coefficient of variation (CV%) for relative consistency
• ⚡ Stacked bar chart for time breakdown
• 📈 Radar chart for normalized performance comparison
• 🎨 Color-coded modes with modern UI design
• 📋 Tab navigation between charts and detailed tables

Structure

system_tests/profiling/
├── run_experiment.sh          # Main entry point for users
├── bin/                        # Internal scripts
│   ├── profile_digital_sales_tasks.py
│   ├── run_profiling.sh
│   └── run_experiment.sh
├── config/                     # YAML configurations
│   ├── default_experiment.yaml
│   ├── fast_vs_accurate.yaml
│   └── .secrets.yaml
├── experiments/                # Experiment results
│   └── comparison.html
└── reports/                    # Individual profiling reports

Configuration Features

• YAML-based configuration with Dynaconf integration
• CLI arguments override config file settings
• Environment variable support for credentials
• Flexible experiment definitions with multiple runs
• Configurable retry settings for Langfuse data fetching

Retry Mechanism

• Exponential backoff: Starting at 2s, multiplying by 1.5x each attempt
• Smart detection: Checks for 404 errors and incomplete observations
• Configurable: Set max_attempts and initial_delay in YAML
• Default: 10 attempts over ~60 seconds total wait time

Migration from Old Structure

Old files moved/removed:

• profile_digital_sales_tasks.py → system_tests/profiling/bin/
• run_profiling.sh → system_tests/profiling/bin/
• run_experiment.sh → system_tests/profiling/bin/ (rewritten)
• update_html.py → removed (HTML loads JSON dynamically)
• PROFILING_README.md → system_tests/profiling/README.md
• experiments/ → system_tests/profiling/experiments/
• profiling_report_*.json → system_tests/profiling/reports/

Usage

# Run experiment with config file
./system_tests/profiling/run_experiment.sh

# With specific config
./system_tests/profiling/run_experiment.sh --config fast_vs_accurate.yaml

# View results in browser
./system_tests/profiling/serve.sh --open

# Serve on different port
./system_tests/profiling/serve.sh --port 3000

Breaking Changes

• Old scripts at project root no longer work
• Use ./system_tests/profiling/run_experiment.sh instead
• Configuration now uses YAML instead of CLI-only

Benefits

• 📁 Better organization - all profiling files in one place
• ⚙️ Easier configuration - YAML files instead of long CLI commands
• 🔄 More reliable - retry mechanism for Langfuse data
• 📊 Automatic HTML generation - no need to run update script
• 🎯 Clear entry point - single script for users