Skip to content

Proxy Gateway

The LLM Telemetry Proxy is an asynchronous reverse proxy built with Python aiohttp. It forwards requests transparently while calculating metrics, managing concurrency, and enforcing safety policies.

⚙ Core Capabilities

1. Adaptive Rate Limiting & Concurrency Control

Many upstream LLM providers (or private clusters) enforce strict concurrent request limits per API key or tenant (e.g., maximum 4 parallel requests). When client code fires parallel async queries, this often results in 429 Too Many Requests errors or connection teardown race conditions.

The proxy utilizes an asynchronous FIFO concurrency manager with slot cooldown gating and transparent 429 auto-retries:

  • Strict Max Concurrency: Gated by UpstreamConcurrencyLimiter (default 4, configurable via --max-concurrent or MAX_CONCURRENT).
  • Slot Cooldown Gap (50ms): When active concurrency is maxed out and queued requests are waiting, a small cooldown gap (--slot-cooldown-ms 50 or CONCURRENCY_SLOT_COOLDOWN_MS) is strictly enforced when releasing a slot before admitting the next queued request. This ensures upstream TCP sockets, reverse proxy gateways, and vLLM/engine sequence teardown complete cleanly before the next request arrives.
  • Opt-in 429 Auto-Retry with Exponential Backoff: By default (--retry-429-max 0), upstream 429 responses are passed directly to the caller and logged to SQLite immediately to preserve 100% telemetry fidelity and avoid compounding retries with downstream agents (e.g. Hermes). If desired for dumb scripts, passing --retry-429-max N enables automatic exponential backoff retries with jitter before failing.

2. Rolling 24-Hour Token Budget Supervisor

The proxy provides hard token consumption caps to prevent runaway agent loops from exhausting monthly quotas or incurring unexpected bills.

  • Default Cap: 480,000,000 tokens / 24 hours (configurable via DAILY_TOKEN_LIMIT).
  • Rolling Window: Token usage is calculated across a continuous 86,400-second window.
  • Persistent State Across Restarts:
  • Restores active usage from data/llm_telemetry.db and fallback snapshots in data/token_budget.json.
  • Response Headers: 
    X-Token-Budget-Used: 12450000
X-Token-Budget-Remaining: 467550000
X-Token-Budget-Percentage: 2.59
X-Token-Budget-Limit: 480000000
  • Rejection Behavior: If the cap is reached, requests are safely rejected with HTTP 429 and a descriptive JSON payload: 
    {
  "error": {
    "type": "token_budget_exceeded",
    "message": "🚫 DAILY TOKEN BUDGET EXCEEDED\nUsed: 480,120,000 tokens (100.0% of daily limit)..."
  }
}

3. Canonical Model Name Resolution

To avoid fragmentation in dashboards where clients pass aliases (e.g. glm, glm-5, or coder instead of canonical names), the proxy utilizes data/model_mapping.json to canonicalize names in RAM before persisting to disk.

{
  "glm-5.2": ["glm", "glm-5", "glm-5.2"],
  "deepseek-v4-flash": ["deepseek", "deepseek-v4-flash"],
  "qwen3.5-122b": ["coder", "agentic", "qwen3.5-122b"]
}

4. Real-Time Raw Payload Logging & Privacy Sanitizer

When debugging prompts, formatting errors, tool calling schemas, or model reasoning tokens, developers can enable Raw Payload Logging via the dashboard or REST API:

curl -X POST http://localhost:9090/v1/raw-log/toggle -H "Content-Type: application/json" -d '{"enabled": true}'

Privacy-Preserving Header Masking

To prevent accidental credential leaks into logs, make_raw_payload_record automatically sanitizes authorization headers: - Authorization: Bearer sk-proj-... → Bearer sk-p...cdef - api-key, x-api-key, x-auth-token → Masked safely

SSE Live Streaming

The proxy offers a Server-Sent Events (SSE) feed at /v1/raw-log/stream that transmits structured JSON payloads to the connected Payload Inspector UI with zero polling overhead.

5. 24/7 Production Engine & Native C-Extension Acceleration

For permanent server deployments with minimal CPU and RAM footprints, the proxy combines native C/Rust runtime accelerators with Nuitka Native C-Extension compilation (--module):

Production Accelerators & Compilation Modes

Accelerator Impact How
Native C-Extensions (.so) Native C machine code for routing, parsing, streaming, and DB logging ./start.sh build (Nuitka --module) — compiled to dist/modules/
uvloop (C / libuv) 2x–3x async I/O socket throughput pip install uvloop — auto-detected at startup
orjson (Rust SIMD) 5x–10x faster JSON serialization pip install orjson — auto-detected at startup
jemalloc Zero Linux glibc malloc heap fragmentation sudo apt install libjemalloc2 — auto-preloaded by start.sh

Measured Production Footprint

Metric Python Script Native C-Modules (.so)
Execution Path Bytecode eval loop Native compiled C machine code
Idle RAM (RSS) ~69 MB ~35–45 MB
Peak RAM (under load) ~72 MB ~45–50 MB
Memory Creep < 5 MB/hour < 2 MB/hour
Active OS Threads 2 2
Open FDs 28 28

Compiling Native C-Extension Modules

# Compile proxy modules to native .so shared libraries
./start.sh build

# Restart proxy (automatically prioritizes compiled .so modules)
./start.sh proxy restart

Benchmarking & Profiling Tool (benchmark_monitor.py)

A standalone profiler is included in scripts/benchmark_monitor.py to sample real-time RSS memory, CPU%, threads, open sockets, and response latencies:

# Profile current proxy for 30 minutes
python scripts/benchmark_monitor.py record --output baseline.csv --duration 1800

# Compare two benchmark recordings
python scripts/benchmark_monitor.py compare baseline.csv native.csv