Auto-Optimizing Inference with GPU Profiling and Telemetry

By Dmitri Melikyan | Jul 17, 2026

vLLM, SGLang, and other engines each expose dozens of startup flags, and they interact. The best configuration is not a property of the model - it’s a property of the model, hardware and the traffic hitting it. A config tuned for short interactive chats is the wrong config for long-context prompts, which is the wrong config for offline batch. Defaults are deliberately conservative, and a config you hand-tuned last month goes stale the moment your traffic shifts.

auto-flags closes that gap. You wrap your engine launch with graphsignal-run --auto-flags, and Graphsignal picks the startup flags from your workload’s own GPU profiling and telemetry - and keeps improving them as your traffic changes. You run your own stack, but you don’t hand-tune it or accept opaque managed defaults.

Traffic-pattern-specific configurations

There is no single “fast” configuration. What’s optimal depends on the shape of your traffic:

• Agentic / harness loops - an agent makes many model calls per task, and modern harnesses route different steps to different models: a small, fast model for tool selection and routing, a larger one for the heavy reasoning. Each step is its own workload - tool-selection calls are short, structured, and latency-critical; reasoning calls are longer and less frequent.
• Interactive chat - short prompts, streaming responses, latency-sensitive. Decode-bound; you optimize for low TTFT and inter-token latency, often at modest concurrency.
• Long-context - large prompts, prefill-heavy. Chunked prefill and prefix caching matter far more than raw decode speed.
• Offline / batch - throughput is the only metric that matters. You want large batches and high memory utilization, and you don’t care about per-request latency.
• Bursty - traffic arrives in spikes. Queueing behavior and headroom for autoscaling dominate.

That first shape is increasingly the norm. Agent harnesses already split work across models - Claude Code, for example, has a larger model plan and orchestrate while a team of the far faster Haiku handles sub-tasks and tool use - and a wave of small models is being built specifically for the tool-calling step, such as Tencent’s open-source Hunyuan series (0.5B–7B), which target agentic tool use and run on consumer-grade GPUs. So a single agent deployment already mixes several traffic shapes at once, and the small tool-selection model and the large reasoning model each want a different engine configuration - exactly the kind of per-workload tuning auto-flags is built for.

The point of profiling is that you don’t have to guess which of these you are. You measure it. A profile - GPU kernel activity, engine and GPU metrics, and request traces for a window of real traffic - is the object that maps your actual workload to a configuration. It’s what the optimization agent reasons over, and it’s why the flags auto-flags picks fit your traffic rather than a generic recipe.

A real example. In one experiment, a profiling-driven loop tuned SGLang on a single H100 for high-concurrency traffic with short, unique prompts. Because no two prompts shared a prefix, the prefix cache was pure overhead - so the winning move was counter-intuitive: turn the prefix cache off, which lifted the concurrency ceiling roughly 5×; enlarging the state cache then soaked up idle GPU. Peak throughput rose about 3.6× (16.4 → 58.9 req/s), and time-to-first-token under load dropped about 16× (12.9s → 0.82s). The same model on the same GPU wants the opposite setting from a shared-prefix chat workload - and only the profile tells you which.

The auto-flags loop

auto-flags automates exactly that - it turns each run’s profile into engine flags for you. Turning it on is a single option added to your existing launch command, with no code changes:

export GRAPHSIGNAL_API_KEY="..."
graphsignal-run --auto-flags vllm serve <model>
graphsignal-run --auto-flags sglang serve --model-path <model>
graphsignal-run --auto-flags trtllm-serve <model>

At launch, auto-flags sends Graphsignal your startup command, the engine and its version, and your GPU and platform details. Graphsignal returns recommended engine flags and merges them into your command: new flags are appended, and flags you already set are replaced. If anything fails, your command launches exactly as you wrote it; auto-flags is strictly best-effort and never blocks your service from starting.

The very first launch is already useful: with no telemetry yet, Graphsignal matches a strong starting configuration to your model, engine, and hardware, drawing on established tuning recipes and documentation. Once your service has run under real traffic, each subsequent launch also uses the previous run’s profile and telemetry, tuning against the flags that were actually in effect - so the configuration compounds toward your workload rather than resetting.

The loop, end to end:

graphsignal-run --auto-flags <cmd>
  ↓ requests recommended flags
Optimization agent
  ↓ fetches previous run telemetry, if available: GPU profiles, engine metrics, errors, etc.
  ↓ then draws on optimization knowledge: recipes, engine docs, etc.
  ↓ reasons and returns optimized flags
Inference engine
  ↓ starts with the optimized flags
  ↻ each engine restart reruns the loop

Built on Graphsignal profiling

auto-flags works because the feedback signal underneath it is good enough to tune against. The Graphsignal profiler attaches to your engine as a low-overhead sidecar - no code changes, no root - and continuously collects CUDA profiles, engine metrics (via OpenTelemetry and Prometheus), and GPU metrics, all tied to real production traffic. That’s the profile the optimization agent reads to find where latency and GPU time actually go, and to propose flags that move TTFT, inter-token latency, and utilization in the right direction.

The optimization agent is not a general-purpose assistant - it’s a harness built specifically for inference optimization. It runs a purpose-designed loop with the tools to read profiles and telemetry, and grounds its recommendations in engine-, model-, and hardware-specific knowledge, so its output is concrete startup flags or instructions rather than generic advice. And because it runs inline at launch, its response time is part of your engine’s startup time - so the harness is tuned for speed: fast models and a tight, bounded loop that return flags in seconds, not minutes.

Because the captured telemetry contains no prompts or completions - only performance signal - this is safe to run on production traffic.

For efficiency, recommendations are cached - auto-flags doesn’t run a fresh optimization on every launch. It reuses the current best flags and re-optimizes when there’s meaningful new telemetry to learn from, so frequent restarts stay cheap.

To get started, install the profiler and add --auto-flags to your launch:

UV_TOOL_BIN_DIR=/usr/local/bin uv tool install 'graphsignal[cu12]'   # CUDA 12.x
# or 'graphsignal[cu13]' for CUDA 13.x

export GRAPHSIGNAL_API_KEY=<my-api-key>
graphsignal-run --auto-flags vllm serve <model>

auto-flags is the automatic, deploy-time way to act on your profiling data, but not the only one. When you want to dig in yourself, Optimize chat in the Graphsignal dashboard lets you ask about any time window and get flag and configuration guidance grounded in the same profiles. And when you’re in your editor, graphsignal-context gives your AI coding agent access to the profiles, metrics, and traces for a chosen time range, so it can reason about performance and root-cause issues without leaving your IDE.

See the AI Optimization guide and the Profiler CLI reference for details, and follow @GraphsignalAI for updates.