Inference Latency Optimizer

Name: OpenModels
Creator: OpenModels
License: https://github.com/openmodelsrun/openmodels

ПродвинутыйopsМинимальный контекст: 32K

Reduces end-to-end latency for LLM-backed features by attacking the right bottleneck. Separates time-to-first-token from generation throughput, applies streaming, prompt and prefix caching, speculative decoding, output length limits, parallel tool calls, and model routing, and shows how to measure each change against p50 and p95 rather than averages.

latency inference performance streaming caching optimization

Варианты использования

Diagnosing whether TTFT or generation speed is the bottleneck
Applying prefix and prompt caching to repeated system prompts
Cutting perceived latency with streaming and partial rendering
Routing easy requests to a faster, cheaper model

Пример промпта

Our chat feature has a p95 response time of 9 seconds and users are complaining.

Setup: single frontier model, 6K-token system prompt resent every turn, no streaming, agent makes
up to 4 sequential tool calls.

Identify the likely bottlenecks in priority order, give concrete fixes for each (caching,
streaming, parallel tool calls, output limits, model routing), and estimate the p95 improvement
per change. Tell me what to instrument to confirm the gains.

Совместимые инструменты

claude-codecursorkiroany

Модальности

Вход: text, code

→

Выход: text, code

Inference Latency Optimizer

Варианты использования

Пример промпта

Рекомендуемые модели

Совместимые инструменты

Модальности

Похожие Skills