In brief
- Record model resolution and fallback state.
- Separate provider latency from end-to-end task time.
- Prefer safe metadata over raw prompt logging.
Start with the questions incidents ask
Which model served the request? Did routing or fallback change it? Was the slowdown in queueing, provider latency, streaming, or a downstream tool? Did the response complete, and did the task succeed?
Capture a compact request record
A useful event combines correlation IDs, requested and resolved model identity, timestamps, token usage, status, retry count, and a privacy-safe task category.
| Group | Fields |
|---|---|
| Identity | request ID, trace ID, tenant-safe key |
| Routing | requested lane, selected model, resolved model |
| Performance | queue, first token, total latency |
| Usage | input, output, cached tokens, estimated cost |
| Outcome | status, finish reason, retry, fallback, task result |
Do not make raw prompts the default
Prompts can contain source code, customer data, secrets, or private conversations. Prefer hashes, sizes, categories, schema validation results, and sampled redacted payloads with explicit retention controls.
Build dashboards around decisions
Track reliability and latency by resolved model, task lane, and customer segment. Add quality or task-success signals where possible. A global average is useful for a status page but insufficient for model selection.
Frequently asked
Questions, answered plainly.
Should AI prompts be logged?+
Not by default. Use privacy-safe metadata and only retain redacted payloads under an explicit policy when debugging requires them.
What is time to first token?+
It measures the delay before streamed output begins. Track it separately from total completion time because users experience them differently.
How do I monitor model quality?+
Connect requests to task-specific outcomes such as schema validity, test results, reviewer acceptance, or user feedback—not only technical success.
Sources and next paths