Non-Functional Requirements for AI Features

Why generic SLOs aren't enough

HTTP 200 and p99 < 200ms are necessary but insufficient. An LLM endpoint can return 200 with a wrong answer, a policy violation, or a $2 invoice for one request. AI features need quality SLIs alongside latency and availability.

See Observability for SLI/SLO mechanics — here we define what to measure for LLM-backed products.

The AI NFR matrix

Dimension	Example SLI	Typical target (varies by product)
Latency	Time to first token (TTFT), end-to-end	TTFT < 500ms; E2E < 5s for chat
Availability	Successful completion (non-error, non-timeout)	99.9%
Quality	Eval pass rate, human rating, task success	> 90% on golden set
Cost	$/session, tokens/DAU	Budget cap per feature
Safety	Policy block rate, escalation rate	Near-zero critical violations
Freshness	Retrieval lag after doc update	< 15 min for internal docs

Quality SLOs are harder than latency SLOs — define them with PM and legal early.

Latency budgets decompose differently

LLM latency is not one hop:

Gateway (5ms) + Retrieval (50ms) + Model prefill (200ms) + Decode (2s) + Policy (30ms)

• TTFT drives perceived responsiveness (streaming helps)
• Decode scales with output length — cap max_tokens
• Prefill scales with input — context engineering is a latency lever

For async workloads (summarization), user-facing SLO is job completion time, not API RTT.

Cost as a first-class NFR

At 1M sessions/day, $0.01/session = $3.65M/year. Engineering owns:

• Per-tenant and per-user budgets
• Alerts on cost anomaly (not just error rate)
• Routing to cheaper models for low-stakes paths

Finance will ask for unit economics before headcount for a bigger GPU cluster.

Safety and compliance NFRs

Enterprise customers care about:

• Data not used for training (zero-retention APIs)
• PII not logged in prompts
• Audit trail for generated content
• Geographic data residency

These are contractual NFRs — fail them and you lose the deal, regardless of model quality.

Error budgets for AI features

Google SRE error budgets apply: if quality SLO is 95% and you're at 93%, freeze prompt experiments and fix regression before shipping new features.

Distinguish:

• Vendor-induced drift (model update) → eval gates, pinned versions
• Self-induced drift (prompt change) → canary + rollback

Interview framing

When asked "what metrics would you track?":

1. Golden signals: latency, traffic, errors, saturation (model queue depth)
2. AI-specific: token usage, eval score, retrieval hit rate, policy blocks
3. Tie alerts to user impact, not model logits

Senior signal: Propose separate SLOs for "answer delivered" vs "answer correct" — and explain how you'd measure the second without blocking every request on human review.

Link forward

Architecture Patterns shows how NFRs drive sync vs async vs streaming path choices.