Shipping a Production LLM App · AI Engineering

Request pipeline, retries, fallback chains, SSE streams, resume, demo limits and a fallback gateway.

Lesson tasks

Task	Code
Install the server stack	`pip install fastapi uvicorn`
Run the app with 4 workers	`uvicorn production_app:app --host 0.0.0.0 --port 8000 --workers 4`
Pick the primary model	`LLM_MODEL=gpt-4o`
Start the demo server	`python3 code/main.py --serve`
Require an access token	`export DEMO_ACCESS_TOKEN=<token>`
Send one prediction	`curl -s -X POST http://127.0.0.1:8000/api/predict -H 'Content-Type: application/json' -d '{"text": "ok"}'`
Watch the event stream	`curl -N -X POST http://127.0.0.1:8000/api/stream -H 'Content-Type: application/json' -d '{"text": "<text>"}'`
Run the demo tests	`python3 -m unittest discover -s code/tests -v`

Sources

Request pipeline

Nodes and connections

Node	Role	Kind
Client	web, mobile, API	client
API Gateway	auth, rate limit	process
Input guardrails	injection, PII redact	process
Prompt Router	template, A/B version	process
Semantic Cache	threshold 0.92	store
LLM call	retry, fallback chain	model
Output guardrails	flagged text replaced	process
Response	eval log, cost, SSE	client

From	To	Connection
Client	API Gateway
API Gateway	Input guardrails
Input guardrails	Prompt Router	pass
Prompt Router	Semantic Cache
Semantic Cache	LLM call	miss
LLM call	Output guardrails
Output guardrails	Response

Sources

Retry with backoff

async def call_llm_with_retry(prompt, model, max_retries=3):

for attempt in range(max_retries + 1):

try:

...one provider call

except (ConnectionError, TimeoutError):

if attempt < max_retries:

backoff = min(2 ** attempt + random.uniform(0, 1), 10)cap 10 s

await asyncio.sleep(backoff)

else:

raise

raise ConnectionError(f"All {max_retries} retries exhausted for {model.value}")

call_with_fallback walks FALLBACK_CHAIN, preferred model first.

Sources

phases/11-llm-engineering/13-production-app/code/production_app.py

Failure handling

Symptom	Cause	Fix
API 429	Rate limit	Retry with backoff. Queue the request
API 500	Server error	Retry 3 times. Switch to the fallback model
API timeout over 30s	Slow provider	Retry once. Shorter prompt, smaller model
Malformed output	Model failure	Retry with the error in the prompt
Guardrail block	Unsafe input	No retry. Explain the block
Vector store down	Application failure	Skip RAG context
Cache down	Application failure	Direct LLM call
All models fail	Chain exhausted	Cached response, then a fixed apology

Sources

phases/11-llm-engineering/13-production-app/docs/en.md

Defaults in the code

Setting	Default	Note
`similarity_threshold`	0.92	Semantic cache hit score
`ttl_seconds`	3600	Cache entry life
`max_entries`	10000	Cache size
`max_retries`	3	Per model in the chain
`MAX_CHARS`	500	Demo text limit
`MAX_BODY_BYTES`	4096	Larger body gives 413
`BUCKET_CAPACITY`	5	Refill 0.5 token per second
`DAILY_CALL_CAP`	2000	Over the cap gives 503

Sources

Demo error codes

Command	What it does
`413 body_too_large`	Body over 4,096 bytes
`408 request_timeout`	Body stops before `Content-Length`
`415 unsupported_media_type`	Content type is not JSON
`400 bad_json`	Body is not a JSON object
`422 text_too_long`	Text over 500 characters
`422 control_characters`	Text has control characters
`429 rate_limited`	Token bucket empty. Read `Retry-After`
`503 usage_cap_reached`	Daily cap reached. Read `Retry-After`

Sources

SSE wire format

event: tokenContent-Type text/event-stream

data: {"index":0,"text":"Label:"}

a blank line ends each event

event: token

data: {"index":1,"text":" positive."}

event: done

data: {"label":"positive","score":1.0,"tokens":11}

Sources

phases/17-infrastructure-and-production/29-model-demo-interfaces/docs/en.md

Resume a streamed answer

SSEDecoder joins split UTF-8 bytes into complete events.
apply needs contiguous positive ids. A gap throws noncontiguous answer event ID.
apply a repeated id with the same data is a no-op. Other data throws.
apply payload type is text, citation or complete. Citations need http or https.
connect a drop before complete sets status interrupted.
connect the next call sends Last-Event-ID and gets only later events.
trace writes schemaVersion: 1 with accepted events. replay rebuilds the state.

Sources

Gateway fallback policy

Factor	2xx	429 or 5xx	Other status	Cancellation
`ClassifyStatus`	`success`	`retry`	`terminal`	none
Next provider	No	Yes	No	No
Route state	`completed`	`exhausted` at end	`terminal-error`	`cancelled`
Proxy reply	Upstream body	502 at end	502	504

Sources

Gateway providers file

{"providers": [

{"name": "primary", "url": "http://127.0.0.1:11434/v1/chat/completions"},

{"name": "backup", "url": "http://127.0.0.1:11435/v1/chat/completions",

"key_env": "ORCHARD_BACKUP_KEY"}],env var name, not the key

"max_attempts": 2,1 to 16, one visit per provider

"max_response_bytes": 65536,1 byte to 16 MiB

"timeout_ms": 4000}1 to 5000, whole route

Sources

Project tasks

Task	Code
Make a work folder	`python3 scripts/project_test.py <project> --init /tmp/<project>-work`
Grade one stage	`python3 scripts/project_test.py <project> --stage 1 --path /tmp/<project>-work --strict`
Stream demo, drop then resume	`cd projects/streaming-answer-recovery/solution && python3 demo.py`
Read your own SSE endpoint	`node --experimental-strip-types cli.ts <url> /tmp/trace.json`
Gateway demo, 503 then backup	`cd projects/llm-gateway-with-fallbacks/solution && go run .`
Run the gateway proxy	`go run . --config /tmp/providers.json --listen 127.0.0.1:8088`
Call the proxy	`curl --fail-with-body http://127.0.0.1:8088/v1/chat/completions --data-binary @fixtures/request.json`

Sources

Targets and checklist

P50 latency under 2s. P99 latency under 10s.
Cache hit rate over 30%. Guardrail block rate under 5%.
Cost per request under $0.01.
Rate limit each user at 10 to 50 requests per minute.
Load test with 100 concurrent users before launch.

Sources

phases/11-llm-engineering/13-production-app/docs/en.md

Q&A

Why POST and not EventSource?: EventSource sends GET only. GET puts the input in logged URLs.
Model fails after status 200?: The status is sent. Send an error event in the stream.
What is in an error body?: code, message, hint and request_id.

Sources

phases/17-infrastructure-and-production/29-model-demo-interfaces/docs/en.md

Original source

---
id: production-llm-app
title: Shipping a Production LLM App
code: SP
subtitle: Request pipeline, retries, fallback chains, SSE streams, resume, demo limits and a fallback gateway.
hue: 15
sources: phases/11-llm-engineering/13-production-app, phases/17-infrastructure-and-production/29-model-demo-interfaces, projects/streaming-answer-recovery, projects/llm-gateway-with-fallbacks
source_hash: 55ab30194be15d70
---

## Lesson tasks
type: task-table
span: 2
from: phases/11-llm-engineering/13-production-app/docs/en.md, phases/11-llm-engineering/13-production-app/code/production_app.py, phases/17-infrastructure-and-production/29-model-demo-interfaces/docs/en.md, phases/17-infrastructure-and-production/29-model-demo-interfaces/code/main.py

| Task | Code |
|---|---|
| Install the server stack | `pip install fastapi uvicorn` |
| Run the app with 4 workers | `uvicorn production_app:app --host 0.0.0.0 --port 8000 --workers 4` |
| Pick the primary model | `LLM_MODEL=gpt-4o` |
| Start the demo server | `python3 code/main.py --serve` |
| Require an access token | `export DEMO_ACCESS_TOKEN=<token>` |
| Send one prediction | `curl -s -X POST http://127.0.0.1:8000/api/predict -H 'Content-Type: application/json' -d '{"text": "ok"}'` |
| Watch the event stream | `curl -N -X POST http://127.0.0.1:8000/api/stream -H 'Content-Type: application/json' -d '{"text": "<text>"}'` |
| Run the demo tests | `python3 -m unittest discover -s code/tests -v` |

## Request pipeline
type: architecture
from: phases/11-llm-engineering/13-production-app/docs/en.md, phases/11-llm-engineering/13-production-app/code/production_app.py

```arch
node | client | 1,1 | Client | web, mobile, API | client
node | gw | 2,1 | API Gateway | auth, rate limit | process
node | gin | 3,1 | Input guardrails | injection, PII redact | process
node | router | 4,1 | Prompt Router | template, A/B version | process
node | cache | 4,2 | Semantic Cache | threshold 0.92 | store
node | llm | 3,2 | LLM call | retry, fallback chain | model
node | gout | 2,2 | Output guardrails | flagged text replaced | process
node | resp | 1,2 | Response | eval log, cost, SSE | client
edge | client | gw |
edge | gw | gin |
edge | gin | router | pass
edge | router | cache |
edge | cache | llm | miss
edge | llm | gout |
edge | gout | resp |
```

## Retry with backoff
type: annotated-code
lang: python
from: phases/11-llm-engineering/13-production-app/code/production_app.py

```python
async def call_llm_with_retry(prompt, model, max_retries=3):
    for attempt in range(max_retries + 1):
        try:
            ...  # one provider call
        except (ConnectionError, TimeoutError):
            if attempt < max_retries:
                backoff = min(2 ** attempt + random.uniform(0, 1), 10)  # cap 10 s
                await asyncio.sleep(backoff)
            else:
                raise
    raise ConnectionError(f"All {max_retries} retries exhausted for {model.value}")
# call_with_fallback walks FALLBACK_CHAIN, preferred model first.
```

## Failure handling
type: troubleshooting
from: phases/11-llm-engineering/13-production-app/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| API 429 | Rate limit | Retry with backoff. Queue the request |
| API 500 | Server error | Retry 3 times. Switch to the fallback model |
| API timeout over 30s | Slow provider | Retry once. Shorter prompt, smaller model |
| Malformed output | Model failure | Retry with the error in the prompt |
| Guardrail block | Unsafe input | No retry. Explain the block |
| Vector store down | Application failure | Skip RAG context |
| Cache down | Application failure | Direct LLM call |
| All models fail | Chain exhausted | Cached response, then a fixed apology |

## Defaults in the code
type: defaults
from: phases/11-llm-engineering/13-production-app/code/production_app.py, phases/17-infrastructure-and-production/29-model-demo-interfaces/code/main.py

| Setting | Default | Note |
|---|---|---|
| `similarity_threshold` | 0.92 | Semantic cache hit score |
| `ttl_seconds` | 3600 | Cache entry life |
| `max_entries` | 10000 | Cache size |
| `max_retries` | 3 | Per model in the chain |
| `MAX_CHARS` | 500 | Demo text limit |
| `MAX_BODY_BYTES` | 4096 | Larger body gives 413 |
| `BUCKET_CAPACITY` | 5 | Refill 0.5 token per second |
| `DAILY_CALL_CAP` | 2000 | Over the cap gives 503 |

## Demo error codes
type: command-table
from: phases/17-infrastructure-and-production/29-model-demo-interfaces/docs/en.md, phases/17-infrastructure-and-production/29-model-demo-interfaces/code/main.py

| Command | What it does |
|---|---|
| `413 body_too_large` | Body over 4,096 bytes |
| `408 request_timeout` | Body stops before `Content-Length` |
| `415 unsupported_media_type` | Content type is not JSON |
| `400 bad_json` | Body is not a JSON object |
| `422 text_too_long` | Text over 500 characters |
| `422 control_characters` | Text has control characters |
| `429 rate_limited` | Token bucket empty. Read `Retry-After` |
| `503 usage_cap_reached` | Daily cap reached. Read `Retry-After` |

## SSE wire format
type: annotated-code
lang: bash
from: phases/17-infrastructure-and-production/29-model-demo-interfaces/docs/en.md

```bash
event: token  # Content-Type text/event-stream
data: {"index":0,"text":"Label:"}
  # a blank line ends each event
event: token
data: {"index":1,"text":" positive."}

event: done
data: {"label":"positive","score":1.0,"tokens":11}
```

## Resume a streamed answer
type: flow
from: projects/streaming-answer-recovery/README.md, projects/streaming-answer-recovery/solution/main.ts, projects/streaming-answer-recovery/solution/cli.ts

1. **SSEDecoder**: joins split UTF-8 bytes into complete events.
2. **apply**: needs contiguous positive ids. A gap throws `noncontiguous answer event ID`.
3. **apply**: a repeated id with the same data is a no-op. Other data throws.
4. **apply**: payload type is `text`, `citation` or `complete`. Citations need http or https.
5. **connect**: a drop before `complete` sets status `interrupted`.
6. **connect**: the next call sends `Last-Event-ID` and gets only later events.
7. **trace**: writes `schemaVersion: 1` with accepted events. `replay` rebuilds the state.

## Gateway fallback policy
type: comparison
from: projects/llm-gateway-with-fallbacks/README.md, projects/llm-gateway-with-fallbacks/solution/stage2.go, projects/llm-gateway-with-fallbacks/solution/stage4.go, projects/llm-gateway-with-fallbacks/solution/config.go

| Factor | 2xx | 429 or 5xx | Other status | Cancellation |
|---|---|---|---|---|
| `ClassifyStatus` | `success` | `retry` | `terminal` | none |
| Next provider | No | Yes | No | No |
| Route state | `completed` | `exhausted` at end | `terminal-error` | `cancelled` |
| Proxy reply | Upstream body | 502 at end | 502 | 504 |

## Gateway providers file
type: annotated-code
lang: bash
from: projects/llm-gateway-with-fallbacks/solution/fixtures/providers.example.json, projects/llm-gateway-with-fallbacks/README.md

```bash
{"providers": [
  {"name": "primary", "url": "http://127.0.0.1:11434/v1/chat/completions"},
  {"name": "backup", "url": "http://127.0.0.1:11435/v1/chat/completions",
   "key_env": "ORCHARD_BACKUP_KEY"}],  # env var name, not the key
 "max_attempts": 2,  # 1 to 16, one visit per provider
 "max_response_bytes": 65536,  # 1 byte to 16 MiB
 "timeout_ms": 4000}  # 1 to 5000, whole route
```

## Project tasks
type: task-table
span: 2
from: projects/streaming-answer-recovery/README.md, projects/llm-gateway-with-fallbacks/README.md

| Task | Code |
|---|---|
| Make a work folder | `python3 scripts/project_test.py <project> --init /tmp/<project>-work` |
| Grade one stage | `python3 scripts/project_test.py <project> --stage 1 --path /tmp/<project>-work --strict` |
| Stream demo, drop then resume | `cd projects/streaming-answer-recovery/solution && python3 demo.py` |
| Read your own SSE endpoint | `node --experimental-strip-types cli.ts <url> /tmp/trace.json` |
| Gateway demo, 503 then backup | `cd projects/llm-gateway-with-fallbacks/solution && go run .` |
| Run the gateway proxy | `go run . --config /tmp/providers.json --listen 127.0.0.1:8088` |
| Call the proxy | `curl --fail-with-body http://127.0.0.1:8088/v1/chat/completions --data-binary @fixtures/request.json` |

## Targets and checklist
type: tips
from: phases/11-llm-engineering/13-production-app/docs/en.md

- P50 latency under 2s. P99 latency under 10s.
- Cache hit rate over 30%. Guardrail block rate under 5%.
- Cost per request under $0.01.
- Rate limit each user at 10 to 50 requests per minute.
- Load test with 100 concurrent users before launch.

## Q&A
type: qa
from: phases/17-infrastructure-and-production/29-model-demo-interfaces/docs/en.md

- **Why POST and not `EventSource`?** `EventSource` sends GET only. GET puts the input in logged URLs.
- **Model fails after status 200?** The status is sent. Send an `error` event in the stream.
- **What is in an error body?** `code`, `message`, `hint` and `request_id`.