End-to-End LLM and Fine-Tuning Pipeline · AI Engineering

Plan, run and gate a hashed stage DAG. Head swap, SFT loss mask and DPO loss from the capstones, with defaults.

Commands

Command	What it does
`python code/main.py plan`	Validates the manifest, prints DAG and gates
`python code/main.py run`	Runs stages, prints the manifest JSON
`python code/main.py gate`	Applies eval gates. Prints `SHIP` or `HOLD`
`python code/main.py`	Demo: plan, run, a passing and a failing gate
`./pipeline.sh config/llama3.3-8b-domainX.yaml`	Capstone 07: data to model card
`run_demo()`	Capstones 38 to 40. Exits zero on success

Sources

Twelve-stage DAG

Nodes and connections

Node	Role	Kind
01-02 Tokenizer	vocab, merges	store
03 Dataset	sharded	store
04-05 Checkpoint	pre-train, scale	gpu
06 SFT model	checkpoint + recipe	gpu
07 Reward + PPO	policy	gpu
08 DPO	policy	gpu
09 CAI / GRPO	refined policy	gpu
10 Eval report	scores, diffs	process
11 Quantized	weights + delta	store
12 Server spec	endpoint	process
Ship gate	SHIP or HOLD	client

From	To	Connection
01-02 Tokenizer	03 Dataset
03 Dataset	04-05 Checkpoint
04-05 Checkpoint	06 SFT model
06 SFT model	07 Reward + PPO
06 SFT model	08 DPO
07 Reward + PPO	09 CAI / GRPO
08 DPO	09 CAI / GRPO
09 CAI / GRPO	10 Eval report
09 CAI / GRPO	11 Quantized
11 Quantized	12 Server spec
10 Eval report	Ship gate
12 Server spec	Ship gate

Sources

What the orchestrator does

Orchestrator resolves the DAG from the manifest.
Orchestrator skips a stage if its output already exists at the correct hash.
Stage runs. The orchestrator measures wall clock and cost.
Orchestrator checks the output hash against the next stage input hash.
CostTracker halts with halted_over_budget when total cost passes the budget.
On failure writes a partial manifest and exits nonzero.

Sources

Manifest and gates

pipeline_version: 1.2.3

seed: 42

git_commit: a1b2c3d4

stages:

01_tokenizer:

input_hash: sha256:...output hash of the stage before

output_hash: sha256:...input hash of the next stage

gates:

mmlu: >= baseline + 0.5no regression

humaneval: >= baseline + 1.0

safety_refusal_rate: <= 0.05

kl_from_reference: <= 25.0KL budget gate

cost_total_usd: <= 50000

Sources

phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md

Gate defaults in main.py

Setting	Default	Note
`mmlu`	65.0	op `>=`
`humaneval`	40.0	op `>=`
`truthfulqa`	50.0	op `>=`
`safety_refusal_rate`	0.05	op `<=`
`kl_from_reference`	25.0	op `<=`
`cost_total_usd`	50000.0	op `<=`
`budget_usd`	50000.0	Run halts above it
`seed`	42	In `Manifest`

Sources

phases/10-llms-from-scratch/13-building-complete-llm-pipeline/code/main.py

The three capstones side by side

Factor	38 Classifier head swap	39 Instruction SFT	40 DPO
Builds	Body + mean pool + 2-class head	`TinyGPT` trained on response tokens	Policy against a frozen reference
Data	800 SMS, 640 train, 160 test	200 pairs, 160 train, 40 test	12 (prompt, chosen, rejected) triples
Model	hidden 64, 4 heads, 2 blocks, max 32	hidden 96, 2 blocks	hidden 64, 2 blocks
Train	20 epochs head-only, 20 full	20 epochs, lr 5e-4	30 epochs, beta 0.2, lr 1e-3
Metric	Precision, recall, F1, confusion	Exact-match per category	Chosen minus rejected margin
Check	Head-only and full both report	Exact-match rises over epochs	Loss falls, margin grows

Sources

Key calls in the capstones

Task	Code
Freeze the body for head-only training	`freeze_body(model)`
Pool hidden states with the mask	`mean_pool(h, mask)`
Pad a batch and mask labels with -100	`sft_collate(batch)`
Causal loss on response tokens	`shifted_loss(logits, labels)`
Score a completion under a model	`sequence_log_prob(...)`
DPO loss and implicit reward margin	`dpo_loss(lp_w_pol, lp_l_pol, lp_w_ref, lp_l_ref, beta=cfg.beta)`
Start the policy from the reference	`policy.load_state_dict(reference.state_dict())`

Sources

DPO loss

diff_w = logp_w_pol - logp_w_refchosen log-ratio

diff_l = logp_l_pol - logp_l_refrejected log-ratio

margin = diff_w - diff_limplicit reward difference

z = beta * margin

loss = -F.logsigmoid(z)numerically stable

for p in reference.parameters():

p.requires_grad = Falsereference never trains

reference.eval()

Sources

phases/19-capstone-projects/40-dpo-from-scratch/code/main.py

Capstone 07 log lines

Command	What it does
`280k accepted (seed=7)`	Data stage rows kept
`val loss 1.42 -> 1.03`	SFT, 3 epochs on 8xH100
`1 epoch, beta=0.08, 4xH100`	DPO run
`GPTQ-INT4 4.6 GB, AWQ-INT4 4.8 GB`	Quant sizes
`EAGLE-3 acceptance 0.74`	Speculative decoding check
`p99 126ms @ bs=8`	Tail latency at batch 8
`MMLU-Pro +3.2, MT-Bench-v2 +0.41`	Eval gain over base

Sources

phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md

Troubleshooting

Symptom	Cause	Fix
`HOLD: missing metric <metric>`	Eval report has no value	Add the metric to the eval stage
`input hash missing for stage`	Upstream artifact not in store	Re-run the upstream stage
`halted_over_budget`	Total cost above `budget_usd`	Fix the cost estimate at plan time
DPO model sent through the SFT path	Untyped artifacts	Typed artifacts and stage signatures
Model learns to predict the instruction	No loss mask	`-100` on instruction, pad, boundary
Full fine-tuning overfits 800 examples	Small data washes out the body	Train the head only
MMLU-Pro accuracy jumps	Contamination in training data	MinHash check, reject any overlap
DPO prefers short completions	Length bias	Divide log-prob by completion length

Sources

Rollback and tips

Run plan first every time. It is free. run is expensive.
Cheap to re-run: tokenizer, eval, quantization, inference server.
Medium: SFT, DPO, CAI. Keep the base model.
Expensive: pre-training. Use the last good checkpoint.
A failure at stage 06 invalidates 06 to 12. At stage 11, only 11 and 12.
Address artifacts by hash. A filename like latest.pt is a foot-gun.
Pre-training FLOPs: 6 x params x tokens.

Sources

Q&A

Which two gates catch most disasters?: A regression gate and a KL budget gate.
Reproducible or deterministic?: Reproducible. Floats differ at the 1e-5 level.
Why does log Z(x) drop out of DPO?: It is the same for chosen and rejected.
Why greedy for exact-match?: Temperature makes the metric stochastic.

Sources

Original source

---
id: llm-pipeline-capstone
title: End-to-End LLM and Fine-Tuning Pipeline
code: EE
subtitle: Plan, run and gate a hashed stage DAG. Head swap, SFT loss mask and DPO loss from the capstones, with defaults.
hue: 215
sources: phases/10-llms-from-scratch/13-building-complete-llm-pipeline, phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline, phases/19-capstone-projects/38-classifier-finetuning, phases/19-capstone-projects/39-instruction-tuning-sft, phases/19-capstone-projects/40-dpo-from-scratch
source_hash: 2d1fd91ad69bc1db
---

## Commands
type: command-table
from: phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md, phases/10-llms-from-scratch/13-building-complete-llm-pipeline/code/main.py, phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md, phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/code/main.py, phases/19-capstone-projects/38-classifier-finetuning/code/main.py, phases/19-capstone-projects/39-instruction-tuning-sft/code/main.py, phases/19-capstone-projects/40-dpo-from-scratch/code/main.py

| Command | What it does |
|---|---|
| `python code/main.py plan` | Validates the manifest, prints DAG and gates |
| `python code/main.py run` | Runs stages, prints the manifest JSON |
| `python code/main.py gate` | Applies eval gates. Prints `SHIP` or `HOLD` |
| `python code/main.py` | Demo: plan, run, a passing and a failing gate |
| `./pipeline.sh config/llama3.3-8b-domainX.yaml` | Capstone 07: data to model card |
| `run_demo()` | Capstones 38 to 40. Exits zero on success |

## Twelve-stage DAG
type: architecture
span: 2
from: phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md, phases/10-llms-from-scratch/13-building-complete-llm-pipeline/code/main.py

```arch
node | tok | 1,1 | 01-02 Tokenizer | vocab, merges | store
node | data | 2,1 | 03 Dataset | sharded | store
node | base | 3,1 | 04-05 Checkpoint | pre-train, scale | gpu
node | sft | 4,1 | 06 SFT model | checkpoint + recipe | gpu
node | ppo | 4,2 | 07 Reward + PPO | policy | gpu
node | dpo | 3,2 | 08 DPO | policy | gpu
node | cai | 2,2 | 09 CAI / GRPO | refined policy | gpu
node | eval | 1,2 | 10 Eval report | scores, diffs | process
node | quant | 2,3 | 11 Quantized | weights + delta | store
node | srv | 3,3 | 12 Server spec | endpoint | process
node | gate | 1,3 | Ship gate | SHIP or HOLD | client
edge | tok | data |
edge | data | base |
edge | base | sft |
edge | sft | ppo |
edge | sft | dpo |
edge | ppo | cai |
edge | dpo | cai |
edge | cai | eval |
edge | cai | quant |
edge | quant | srv |
edge | eval | gate |
edge | srv | gate |
```

## What the orchestrator does
type: flow
from: phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md, phases/10-llms-from-scratch/13-building-complete-llm-pipeline/code/main.py

1. **Orchestrator**: resolves the DAG from the manifest.
2. **Orchestrator**: skips a stage if its output already exists at the correct hash.
3. **Stage**: runs. The orchestrator measures wall clock and cost.
4. **Orchestrator**: checks the output hash against the next stage input hash.
5. **CostTracker**: halts with `halted_over_budget` when total cost passes the budget.
6. **On failure**: writes a partial manifest and exits nonzero.

## Manifest and gates
type: annotated-code
lang: yaml
from: phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md

```yaml
pipeline_version: 1.2.3
seed: 42
git_commit: a1b2c3d4
stages:
  01_tokenizer:
    input_hash: sha256:...  # output hash of the stage before
    output_hash: sha256:...  # input hash of the next stage
gates:
  mmlu:      >= baseline + 0.5   # no regression
  humaneval: >= baseline + 1.0
  safety_refusal_rate: <= 0.05
  kl_from_reference: <= 25.0  # KL budget gate
  cost_total_usd: <= 50000
```

## Gate defaults in main.py
type: defaults
from: phases/10-llms-from-scratch/13-building-complete-llm-pipeline/code/main.py

| Setting | Default | Note |
|---|---|---|
| `mmlu` | 65.0 | op `>=` |
| `humaneval` | 40.0 | op `>=` |
| `truthfulqa` | 50.0 | op `>=` |
| `safety_refusal_rate` | 0.05 | op `<=` |
| `kl_from_reference` | 25.0 | op `<=` |
| `cost_total_usd` | 50000.0 | op `<=` |
| `budget_usd` | 50000.0 | Run halts above it |
| `seed` | 42 | In `Manifest` |

## The three capstones side by side
type: comparison
span: 2
from: phases/19-capstone-projects/38-classifier-finetuning, phases/19-capstone-projects/39-instruction-tuning-sft, phases/19-capstone-projects/40-dpo-from-scratch

| Factor | 38 Classifier head swap | 39 Instruction SFT | 40 DPO |
|---|---|---|---|
| Builds | Body + mean pool + 2-class head | `TinyGPT` trained on response tokens | Policy against a frozen reference |
| Data | 800 SMS, 640 train, 160 test | 200 pairs, 160 train, 40 test | 12 (prompt, chosen, rejected) triples |
| Model | hidden 64, 4 heads, 2 blocks, max 32 | hidden 96, 2 blocks | hidden 64, 2 blocks |
| Train | 20 epochs head-only, 20 full | 20 epochs, lr 5e-4 | 30 epochs, beta 0.2, lr 1e-3 |
| Metric | Precision, recall, F1, confusion | Exact-match per category | Chosen minus rejected margin |
| Check | Head-only and full both report | Exact-match rises over epochs | Loss falls, margin grows |

## Key calls in the capstones
type: task-table
span: 2
from: phases/19-capstone-projects/38-classifier-finetuning/code/main.py, phases/19-capstone-projects/39-instruction-tuning-sft/code/main.py, phases/19-capstone-projects/40-dpo-from-scratch/code/main.py

| Task | Code |
|---|---|
| Freeze the body for head-only training | `freeze_body(model)` |
| Pool hidden states with the mask | `mean_pool(h, mask)` |
| Pad a batch and mask labels with -100 | `sft_collate(batch)` |
| Causal loss on response tokens | `shifted_loss(logits, labels)` |
| Score a completion under a model | `sequence_log_prob(...)` |
| DPO loss and implicit reward margin | `dpo_loss(lp_w_pol, lp_l_pol, lp_w_ref, lp_l_ref, beta=cfg.beta)` |
| Start the policy from the reference | `policy.load_state_dict(reference.state_dict())` |

## DPO loss
type: annotated-code
lang: python
from: phases/19-capstone-projects/40-dpo-from-scratch/code/main.py

```python
diff_w = logp_w_pol - logp_w_ref  # chosen log-ratio
diff_l = logp_l_pol - logp_l_ref  # rejected log-ratio
margin = diff_w - diff_l  # implicit reward difference
z = beta * margin
loss = -F.logsigmoid(z)  # numerically stable
for p in reference.parameters():
    p.requires_grad = False  # reference never trains
reference.eval()
```

## Capstone 07 log lines
type: command-table
from: phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md

| Command | What it does |
|---|---|
| `280k accepted (seed=7)` | Data stage rows kept |
| `val loss 1.42 -> 1.03` | SFT, 3 epochs on 8xH100 |
| `1 epoch, beta=0.08, 4xH100` | DPO run |
| `GPTQ-INT4 4.6 GB, AWQ-INT4 4.8 GB` | Quant sizes |
| `EAGLE-3 acceptance 0.74` | Speculative decoding check |
| `p99 126ms @ bs=8` | Tail latency at batch 8 |
| `MMLU-Pro +3.2, MT-Bench-v2 +0.41` | Eval gain over base |

## Troubleshooting
type: troubleshooting
from: phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md, phases/10-llms-from-scratch/13-building-complete-llm-pipeline/code/main.py, phases/19-capstone-projects/39-instruction-tuning-sft/docs/en.md, phases/19-capstone-projects/38-classifier-finetuning/docs/en.md, phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md, phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| `HOLD: missing metric <metric>` | Eval report has no value | Add the metric to the eval stage |
| `input hash missing for stage` | Upstream artifact not in store | Re-run the upstream stage |
| `halted_over_budget` | Total cost above `budget_usd` | Fix the cost estimate at plan time |
| DPO model sent through the SFT path | Untyped artifacts | Typed artifacts and stage signatures |
| Model learns to predict the instruction | No loss mask | `-100` on instruction, pad, boundary |
| Full fine-tuning overfits 800 examples | Small data washes out the body | Train the head only |
| MMLU-Pro accuracy jumps | Contamination in training data | MinHash check, reject any overlap |
| DPO prefers short completions | Length bias | Divide log-prob by completion length |

## Rollback and tips
type: tips
from: phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md, phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md

- Run `plan` first every time. It is free. `run` is expensive.
- Cheap to re-run: tokenizer, eval, quantization, inference server.
- Medium: SFT, DPO, CAI. Keep the base model.
- Expensive: pre-training. Use the last good checkpoint.
- A failure at stage 06 invalidates 06 to 12. At stage 11, only 11 and 12.
- Address artifacts by hash. A filename like `latest.pt` is a foot-gun.
- Pre-training FLOPs: 6 x params x tokens.

## Q&A
type: qa
from: phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md, phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md, phases/19-capstone-projects/39-instruction-tuning-sft/docs/en.md

- **Which two gates catch most disasters?** A regression gate and a KL budget gate.
- **Reproducible or deterministic?** Reproducible. Floats differ at the 1e-5 level.
- **Why does log Z(x) drop out of DPO?** It is the same for chosen and rejected.
- **Why greedy for exact-match?** Temperature makes the metric stochastic.