ControlNet clones the encoder; LoRA adds a low-rank delta
ControlNet
frozen SD U-Net
encoder
bottleneck
decoder
decoder
ControlNet clone (trainable)
enc copy
bottleneck copy
depth map
zero-conv
zero-conv init => starts as identity; learns a delta
LoRA
W' = W + α · B · A
W
d × d, frozen
+
B
d × r
A
r × d
r = 4-16 typical; rank-r compression
params: 2 · d · r instead of d²
runtime knob: α ∈ [0.5, 1.5]
composability in 2026 pipelines
ControlNet
spatial (pose, depth,
edges, scribble, seg)
70-360 MB per modality
LoRA
style, subject, concept
20-200 MB
stack multiple with α scaling
IP-Adapter
reference image as condition
via CLIP image tokens
~20 MB
DreamBooth
full fine-tune of base
strongest identity
2-5 GB