CLIP similarity matrix — positives on the diagonal, negatives everywhere else
similarity matrix S (N=4)
S[i,j] = cos(img_i, txt_j) / tau
txt 0
txt 1
txt 2
txt 3
img 0
+0.82
-0.11
+0.04
-0.22
img 1
-0.18
+0.77
+0.12
+0.09
img 2
+0.06
+0.14
+0.79
-0.03
img 3
-0.21
+0.08
+0.03
+0.84
InfoNCE = - sum log softmax(diag)
each row pushes the diagonal up, negatives down
symmetric: also do it column-wise
training ingredients
· 400M image-text pairs (CLIP), 10B+ for SigLIP 2
· batch 32k-512k
· learnable temperature tau (init 0.07)
· dual encoder: ViT + small text transformer
· normalize both embeddings before cosine
softmax vs sigmoid loss
InfoNCE (CLIP)
per row: softmax normalizes across N
needs full similarity matrix in sync
distributed: all-gather every batch
comm cost: O(world_size x batch x D)
loss_i2t = CE(S, eye)
loss_t2i = CE(S^T, eye)
loss = (loss_i2t + loss_t2i) / 2
temperature controls sharpness
scale ceiling: 32k batch before comm dominates
Sigmoid pairwise (SigLIP)
per pair: independent BCE
y=1 on diagonal, y=0 off-diagonal
loss = -y log sig(S+b) - (1-y) log sig(-S-b)
no all-gather; local blocks only
comm cost: O(world_size x D)
scale ceiling: 512k+ batch feasible
extra bias parameter b handles class imbalance
SigLIP 2 (2025) ships with NaFlex
+ multilingual (100+ langs)