I saved this one for last, because it is the only architecture here that starts from two inputs. A two-tower network processes them through independent MLPs, then combines their outputs. Each tower learns its own representation. This pattern appears in recommendation systems (user tower + item tower) and in contrastive learning (two views of the same data). Each tower can have different input sizes and different compression rates. What matters is that their outputs combine at the end.
Connecting the towers is a concatenation, stacking the outputs on top of each other. The final output is a linear projection of the concatenated towers.
The most far-reaching use is multi-modal: give one tower text and the other images, and the two towers do not even have to read the same kind of data. A picture goes up one side, a caption goes up the other, and training pulls the matching pairs together in the same output space. That is how CLIP works, and it is why we can ask a model to find the photo that goes with a sentence. Once both modalities land in one space, the comparison is just arithmetic.
Next:
12. Heads
Awesome PROF. TOM YEH