An autoencoder compresses the input down to a small representation, then reconstructs it back to the original size. The dimensions shrink in the encoder, 64 → 32 → 16 → 8, and mirror back in the decoder, 8 → 16 → 32 → 64.
The narrowest point is the bottleneck, and the code is what passes through it. It forces the network to learn a compact representation of whatever we feed it. Everything the decoder needs to reconstruct the output must pass through those 8 dimensions.
Here I have joined the encoder and the decoder at the code. Together they form an hourglass.
The transformer uses the same encoder–decoder shape. An encoder compresses the input tokens into a compact representation; a decoder reconstructs output tokens from it. The building blocks get fancier (attention, feed-forward layers, residual connections), but the hourglass is still there.