Layer geometry.
A forward pass has a shape.
Treat each layer as a step and a prompt traces a path through representation space. Those paths are not arbitrary: related meanings fall into the same basin, harder questions bend further, and an ambiguous word visibly splits in two at a depth that holds across three different models.
Trajectory Geometry of Transformer Representations Across Layers. Vishal Pandey, Gopal Singh and Yacine Mahdid, Metriqual.
- Phases in the forward pass
- 3
- in all three models
- Curvature, reasoning over lexical
- 2.7×
- 0.71–0.83 against 0.27–0.31 rad
- Depth where meaning splits
- ≈22%
- consistent across architectures
- Prompts, five families
- 150
- 30 each, versioned
Measure the path, not the layer.
Interpretability usually inspects a layer, a head or a neuron. This measures the trajectory those components produce.
Each prompt is run once with hidden states retained, mean-pooled per layer into a single vector, and stacked into a path of L+1 points through the model's representation space. Four geometric properties of that path are then measured: length, curvature, how far related prompts converge, and how much each layer changes its input.
None of it needs labelled data, a probing classifier, fine-tuning or any change to the model. That matters because a probe can only find what you already told it to look for, and it needs a training set to do it. These metrics come out of the forward pass itself.
Every metric is computed in the model's full ambient dimension. Projections are used for the pictures only, and no reported number depends on one, which is the failure mode that makes a lot of representation visualisation unreliable.
Models
- GPT-2 Small
- 117M, 12 layers
- TinyLlama
- 1.1B, 22 layers
- Qwen2.5
- 1.5B, 28 layers
- Prompts
- 150, five families
- Prompt length
- 5–15 tokens
- Decoding
- greedy, no sampling
- Hardware
- 1× RTX 3090
- Full extraction
- under 4 hours
The middle of the network is where meaning concentrates.
Layerwise cosine similarity separates the forward pass into three phases with the same proportional boundaries in every model tested.
Semantically related prompts converge as they pass through. The convergence index sits near zero in early layers and rises sharply from the midpoint, peaking between 0.41 and 0.58 depending on the model. Shuffle the category labels and it collapses to roughly 0.02, so the effect is learned semantic structure rather than a coincidence of the space. Shuffle the layer order instead and the monotonic rise disappears, which puts the cause in the sequence of layers rather than in the set of representations they produce.
Harder questions bend the path further.
Mean trajectory curvature separates multi-step reasoning from surface-form variation by a factor of about 2.7, in every model.
| Prompt family | GPT-2 | TinyLlama | Qwen2.5 |
|---|---|---|---|
| Multi-step reasoning | 0.78 | 0.83 | 0.71 |
| Analogical reasoning | 0.54–0.61 | 0.54–0.61 | 0.54–0.61 |
| Lexical variation | 0.31 | 0.29 | 0.27 |
Curvature peaks in a consistent band: layers 2 to 5 of 12 in GPT-2, 5 to 10 of 22 in TinyLlama, 5 to 9 of 28 in Qwen2.5. That is roughly 20 to 45% of depth in all three, and it lines up with the layers where prior mechanistic work locates induction head formation and MLP-based knowledge retrieval. Two very different methods pointing at the same region is the useful part.
Because curvature needs no probe and no labels, it could work as an unsupervised readout of how much computation an input is demanding, computed during inference. The paper is careful to call that a hypothesis for future work rather than a result, and so are we.
Ambiguity resolves gradually, not at one layer.
The same word in two disambiguating contexts starts as one point and comes apart over a span of layers.
River bank against savings bank. At layer 0 the two representations are essentially the same point, 0.11 apart in GPT-2's normalised space. From about layer 5 they separate monotonically, reaching 0.67 by the final layer: a 5.6× spread. TinyLlama and Qwen2.5 do the same thing at 4.9× and 5.1×.
Matched unambiguous controls in the same syntactic frames stay flat at 1.1×, so this is not two prompts drifting apart because they are two prompts. Shuffling the layer order destroys the monotonic ordering, which again puts the effect in the learned sequence.
The onset lands at roughly 20 to 25% of depth in all three models. If the commitment is progressive rather than instantaneous, then an intervention aimed at redirecting an interpretation has a window: before the split completes rather than after.
Bifurcation
- GPT-2
- 5.6× spread
- TinyLlama
- 4.9× spread
- Qwen2.5
- 5.1× spread
- Unambiguous controls
- 1.1×
- Onset, GPT-2
- ≈22% depth
- Onset, TinyLlama
- ≈23% depth
- Onset, Qwen2.5
- ≈21% depth
- Homograph pairs
- 15
Four ways to try to break the result.
Every finding is reported against a null built to produce the same effect by accident.
| Control | What it removes | What happens |
|---|---|---|
| Random labels | semantic grouping | convergence collapses to ≈0.02 |
| Random embeddings | trained weights | curvature gap falls under 0.05 rad |
| Shuffled layers | layer ordering | monotonic rise and split disappear |
| Multiple projections | one view of the data | structure holds across PCA, UMAP, t-SNE |
Three models is a consistency check, not a universal law.
The strongest claim this design supports is that the structure is consistent where it was measured.
All three models are decoder-only and none exceeds 1.5B parameters. That is enough to check whether a result survives a change of architecture and training corpus. It is not enough to claim universality: the specific layer indices could well move in a 70B model, in encoder-only architectures, or in encoder-decoder models, even if the qualitative shape survives.
The prompt set is deliberately controlled for semantic precision, which makes it linguistically narrow. Multilingual, long-context and domain-specific inputs need their own validation. Trajectories are mean-pooled sequence vectors, so anything positional or syntactic that lives at the token level is invisible here; tracking each position separately is possible but the memory cost grows with sequence length, depth and width at once.
The most important limit is the kind of claim being made. These findings are observational. Geometric structure correlates with semantic and computational properties, and nothing here shows that the geometry causes any downstream behaviour. That needs activation patching or representation surgery, and it is the obvious next step rather than something being asserted now.
The whole pipeline is open.
Extraction, metric computation, statistical validation and visualisation ship together, along with the versioned 150-prompt dataset. It runs on one consumer GPU in under four hours per model, and once hidden states are saved the metrics reproduce on a CPU.