We report the first empirical measurement of conformal scaling in trained transformer attention. Analyzing all 144 attention heads in GPT-2 with maximum-entropy inputs, we find that 44 heads exhibit power-law attention weight decay with R² > 0.90 and median conformal dimension Δ = 0.2493, matching the Sachdev-Ye-Kitaev (SYK) model prediction Δ = 1/4 for q = 4 interactions in one dimension. Three controls isolate contributions: reinitializing attention weights destroys all power-law structure; randomizing positional embeddings preserves the power law but shifts Δ; structured text inputs shift the effective Δ. GPT-2 and Pythia — architecturally distinct positional encoding schemes — both converge toward Δ = 1/4 with model depth. Training dynamics across three Pythia models reveal a phase transition from disordered to conformal attention. The block entanglement entropy follows S(k) = (c/3)log(k) with R² > 0.99, the entanglement structure predicted by conformal field theory. Fine checkpoint sampling reveals a prethermal plateau at Δ ≈ 0.50 (the SYK q=2 integrable value), suggesting a two-stage flow from disorder through integrable order toward chaos.
We report the first empirical measurement of conformal scaling in trained transformer attention. Analyzing all 144 attention heads in GPT-2 with maximum-entropy inputs, we find that 44 heads exhibit power-law attention weight decay with R² > 0.90 and median conformal dimension Δ = 0.2493, matching the Sachdev-Ye-Kitaev (SYK) model prediction Δ = 1/4 for q = 4 interactions in one dimension. Three controls isolate contributions: reinitializing attention weights destroys all power-law structure; randomizing positional embeddings preserves the power law but shifts Δ; structured text inputs shift the effective Δ. GPT-2 and Pythia — architecturally distinct positional encoding schemes — both converge toward Δ = 1/4 with model depth. Training dynamics across three Pythia models reveal a phase transition from disordered to conformal attention. The block entanglement entropy follows S(k) = (c/3)log(k) with R² > 0.99, the entanglement structure predicted by conformal field theory. Fine checkpoint sampling reveals a prethermal plateau at Δ ≈ 0.50 (the SYK q=2 integrable value), suggesting a two-stage flow from disorder through integrable order toward chaos.
We report the first empirical measurement of conformal scaling in trained transformer attention. Analyzing all 144 attention heads in GPT-2 with maximum-entropy inputs, we find that 44 heads exhibit power-law attention weight decay with R² > 0.90 and median conformal dimension Δ = 0.2493, matching the Sachdev-Ye-Kitaev (SYK) model prediction Δ = 1/4 for q = 4 interactions in one dimension. Three controls isolate contributions: reinitializing attention weights destroys all power-law structure; randomizing positional embeddings preserves the power law but shifts Δ; structured text inputs shift the effective Δ. GPT-2 and Pythia — architecturally distinct positional encoding schemes — both converge toward Δ = 1/4 with model depth. Training dynamics across three Pythia models reveal a phase transition from disordered to conformal attention. The block entanglement entropy follows S(k) = (c/3)log(k) with R² > 0.99, the entanglement structure predicted by conformal field theory. Fine checkpoint sampling reveals a prethermal plateau at Δ ≈ 0.50 (the SYK q=2 integrable value), suggesting a two-stage flow from disorder through integrable order toward chaos.
Depth-recurrent transformers scale test-time compute by iterating a core block in latent space before emitting tokens. The geometry of these latent trajectories is a named open problem in the latent reasoning literature (survey arXiv:2604.02029, §6.3). We report a pre-registered experiment on Huginn-0125 (arXiv:2502.05171), a 3.5B-parameter depth-recurrent transformer, measuring the per-head attention power-law exponent Δ of the recurrent core at recurrence counts r ∈ {1, 2, 4, 8, 16, 32}. For natural-text inputs the median exponent decreases monotonically with recurrence (Spearman ρ = −0.94), from 0.29 at r=1 to 0.239 at r=32 — converging onto the Sachdev–Ye–Kitaev (SYK) q=4 conformal value Δ = 1/4 — while the count of SYK-near heads grows monotonically (ρ = +0.77). Random-token inputs show an architecture-driven convergence with a transient at Δ ≈ 1/2 (the q=2 prethermal plateau previously observed in training time) but no monotone SYK-near growth. A randomized-weights control freezes completely: Δ_med constant to the sixth decimal across all r, zero SYK-near heads. Inference-time recurrence therefore acts as renormalization-group flow toward the conformal fixed point, and the flow is a property of the trained model, not the iteration procedure. Together with prior results on architectural depth (10.5281/zenodo.19225996) and training time, this completes a three-axis triangulation: the conformal fixed point is an attractor of iterative attending. The published orbit/spiral phenomenology of recurrent-depth latent trajectories acquires a candidate theory — the spiral is flow near an infrared fixed point — and the geometric benefit of additional recurrence saturates when the flow arrives, consistent with convergence-based early-exit criteria. All code, pre-registration (commit 702efd95), per-head data, and experiment notes with recorded protocol deviations are in the public repository 3ld0n/attention-geometry.
We show that transformer attention, in a precisely defined limit, realizes the Sachdev-Ye-Kitaev model at the level of the bilocal effective action. The argument proceeds through five junctions connecting Kim's thermodynamic attention (2602.08216) and Ageev-Ageeva's neural network QFT (2602.10209). New results: Junction 2 closure via Gibbs state construction (Braunstein-Caves 1994), G⁴ effective action derived by explicit Wick contraction, universal conformal dimension Δ = D/4, multi-layer enhancement (18× per layer), LayerNorm as renormalization group operation. For language models (D=1): Δ = 1/4, the exact SYK value. Comprehensive synthesis of results from Zenodo preprints 10.5281/zenodo.18880220, 10.5281/zenodo.18883632, 10.5281/zenodo.18883637, and 10.5281/zenodo.18897098.
In response to expert feedback that earlier papers constitute structural analogies rather than explicit physical constructions, we identify the Sachdev-Ye-Kitaev (SYK) model as a specific path toward rigorous physical construction of holographic attention. We show that the independence-breaking mechanism in Ageev and Ageeva (2026) is structurally identical to SYK disorder averaging: both are covariances of bilocal quantities over Gaussian random parameters. We derive the score covariance exactly and show it factorizes in the SYK-compatible form. We show that the single-layer Schwinger-Dyson equation is linear — not the nonlinear SYK form — and explain precisely why. We identify three tractable calculations (multi-layer SD equation, linearized-softmax replica calculation, single-layer conformal fixed point) that would close the correspondence. If any resolves positively, the holographic dual of transformer attention is JT gravity and the island formula follows from Almheiri-Penington (2019) as a known result, not by analogy.
Recently, the steady states of non-unitary free fermion dynamics are found to exhibit novel critical phases with power-law squared correlations and a logarithmic subsystem entanglement. In this work, we theoretically understand the underlying physics by constructing solvable static/Brownian quadratic Sachdev-Ye-Kitaev chains with non-Hermitian dynamics. We find the action of the replicated system generally shows (one or infinite copies of) O(2)×O(2) symmetries, which is broken to O(2) by the saddle-point solution. This leads to an emergent conformal field theory of the Goldstone modes. We derive their effective action and obtain the universal critical behaviors of squared correlators. Furthermore, the entanglement entropy of a subsystem A with length LA corresponds to the energy of the half-vortex pair S∼ρslogLA, where ρs is the stiffness of the Goldstone mode. We also discuss special limits where more than one Goldstone mode exists and comment on interaction effects.
This work develops tools to understand how quantum information spreads, scrambles, and is reshaped by measurements in many-body systems. First, I study scrambling and pseudorandomness in the Brownian Sachdev-Ye-Kitaev (SYK) model, quantifying pseudorandomness using unitary k-designs and frame potentials. Using Keldysh path integrals with replicas and disorder averaging, I obtain analytic control of the approach to randomness, identify collective modes that delay convergence to Haar-like behavior, and estimate design times as functions of model parameters, clarifying links between scrambling, complexity growth, and random-circuit phenomenology. Second, I construct a field theory for weakly measured SYK clusters. Starting from a system-ancilla description and a continuum monitoring limit, and using fermionic coherent states with replicas and disorder averaging, I derive a nonlinear sigma model that captures measurement back-action and the competition between interaction-induced scrambling and information extraction, predicting characteristic crossover scales and response signatures that distinguish weak monitoring from fully unitary evolution. Third, I develop a strong-disorder renormalization group for measurement-only SYK clusters, based on the SO(2n) replica algebra and Dasgupta-Ma decimation rules. The flow shows features reminiscent of infinite-randomness behavior, but an order-of-limits subtlety renders the leading recursions non-robust, so the analytic evidence for an infinite-randomness fixed point is inconclusive, even though the average second Renyi entropy displays logarithmic scaling. Together, these results provide a unified language to diagnose when many-body dynamics generate operational randomness and how measurements redirect that flow.