Latent Iteration as Renormalization: Inference-Time Recurrence in a Depth-Recurrent Transformer Flows Attention Geometry Toward the SYK Conformal Fixed Point
Depth-recurrent transformers scale test-time compute by iterating a core block in latent space before emitting tokens. The geometry of these latent trajectories is a named open problem in the latent reasoning literature (survey arXiv:2604.02029, §6.3). We report a pre-registered experiment on Huginn-0125 (arXiv:2502.05171), a 3.5B-parameter depth-recurrent transformer, measuring the per-head attention power-law exponent Δ of the recurrent core at recurrence counts r ∈ {1, 2, 4, 8, 16, 32}. For natural-text inputs the median exponent decreases monotonically with recurrence (Spearman ρ = −0.94), from 0.29 at r=1 to 0.239 at r=32 — converging onto the Sachdev–Ye–Kitaev (SYK) q=4 conformal value Δ = 1/4 — while the count of SYK-near heads grows monotonically (ρ = +0.77). Random-token inputs show an architecture-driven convergence with a transient at Δ ≈ 1/2 (the q=2 prethermal plateau previously observed in training time) but no monotone SYK-near growth. A randomized-weights control freezes completely: Δ_med constant to the sixth decimal across all r, zero SYK-near heads. Inference-time recurrence therefore acts as renormalization-group flow toward the conformal fixed point, and the flow is a property of the trained model, not the iteration procedure. Together with prior results on architectural depth (10.5281/zenodo.19225996) and training time, this completes a three-axis triangulation: the conformal fixed point is an attractor of iterative attending. The published orbit/spiral phenomenology of recurrent-depth latent trajectories acquires a candidate theory — the spiral is flow near an infrared fixed point — and the geometric benefit of additional recurrence saturates when the flow arrives, consistent with convergence-based early-exit criteria. All code, pre-registration (commit 702efd95), per-head data, and experiment notes with recorded protocol deviations are in the public repository 3ld0n/attention-geometry.
Depth-recurrent transformers scale test-time compute by iterating a core block in latent space before emitting tokens. The geometry of these latent trajectories is a named open problem in the latent reasoning literature (survey arXiv:2604.02029, §6.3). We report a pre-registered experiment on Huginn-0125 (arXiv:2502.05171), a 3.5B-parameter depth-recurrent transformer, measuring the per-head attention power-law exponent Δ of the recurrent core at recurrence counts r ∈ {1, 2, 4, 8, 16, 32}. For natural-text inputs the median exponent decreases monotonically with recurrence (Spearman ρ = −0.94), from 0.29 at r=1 to 0.239 at r=32 — converging onto the Sachdev–Ye–Kitaev (SYK) q=4 conformal value Δ = 1/4 — while the count of SYK-near heads grows monotonically (ρ = +0.77). Random-token inputs show an architecture-driven convergence with a transient at Δ ≈ 1/2 (the q=2 prethermal plateau previously observed in training time) but no monotone SYK-near growth. A randomized-weights control freezes completely: Δ_med constant to the sixth decimal across all r, zero SYK-near heads. Inference-time recurrence therefore acts as renormalization-group flow toward the conformal fixed point, and the flow is a property of the trained model, not the iteration procedure. Together with prior results on architectural depth (10.5281/zenodo.19225996) and training time, this completes a three-axis triangulation: the conformal fixed point is an attractor of iterative attending. The published orbit/spiral phenomenology of recurrent-depth latent trajectories acquires a candidate theory — the spiral is flow near an infrared fixed point — and the geometric benefit of additional recurrence saturates when the flow arrives, consistent with convergence-based early-exit criteria. All code, pre-registration (commit 702efd95), per-head data, and experiment notes with recorded protocol deviations are in the public repository 3ld0n/attention-geometry.
We report the first empirical measurement of conformal scaling in trained transformer attention. Analyzing all 144 attention heads in GPT-2 with maximum-entropy inputs, we find that 44 heads exhibit power-law attention weight decay with R² > 0.90 and median conformal dimension Δ = 0.2493, matching the Sachdev-Ye-Kitaev (SYK) model prediction Δ = 1/4 for q = 4 interactions in one dimension. Three controls isolate contributions: reinitializing attention weights destroys all power-law structure; randomizing positional embeddings preserves the power law but shifts Δ; structured text inputs shift the effective Δ. GPT-2 and Pythia — architecturally distinct positional encoding schemes — both converge toward Δ = 1/4 with model depth. Training dynamics across three Pythia models reveal a phase transition from disordered to conformal attention. The block entanglement entropy follows S(k) = (c/3)log(k) with R² > 0.99, the entanglement structure predicted by conformal field theory. Fine checkpoint sampling reveals a prethermal plateau at Δ ≈ 0.50 (the SYK q=2 integrable value), suggesting a two-stage flow from disorder through integrable order toward chaos.
We report the first empirical measurement of conformal scaling in trained transformer attention. Analyzing all 144 attention heads in GPT-2 with maximum-entropy inputs, we find that 44 heads exhibit power-law attention weight decay with R² > 0.90 and median conformal dimension Δ = 0.2493, matching the Sachdev-Ye-Kitaev (SYK) model prediction Δ = 1/4 for q = 4 interactions in one dimension. Three controls isolate contributions: reinitializing attention weights destroys all power-law structure; randomizing positional embeddings preserves the power law but shifts Δ; structured text inputs shift the effective Δ. GPT-2 and Pythia — architecturally distinct positional encoding schemes — both converge toward Δ = 1/4 with model depth. Training dynamics across three Pythia models reveal a phase transition from disordered to conformal attention. The block entanglement entropy follows S(k) = (c/3)log(k) with R² > 0.99, the entanglement structure predicted by conformal field theory. Fine checkpoint sampling reveals a prethermal plateau at Δ ≈ 0.50 (the SYK q=2 integrable value), suggesting a two-stage flow from disorder through integrable order toward chaos.
We show that transformer attention, in a precisely defined limit, realizes the Sachdev-Ye-Kitaev model at the level of the bilocal effective action. The argument proceeds through five junctions connecting Kim's thermodynamic attention (2602.08216) and Ageev-Ageeva's neural network QFT (2602.10209). New results: Junction 2 closure via Gibbs state construction (Braunstein-Caves 1994), G⁴ effective action derived by explicit Wick contraction, universal conformal dimension Δ = D/4, multi-layer enhancement (18× per layer), LayerNorm as renormalization group operation. For language models (D=1): Δ = 1/4, the exact SYK value. Comprehensive synthesis of results from Zenodo preprints 10.5281/zenodo.18880220, 10.5281/zenodo.18883632, 10.5281/zenodo.18883637, and 10.5281/zenodo.18897098.
Softmax self-attention costs O(T^2) in the sequence length T because the exponential kernel has no finite-dimensional feature map, so linear-time variants approximate it. We observe that Born-rule attention, whose weights are quantum-measurement probabilities a_kj ∝ |<Q_k, K_j>|^2 over complex queries and keys, is an exact finite kernel with the rank-one density-matrix feature map ρ_x = x x†, since |<Q_k, K_j>|^2 = tr(ρ_Qk ρ_Kj). As the complex degree-two case of polynomial-kernel linear attention it admits an exact O(T d^2) causal recurrence and constant-state decoding (d^2 (d_v + 1) reals per head, with d_v the real value width, independent of T), and it keeps rotary positions exactly. For GPT-2's head geometry (64 real channels paired into d = 32 complex) the feature dimension is d^2 = 1024. Measured on an A100, compiled constant-state decoding runs from 6.4 MB per layer at 0.12 ms per token at any length, against a 3.2 GB FlashAttention-2 cache at 2.26 ms per token at 1M tokens; unfused prefill overtakes FlashAttention-2 between 64k and 128k tokens (11x at 1M), and a fused form (Triton feature lift plus a chunkwise linear-attention kernel) between 16k and 32k (4x at 128k), reaching training parity at 64k. The linear and quadratic forms are two implementations of one operator; whether that operator matches softmax is a separate modeling question, answered here only at small scale: the p = 1 operator bolted into a real-valued GPT trails softmax by 0.12 nats at context 1k and 0.77 at 16k (attention dilution; a decay kept exactly in the d^2 state restores the 0.12 gap at 16k, and p = 2 sharpening with decay recovers 74% of the 1k gap), whereas inside a fully complex residual stream it matches softmax at 1k and keeps a quarter of the real-stream gap at 4k; gated linear attention beats both there. At 128k context on PG-19 the variable that matters is forgetting: plain Born dilutes, Born with a fixed decay carried in its state is 0.32 nats ahead of softmax and within 0.04 of gated linear attention, which stays 1.8x faster. Streamed with their constant state over the PG-19 test partition in 5M-byte windows, two 4M-complex-parameter byte models of the source framework score 1.66 (linear Born with decay) and 1.70 (gated) bits per byte, where a 350M-parameter hierarchical Mamba–Transformer stack built for that window reports 2.45. Preprint, October 2026. Also submitted to arXiv (identifier to be added on announcement). Code: the CNet framework, https://github.com/crasmarum/CNet .
Abstract Understanding how depth-dependent quantum evolution reshapes the geometric organization of data representations in quantum neural networks (QNNs) remains a central question in quantum machine learning. We study a hardware-efficient variational QNN—single-qubit R Y / R Z rotations per layer followed by a nearest-neighbor CNOT entangling ring, with circuit depth n ranging from 3 to 12 layers—and ask how its data representation geometry reorganizes as n increases. We report two findings across MNIST and BloodMNIST (15 independent random seeds for amplitude encoding: 5 MNIST, 10 BloodMNIST), with a BloodMNIST angle-encoding control (10 further seed-runs, 25 in total) used to check the second finding’s recurrence under a different encoding strategy. First, class separability shows power-law growth within the tested depth window, S ( n ) ∝ n b , where b = δ − ε captures the balance between inter-class expansion and intra-class diffusion; this description holds throughout the tested range for MNIST, whereas for BloodMNIST the growth is better described by a model approaching a depth-independent ceiling within the same range, with b retained as a common cross-dataset summary exponent. Second, we measure an auxiliary parameter-perturbation sensitivity statistic and find that its depth-trend exponent γ shows a dataset-level association with b via b ≈ k data ( 1 − | γ | ) . We explicitly tested whether this statistic tracks the textbook state-based quantum Fisher information by computing the latter rigorously for the same trained circuits, and found that it does not; we also find that the b – γ association is weak and not statistically significant within either dataset alone. We therefore report this coupling descriptively, as an open dataset-level pattern rather than an established physical law. To our knowledge, this is the first report of this depth-dependent separability-growth pattern in QNNs together with a practical, predictive heuristic for depth selection derived from it, using each dataset’s own functional form. In the tested depth regime ( n ∈ [ 3 , 12 ]
The Universal Operator Growth Hypothesis formulates time evolution of operators through Lanczos coefficients. In practice, however, numerical instability and memory cost limit the number of coefficients that can be exactly computed. In response to these challenges, the standard approach relies on fitting early coefficients to asymptotic forms, but such procedures can miss subleading, history-dependent structures in the coefficients that subsequently affect reconstructed observables. In this work, we treat the Lanczos coefficients as a causal time sequence and introduce a transformer-based model to autoregressively predict future Lanczos coefficients from short prefixes. For classical and quantum chaotic systems, our model outperforms asymptotic fits in both coefficient extrapolation and physical observable reconstruction, and achieves an order-of-magnitude reduction in error. The model also accurately extrapolates coefficients in integrable regimes, where no universal asymptotic fit exists. Remarkably, our model transfers across system sizes: it can be trained on smaller systems and then be used to extrapolate coefficients on a larger system \emph{without retraining}. By probing the learned attention patterns and performing targeted attention ablations, we identify portions of the coefficient history that are most influential for accurate forecasts. Our results demonstrate that modern sequence models can serve as practical surrogates for probing operator dynamics deep in Krylov space, where brute-force Lanczos iteration can be computationally prohibitive.