How deep can a signal go before it dies?
A deep network is a signal passed through many layers. Whether it survives or scrambles is a phase transition. I use mean-field theory to predict the boundary — replicating Poole et al. (2016) on random tanh networks.
The idea
Borrow the dictionary of phase transitions and point it at a network. Depth plays the role of distance; the correlation length becomes the number of layers a pattern can survive; the edge of chaos is the knife-edge where that survival depth diverges. Sit there and a signal propagates through arbitrarily many layers without fading or scrambling — which is exactly where deep networks train best.
Signal length
The first quantity is the variance of activations, q. Its layer-to-layer map has a single stable fixed point, and both theory and simulation converge to it within a handful of layers — the weight scale sets where.
Order vs. chaos
The second quantity is the correlation between two inputs as they descend the network. Below the critical weight scale every pair collapses to identical (ordered); above it, nearby inputs are driven apart (chaotic). The crossover is sharp.
The phase boundary
Setting the correlation slope χ1 = 1 traces the order-to-chaos boundary across weight and bias variance — the line a network should be initialised on to train at any depth.
χ1 = 1 curve separates the ordered and chaotic phases in the (weight, bias) plane.What's next
The clean theory assumes a symmetric network. Real ones break that symmetry, blurring the independent modes together. The next step is a sparse autoencoder used as a measurement instrument — to read the hidden modes back out where the closed-form theory no longer holds.