Introduction
Cells started as mere combinations of increasingly esoteric chemicals, assorting for only their own survival. Each one grabbing whatever it could, but as much of the universe goes, resources are not spread in a linear effort→reward curve, some resources are simply out of reach for lone cells, collaboration was how cells broke past that ceiling.
Now it gets interesting. A single bacterium understands nothing beyond what's touching it. But a typical bacterial colony spans many millimeters, this theoretically allows them to coordinate against a common invading cell/parasite, slide along to a region with higher concentration of resources, and get dormant/active in cases of suboptimal environment being detected. In expectation it’s a disadvantage not being in a bacterial colony.
In animal models, cells have distinct functions, but can’t survive on their own. They gave up control for efficiency. A muscle cell working with millions of others can move you across miles. Alone, it flakes out within mere minutes, unlike a bacterial colony that can regrow itself almost immediately. This is where the lines between just a “colony of cells” vs a “distinct new organism” blurs, I'm not the philosophical type to make claims here, Kurzgesagt’s What Are You? does a better job at this :).
Cell Signaling
Animal cells have built a very complex information highway to coordinate across the body. Mainly the nervous system, hormonal (systemic broadcast) and least known of all cellular signaling. The nervous system operates with a closed-loop top-down architecture where it decides what each signal means, while the cellular signaling is bottoms-up, cell to cell.
I find cellular signaling much more interesting for reasons you might be able to guess. All the other signaling methods depend on this substrate. Even neurons need neurotransmitters to coordinate. In my eyes, this makes it the mother pathway. The one all others built upon.
Cellular signaling achieves specificity through combinatorial logic, a signal’s effect heavily depends on the receiver. The same molecule can mean wildly different things to different cells. In fact a single molecule can travel as long as a meter to reach the target receptor making their effects really hard to isolate in nature.
Wildest of all? there are millions of different molecules in the human body, but we have only ever confidently mapped thousands if not hundreds. We just really don’t know how these cells do what they do! Every day we learn contradictory effects of these molecules.
Neurons did something stranger, they built an entire internal architecture around electrical impulses, then used cell signaling for handoff among each other. a recurring pattern, eh?
Neurons
For a biological substrate to build machinery around electrical impulses is absurd. This is why the brain uses 20% of the body’s energy. Crazy given it’s only 2% of the body mass, in contrast, at rest, muscles only use 20% of the body’s energy even though they account for 40% of the body’s mass.
Muscles gave up autonomy for a new job, to contract, enabling movement. Neurons gave up everything, a neuron’s entire purpose is to listen, integrate and pass messages to others. They have no function in isolation, they’re the cells that sacrificed the most for the collective. In doing that, they somehow gave rise to the thing reading this sentence. YOU!
Without the brain coordinating higher order objectives the body is as good as dead. Same as those bacteria colonies that help each other seek resources to flourish, the brain seeks the most optimal state, regardless of feelings of a singular cell. I find it funny that evolution woke up one day and decided building a computer to process information is the most optimal thing to do. Sounds familiar? We also did!
How do neurons work
Neurons form close-range communication channels called synapses. A synapse uses chemical signals to transmit information to other neurons. But this process is slow! for a chemical to reach a side of the brain from another, it would take a 1 year…
Neurons devised a beautiful solution to this, they only transmit these chemical signals across a cleft that’s smaller than 20 nano meters. The neuron only carries electrical impulse to the synaptic bridge. With this, it only takes 0.5 microseconds to travel to the next neuron and the total process of a synapse only takes ~ 1 millisecond.
Neurotransmission :)
Post-synaptic neuron takes in neurotransmitter → decides if it should open its ion gate and absorb Na+, it integrates from all the other synapses → if above threshold releases electrical impulse through axon reaching the terminal → the impulse activates the calcium channel and absorbs calcium → calcium arrival triggers vesicle fusion → neurotransmitter gets released → next post-synaptic receptor receives neurotransmitters → cycle continues to the other neurons
Contrary to how I used to think of neurons earlier in my life, they don’t only communicate through axon-dendritic synapses! There are receptors on most of the body of the neuron, But it heavily depends on whether the axon has high affinity towards a specific region/receptor of the neuron i.e., GABA receptors are very close to the Axon hillock region, thus GABAergic neurons also cluster around there.
Neurotransmitters
As I will describe later in the blog electrical synapses not only are infeasible in a biological substrate but are also fundamentally hard to control and gate, this necessitates a need for more biologically friendly operators. And what is better than molecules!
I like to look at neurotransmitters as what operators are to computers, letting it build much more complex functions. You can try building a computer with just the “NAND” operator as it’s universal, but that will require significantly larger serial depth to represent the other semantically easier operators.
A friend pointed out that I reduced the neurotransmitter’s effects and ignored how relevant timing and frequency are. This is fair. But for the analysis that follows, I'm treating modulatory state as fixed context. The "function" is a snapshot at a moment in time.
The most important transmitters are GABA and Glutamate, these are inhibitory and excitatory respectively, doing exactly what the name suggests, GABA tells the neuron to not have any action potential while glutamate tells it to act immediately.
The rest of the neurotransmitters build upon these by introducing notions of relevance, gain, dependence, … this allows a pre-synaptic neuron to for example tell a post-synaptic neuron to focus more on this specific synapse than the rest. e.g., learning is widely understood to require a spike in dopamine to assist in the encoding process.
We've confidently mapped maybe hundreds of neurotransmitters. The real number could be far higher, we also have recently discovered that the gray matter also produces neurotransmitters contrary to popular belief.
Why not electrical impulse all the way down, why chemical?
Pure electrical synapses exist, they’re called Gap junctions. They’re significantly faster than chemical synapses. So why are 99% of brain synapses chemical? Well it’s “inhibition”, how can you command a cell to not fire by electrically charging it? You can’t. That’s why they're mostly reserved for synchronization tasks like the heart, muscle. Furthermore:
Pros
Cons
Physically attached, this makes rewiring neurons incredibly hard.
Gap junctions only allow these cells to flow ions to the postsynaptic cell, no inhibition, thus can only compute monotonic functions. Monotonic functions aren't computationally complete. (GABA and certain other neurotransmitters allow neurons to inhibit activity)
Perhaps evolution hasn’t found the right way of doing electrical synapses. Or maybe the artificial neural networks we have built on Nvidia gpus are the electrical synapses that evolution led us to, and we are yet to integrate with them through BCI.
What other kinds of synapses would one imagine :P
photovoltaic ?
ultrasonic ?
These sound expensive for biological machinery, but I do wonder what life with just slightly different cost functions for these processes would have allowed!
Depth vs Width
It’s a rite of passage that everyone doing ML wants deeper networks. But a lot of my neuroscience friends describe the brain as wider than deeper. They also suggest maximum synaptic chain is often 20-30. I consider this nuts, a lot of ml research tells us how important depth is, in fact inference time compute (LLM-RL) is considered a virtual depth extension (discretized through the sampling process), we do all these tricks because we really can’t effectively train 10,000 layer networks with back-propagation (vanishing gradients win eventually)!
Diffusion models do something similar (although up to some extent), the integration of a differential equation can be done with however number of steps, but because of non-linear complexity of the data distribution we often have to take much smaller integration steps, recurring over the model to faithfully arrive at the target distribution. ResNets also in the infinite-depth limit converge to ODEs.
It’s totally plausible that biological neurons form looping circuits, this can allow information to go through a branching network and come back on the other end of the neuron as a more processed signal allowing it to capture more and more complex information over time.
There are several papers that suggest this conduct of neurons in specific regions
“Going in circles is the way forward” suggests recurrence lets the brain achieve flexible computational depth—the same circuit can run more iterations for harder problems, dynamically trading speed for accuracy. A 5-layer recurrent network running 10 iterations ≈ a 50-layer feedforward network. This resolves the apparent paradox: the brain is anatomically shallow but computationally deep.
“Cortico-basal ganglia-thalamo-cortical loop” , is a specific pattern of neuron connections between the cortex, the basal ganglia, the thalamus, and back to the cortex a multi-region recurrent circuit.
Analytics of neurons
Given we can at a basic level look at a neuron as a gate with parameters (akin to ordinary ML neurons), I’ll start with Level 1 and add more details while considering the intricate nature of neurons, and also keep track of the function space.
Sum → evaluate → Fire
output = threshold(Σ weights × inputs, Θ)
# Sum all the signals the neuron receives, if it's above Θ, neuron will fire
# Synapses: 10,000
# Possible Input patterns: 2^10,000 = 10^3,010
# Learnable Parameters: 10,001
# Linearly separable functions: ≈ 2^(n²)
= 2^(10,000²)
= 2^(10^8)
≈ 10^30,000,000
# Equivalent Depth: 1 layer
# Can't do XOR & temporal patterns
+ Timed
output(t) = threshold(∫ inputs(τ) × decay(t-τ) dτ)
# Sum all the signal based on how recent or old it was
# Integration window: ~20ms
# Temporal resolution: ~1ms
# Time bins: 20
# Effective input dimension: 10,000 x 20 = 200,000
# New linearly separable functions: ≈ 2^((200,000)²)
= 2^(4×10^10)
≈ 10^(10^10)
# we have linear temporal smoothing, basically a Conv1D (over temporal dimension)
# Effective depth: ~1 with a temporal smearing
# This now allows us to detect co-incidence of inputs
+ Distance to the axon hillock
output(t) = threshold(∫ Σ inputs(τ, loc) × decay(t-τ) × dist_from_hillock(loc) dτ, θ)
# Weight a signal based on how close it was to the axon hillock over the surface of the neuron (an axon can get close to any part of the neuron) and perform timed sum, Plus dendritic branches also are able to do non-linear transformation of the signal before reaching the hillock
# It's actually well studied that GABA receptors cluster much closer to the axon hillock, intuitively inhibition requires a strong veto power to stop the neuron from firing
# Number of dendrites: ~50 , ~200 synapses per branch
# Distance range: 0 - 500 μm
# Each branch: ~200 input weights + 1 threshold
# Soma: ~50 branch weights + 1 threshold
# Total: ~10,000 + 50 + 50 ≈ 10,100 params
# Dendrites can sum signals independently before passing down to the axon hillock, making the neuron act like a "nonlinear?" 2-layer network,
- Batched sum of signals on the dendrites
- Sum across the somatic signal and summed signal from the dendritic bus
|Functions| ≈ |LTF(200)|^50 · |LTF(50)|
≈ (2^(200²))^50 · 2^(50²)
= 2^(2·10⁶ + 2500)
≈ 2^(2·10⁶)
# Depth-2 or LTF^2 (linear threshold function) networks can represent all boolean functions, including XOR
+ Neurotransmitter / Modulator diversity
f = f(glutamate, GABA) # fast: fire / don't fire
θ = θ(dopamine, serotonin, ...) # slow: change HOW it fires
output(t) = f(inputs; θ(modulators))
# Modulators: ~10 types × ~10-100 levels each ≈ 10^10 - 10^20 states
# Each state selects ONE function from the dendritic repertoire
# This is conditional computation, not more functions
# Like a mixture-of-experts: modulatory state picks which "mode" to run
# Adds ~1 layer of gating (similar to attention in transformers)
# Effective depth so far: ~3-4 layers
+ Plasticity & Hormones
There is even more! the neurons can self reorganize based on external stimulus, they also can carry learned information with them throughout the transition allowing the neuron to traverse regions of function space unreachable by fixed weights
In summary
From my “naive” computation, a single biological neuron is able to represent a function of 4 layers deep, this more than ever tells me how little we understand of the brain, to list:
Layer 1: Dendritic branch nonlinearities (~50 parallel)
Layer 2: Somatic integration + threshold
Layer 3: Modulatory gating (context-dependent)
Layer 4: Temporal convolution (if you count it)
In fact this a more conservative version of the claim made by Beniaguev, Segev & London (2021). Where they needed 5-8 layers to be able to model a layer 5 cortical pyramidal cell (L5PC)
Conclusion
86 billion neurons. 100 trillion synapses running at 20 watts on a cushy wet substrate. We've mapped maybe 1% of how it works. The rest is still beyond our understanding. That's either terrifying or exciting depending on your disposition. I personally find this exciting as heck.
A question I’d like answered:
What would it take to actually train networks with neuron-like units? are MoEs early versions of these? Is the inductive bias worth the complexity?
Hit me up @rami_mmo if there’s something i missed or something interesting I can look into.
Thanks to my friends Ethan Smith, Joshua Belofsky, Tanishq Abraham, Stefan Bauman, Lucas Nestler, Kaleb & Francessco Sacco for reading the first draft of this post!
PS: this blog is building up to another blog I wrote for my friends about neurotransmitters and peptides <dm me on twitter for access>
