[PROLOGUE – EVERYBODY WANTS A ROCK]
[PART I – THERMOSTAT]
[PART II – MOTIVATION]
[PART III – PERSONALITY AND INDIVIDUAL DIFFERENCES]
[PART IV – LEARNING]
[PART V – DEPRESSION AND OTHER DIAGNOSES]
[PART VI – CONFLICT AND OSCILLATION]
A crucial mood, the “No Really, Seriously, What Is Going On?” mood, the mood of true curiosity. I don’t know if they’ve found the truth, but I can tell they are earnestly looking.
— Emmett Shear
Freeze and Cower
Consider: You are in a terrible situation and your fear governor expects you to die in a horrifying way, -1000 points. There is nothing you can do, so you take the action “freeze and cower”. But instead, you survive and it’s not nearly as bad as expected, only -100 points.
Sounds great, right? Wrong! The sad truth is that as far as the fear governor can tell, “freeze and cower” has just given you +900 points! It is the best action you have ever taken, possibly the best action of all time. It doesn’t matter that the prediction was wrong, that things went badly but not as badly as the worst possible outcome, that’s not how the system works.
Your fear governor has learned that “freeze and cower” is the best action it can take. So whenever you feel fear in the future, it votes that you should freeze and cower, and you usually do. Since (in the modern world) most things you are afraid of do eventually go away if you freeze and cower for long enough, your fear governor continues to believe that “freeze and cower” is a good action, and keeps voting “freeze and cower” up and down the ballot.
This sounds a lot like PTSD. Take for example this description from Scott Alexander’s review of Van der Kolk’s book, The Body Keeps The Score:
Van der Kolk thinks that traumas are much more likely to cause PTSD when the victim is somehow unable to respond to them. Enemy soldiers shooting at you and you are running away = less likelihood of trauma. Enemy soldiers shooting at you and you are hiding motionless behind a tree = more likelihood of trauma. Speculatively, your body feels like its going into trauma mode hasn’t gotten you to take the right actions, and so the trauma mode cannot end.
In the real world PTSD is often more complex, but that added complexity usually makes things worse, not better. For example, a civilian is mostly not afraid of loud noises like fireworks, and the fear governor does not even pay attention. But if you go to war and you learn that loud noises are dangerous, they become something that your fear governor considers its rightful business. You also learn that “freeze and cower” is a great action that always seems to work. So you return to civilian life ready to freeze and cower not only when you encounter the things that have always scared you, but also in response to things that you previously would not have found frightening.
With enough exposure, this response might go away. But not all exposure will be equal. If we play a loud sound, and you freeze and cower, and nothing bad happens, this actually reinforces the response. It’s just more evidence that “freeze and cower” is a good action that is reliably followed by a safe outcome.
Instead, you need to play a loud noise, but quiet enough that you can force yourself to not freeze and cower. Over time, this should teach you that other actions, like “stand there calmly”, are just as good. But it may take a long time for these estimates to balance out. From our example above, “freeze and cower” is valued at +900 safety points. And it only got there because your fear governor expected to die, and then didn’t. So a program of exposure where you get exposed to a little danger at a time will take a long time to catch up.
The really weird implication here is that a faster form of exposure therapy might be to get you to freeze and cower, and then actually hurt you! This would hopefully teach your fear and pain governors that “freeze and cower” has negative value, and it should get you to that conclusion faster. That said, the ethics of actually harming your patients seems questionable.
We can also think a little bit about individual differences. Some people probably have faster or slower learning rates, and this makes them more or less easy to traumatize from a single event.
Remember this equation from earlier? There’s that term α, which is the learning rate. If the learning rate is high, then your opinion of an action will change a lot every time you try that action. So a person with a high learning rate is more likely to become traumatized from a single experience. Just one very good or bad experience with an action would hugely change their estimate of how reasonable it is to take that action. But someone with a low learning rate will only ever change their estimate of an action by a small amount, so would be less likely to get traumatized unless the same outcome happened to them over and over.
Some people have more or less previous experience with the frightening stimuli. If you have a lot of previous experiences with loud noises, then artillery explosions won’t be so unusual, and you probably won’t learn to treat all loud noises as dangerous. But if you grew up in the silence of rural North Dakota, the experience of an artillery barrage might teach a lesson that gets passed on to all loud noises, for lack of experience.
Never Drink Alone
Consider: Drugs often interfere with specific error signals.
Take alcohol, which is especially famous for the way it reduces feelings of shame. When you get drunk, you feel much less shame than you did before, and you might end up doing some very shameless things. It also seems to suppress emotions like fear and thirst, and maybe increases emotions like hunger and exhaustion, which is why you do something dangerous, eat a whole basket of fries, and then pass out.
Imagine a guy named Chris who has just turned 21. He goes out drinking for the first time and he is feeling pretty good. He isn’t feeling ashamed of anything at all, so the alcohol can’t reduce his feelings of shame, because shame is at zero.
A few weeks later, someone finds out a dark secret about Chris’s past and confronts him with it. Now Chris is feeling deeply ashamed. He has plans to go out drinking with friends though, so he still goes to join them. When he lifts the bottle of beer to his lips, he has a very large shame error signal. And when he drinks the alcohol, that shame error is reduced. By his fifth beer, he doesn’t feel ashamed at all.
What does his shame governor learn from this experience? It learns that drinking beer is a great way to reduce its error. This is great news for the shame governor, because there are not many ways to quickly reduce shame. It resolves to vote for Chris to drink beer in the future whenever he feels ashamed.
We see similar patterns across the world of drugs. Caffeine and amphetamines reduce fatigue and hunger. Opiates reduce pain. And MDMA seems to interact somehow with the social emotions — maybe it reduces shame while increasing other drives, ones that make people more social and touchy-feely.
With us so far? Let’s try a more complicated example.
Let’s say you are hanging around one day, feeling pretty ok. All your errors are close to zero, so you’re mostly in alignment. You’re a little chilly, but only a little: your cold governor has an error of 5 points.
You decide to take some amphetamines just for fun, like one does. You’ve never taken amphetamines before, but you feel fine and you have nothing better to do.
Let’s say that amphetamines have only one effect: they immediately change any error you have to zero. This lasts until the amphetamines wear off, at which point the error goes back to whatever it was before. So when you take the amphetamines, your cold error goes to zero for a while, and you no longer feel chilly.
Your cold governor picks up on this. Its job is to generate and keep track of that error, and keep track of ways to reduce that error towards zero. So your cold governor learns that “take amphetamines” reduces its error by about 5 points.
The amphetamines don’t actually change your body temperature (at least not in this hypothetical). They just adjust the error signal and temporarily turn it to zero. But your cold governor isn’t sensitive to this distinction. It’s wired to correct an error term. Anything that corrects the error is still a correction, and is perfect in your governor’s eyes.
It will remember this information, but this factoid doesn’t really matter because the correction is so small. A fairly normal option, like “put on a sweater”, is already valued at 50 points. Even “just stand there and shiver” is valued at 10 points! So your cold governor will remember that amphetamines correct its error a little bit, but it will never vote for amphetamines, since it will always have better options.
Later, you are feeling kind of tired and lonely. Let’s say your tired error is 50 and your loneliness error is 100. You would normally go to sleep when you’re this tired, but you want to go out to see friends. But all your friends happen to be out of town this weekend, and no one is responding to texts. So you’re doubly uncomfortable. You decide to try some amphetamines again, just for laughs.
What happens? Well, your tired error and your loneliness error both go to zero. Your tired and lonely governors dutifully mark this down in their ledgers: amphetamines have a +50 correction for tiredness and +100 correction against loneliness.
From the governors’ point of view, this is pretty good. Their only job is to reduce their error signals — they have no ability or reason to care about anything else. So they consider this drug just as good as or even better than their other options.
Now you are in trouble. Your governors have learned that amphetamines are a really good solution to being tired, and one of the best solutions ever to being lonely. In the future, your tired governor will often vote for you to take amphetamines instead of sleeping, and your loneliness governor will reliably vote for you to take amphetamines instead of anything else. From the governor’s point of view, why leave the house and socialize, which has a small chance of making you slightly less lonely, when you could take amphetamines and immediately feel not lonely at all.
Things will only get worse from here. As these two governors vote for you to take amphetamines, any other governors who happen to be out of alignment will learn the same bizarre lesson.
You get a little lonely, and so your loneliness governor votes for you to take some amphetamines. This time, your hunger governor also happens to be a bit out of alignment; you have an error of 20 for hunger.
When the loneliness governor makes you take amphetamines, your hunger error also goes to zero, and your hunger governor picks up on a new idea: amphetamines reduce hunger by about 20 points! That’s almost as good as a quesadilla. Now the hunger governor will often vote for amphetamines, since as far as it can tell, amphetamines are just as good as food. In fact, it will soon learn that amphetamines are better than food, since they correct its errors so reliably.
It’s probably clear why this is very dangerous. If you take a drug that reduces all error signals, then any governor with a large error signal when you take the drug (you’re hungry, you’re angry, etc.) learns to vote in favor of that drug, because it learns that the drug reduces its error signal. Because the drug reduces every error signal.
If you take the drug over and over, you dig yourself deeper every time. Different systems will be out of alignment each time you take the drug, so more governors will see that the drug corrects their error and will learn to vote for it. Eventually, all governors are voting for the drug — hunger is voting for the drug instead of eating, horny is voting for the drug instead of sex, and so on. And the drug feels pleasurable to take — it causes happiness when it corrects the errors, because correcting errors always causes happiness.
When the drug wears off, the error signal goes right back to where it was before. So you’re still tired, hungry, or whatever else. All you did was waste some hours.
You are now very truly addicted. Or at least, that’s what it looks like to us. If you follow this model to its logical conclusion, you get something that looks very much like addiction.
This fits with the observation that some people can take supposedly addictive drugs without getting addicted. In addition to things like genetic differences, if you don’t have large errors when you take your drugs, you won’t get addicted. If you’re already chair of the psychiatry department at Columbia, there’s not much more that drugs can do for you.
This may have something to do with why drug withdrawal feels like abstract “suffering” rather than bad in any particular way. If you have a serious addiction, then when the drug is taken away, every sense you have for finding something wrong with the world will be firing all at once, so in withdrawl you feel not only hungry and thirsty but also tired/pain/cold/hot/scared/angry/jealous/…
Your governors are always trying to estimate the value of different actions (value in the sense of what effect it will have on the signal they care about). But it’s impossible to accurately estimate the value of an action, like taking a drug, that temporarily sends all errors to zero. These drugs don’t have a true value — they interfere with the signal directly. Since they are so tangential to what your governors are trying to do, they really mess with your normal process of learning.
You are lonely — and smoking a cigarette will extinguish that loneliness for about 30 minutes. When it’s over, you will be just as lonely as before. But if you have no better options, the forces inside you that seek connection will still demand you smoke that cigarette, even against your better judgment. A promising but never-fulfilling solution to a basic desire is the core mechanism of addiction.
Most drugs don’t appear to interfere with every error signal, they only interfere with a few. If alcohol only interferes with your shame and your fear governors, then you would only get addicted to alcohol if you drink when you are ashamed and/or afraid. Your hunger or dominance governors won’t be tricked by alcohol, because it doesn’t mess with their signals.
The model predicts that you’re more likely to get addicted to a drug if you take it when you are out of equilibrium in some way. If you take a drug when all your error signals are near zero, then the drug won’t correct those errors by very much, it can’t. Your governors will learn that taking the drug only corrects their errors a bit, and will probably ignore it. This means that social drinking and social smoking might in fact be safer than other forms of smoking and drinking — the folk wisdom that says “never drink alone” may just be onto something.
This also implies some things about what it means to have an addictive personality, what kinds of people are more likely to get addicted to a substance. People with a high learning rate are more likely to get addicted, because their governors will take a single experience with a drug more seriously.
If two people take cocaine when their fatigue error is at 100 points, the drug sends both of their fatigue errors to zero. The person with the high learning rate estimates that the fatigue correction of cocaine is 80 points, and the person with the low learning rate estimates that the fatigue correction of cocaine is 20 points. Obviously one of these people will be more likely to take cocaine in the future, and will be more prone to a serious runaway addiction.
There are some kinds of addiction that don’t involve an external chemical at all. These addictions must have some other origin, since they don’t come about by directly messing with error signals in the brain. We think these can be explained in a couple of different ways.
First of all, the word “addiction” is used very broadly, to mean something like “a habit I do more than I’m supposed to” or “something I do all the time but I don’t endorse”. Sometimes addiction is just a word for something that’s not socially normal or something that is more extreme than normal.
When someone describes themselves as a sex addict, they are often just someone with an unusually high sex drive. Or when someone says they are addicted to cupcakes, they really just mean that their perfectly natural drive to eat sugar is in conflict with their drives to avoid shame and social stigma. Or when someone appears addicted to video games, but that person is in fact feeding their drive for domination in the only way that won’t get them into serious trouble.
This form of addiction can still be pathological. It’s possible that your drive for sex or for cupcakes is so strong that it prevents you from taking care of your other responsibilities, so strong that it comes to ruin your life. These drives are “natural” in the sense that you were born that way, but that doesn’t make them any less destructive. And it could still be pathological in the sense that it’s the result of illness or damage, like if your drive became unusually strong as a result of some kind of traumatic brain injury. But ultimately this “addiction” is one of your drives performing its functions in the normal way, just in a way that you don’t endorse.
One sign that these cases are extreme forms of normal behavior is that these addictions are all linked to one of the basic drives, like food or sex. Even some behaviors that are viewed as purely compulsive, like pica, may actually be the result of a drive that is not commonly recognized, like the drive to eat enough iron. The people on edible dirt etsy must be spending all that money on dirt for some reason. In comparison, we know that evolution did not give us a natural drive to do lines of cocaine. That’s an unnatural desire, driven by some kind of basic malfunction in the systems that are in charge of learning and motivation.
All that said, there are some kinds of addiction that aren’t obviously linked to any drive, and aren’t the result of drugs. The clearest example is gambling.
This kind of addiction might be explained by simple facts of learning. Gambling seems to be the most addictive when the rewards are really variable and the gambles happen on very short time loops. Or they might be most easily explained by something like time discounting — you can put off the shame of failure by doubling down, at least until you can’t anymore.
But the real lesson here might just be that there could be more than one kind of addiction. The idea of looking for a single explanation for every kind of compulsive behavior is exactly the kind of superficial word-chasing that we should try to avoid.
Bloody Knuckles
Consider: If cutting yourself gives -10 points, but stopping cutting gives +15 points, then your pain governor will consistently vote for you to cut yourself, so it can then vote for you to stop cutting yourself, netting it a cool 5 points. It will do this whenever you are idle enough that there’s nothing more important going on, when nothing else can beat out its votes.
We think of self-harm as pathological, but there may be some part of it that is very normal. There’s a famous study from 2015, where the authors put people alone in a room for 15 minutes, hooked up to a shock generator that they could shock themselves with as much as they want. They had previously felt the shock and said they found it painful, but 67% of men and 25% of women still chose to shock themselves at least once more, instead of sitting in silence for 15 minutes. Men gave themselves more shocks on average (an average of 1.47 shocks) than women (an average of 1.00 shocks), not counting one man who shocked himself 190 times.
This might follow the same logic as other self-harm, just on a smaller scale. If the shock is more startling than truly painful (-1 point), and the relief of not being shocked any more is a bit pleasant (+5 points) then it’s easy enough to end up in a situation where on net you enjoy shocking yourself.
This issue seems important for making cybernetic systems work. You need these numbers to be in the right ratio. Otherwise, you get caught in a loop — stick your head underwater so you get an error, then take it out so that the error is reduced. But self-harm does sometimes happen, meaning that whatever steps our psychology has taken to resolve this issue don’t work perfectly, or don’t work in every situation.
One way to save on energy, and keep an animal from getting too distracted, would be for governors to become dormant when they don’t have a very big error. After all, if they don’t have a big error, there’s not much they need to pay attention to. This seems to fit that trope where someone does something they thought they wanted, then immediately regrets the consequences. Why didn’t they see that coming? Because the governor that disapproves couldn’t get through. It was asleep.
If this is how we’re designed, then before self-harm the pain governor would be partially offline and wouldn’t be able to vote against “cut yourself”. Then it would be fully online for “stop cutting yourself”.
But this doesn’t work for a few reasons. First of all, governors do seem to get some votes even when their errors are at zero, which is obvious from how often they vote against stuff. If you’re nice and warm inside and you don’t want to slog out into the cold to scrape off your car, that’s your cold governor voting against it, even though it doesn’t have any error at the moment.
Pain seems like a governor that should always be at least somewhat awake. Pain is probably strong even if it’s dormant, because its main job is to prevent injury. It needs to be able to vote against dangers to life and limb, even when you’re not currently in any pain. The hunger governor can turn off when you’re not hungry, but the pain governor can’t turn off when you’re not in pain.
More importantly, even if the pain governor were off, then what governor would be voting for you to cut yourself in the first place? Maybe there’s a way to confuse the pain governor to vote for “hurt yourself then stop hurting yourself” but if so, it would need to be online to vote for it. It wouldn’t do that if it were dormant.
Maybe self-harm only works over very short time horizons. Pleasure and pain in the future should be both discounted, they should be counted as less than 100% of their values because the future is uncertain. Even if you expect something good or bad to happen, there’s always some chance that it won’t come through.
If negative events in 1 minute are weighted so that they’re less than positive events in 2 minutes, that could maybe lead to self-harm. Getting cut is 60% of -10 points, stopping getting cut is 80% of +8 points, suddenly the pain governor is happy to vote for one so it can get to the other.
A more straightforward explanation is that some other governor is voting for you to cut yourself, and it’s much more powerful than the pain governor, so it can overrule it. This is most obvious in cases like ritual self-harm, games like chicken or bloody knuckles. In this case, failing to go through with the painful or embarrassing experience would lose the confidence of your community, you would be ashamed, it would show that you were a loser or a coward. These are high enough stakes that other drives are able to out-vote the pain governor.
There might be many governors that could vote for this. For example, there might be an actual drive to harm others, that motivates things like sadism and serial killing. If you have a drive to hurt something, but the only thing you can hurt without consequence is yourself, then this drive might eventually overpower your drive for pain, and you would hurt yourself as a compromise.
Or, maybe the systems set in place to prevent self-harm do work perfectly fine, and so self-harm only happens when something has happened and those systems aren’t working properly, more like depression. If a malfunction makes it so that bad experiences are estimated as less bad than usual, that would make you pretty reckless — and it could mean that you’re willing to hurt yourself to get the benefits of not hurting yourself later.
[Next: INTERLUDE]
SLIME MOLD TIME MOLD 