# Map and Territory (Rationality  ## Metadata - Author: [[Eliezer Yudkowsky]] - Full Title: Map and Territory (Rationality - Category: #books ## Highlights - When your method of learning about the world is biased, learning more may not help. Acquiring more data can even consistently worsen a biased prediction. ([Location 119](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=119)) - Tags: [[aqua]] - A cognitive bias is a systematic error in how we think, as opposed to a random error or one that’s merely caused by our ignorance. Whereas statistical bias skews a sample so that it less closely resembles a larger population, cognitive biases skew our thinking so that it less accurately tracks the truth (or less reliably serves our other goals). ([Location 123](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=123)) - Tags: [[aqua]] - Even when we correctly identify others’ biases, we exhibit a bias blind spot when it comes to our own flaws.4 Failing to detect any “biased-feeling thoughts” when we introspect, we draw the conclusion that we must just be less biased than everyone else.5 ([Location 145](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=145)) - Tags: [[aqua]] - To the extent this volume does its job, its approach can be compared to the one described in Serfas (2010), who notes that “years of financially related work experience” didn’t affect people’s susceptibility to the sunk cost bias, whereas “the number of accounting courses attended” did help. ([Location 151](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=151)) - Tags: [[aqua]] - distinguish between experience and expertise, with expertise meaning “the development of a schematic principle that involves conceptual understanding of the problem,” which in turn enables the decision maker to recognize particular biases. ([Location 154](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=154)) - Tags: [[aqua]] - Humans aren’t rational; but, as behavioral economist Dan Ariely notes, we’re predictably irrational. There are patterns to how we screw up. And there are patterns to how we behave when we don’t screw up. Both admit of fuller understanding, and with it, the hope of leaning on that understanding to build a better future for ourselves. ([Location 179](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=179)) - Tags: [[aqua]] - The usual finding is that exponential increases in scope create linear increases in willingness-to-pay—perhaps corresponding to the linear time for our eyes to glaze over the zeroes; this small amount of affect is added, not multiplied, with the prototype affect. This hypothesis is known as “valuation by prototype.” ([Location 214](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=214)) - Tags: [[aqua]] - The moral: If you want to be an effective altruist, you have to think it through with the part of your brain that processes those unexciting inky zeroes on paper, not just the part that gets real worked up about that poor struggling oil-soaked bird. ([Location 229](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=229)) - Tags: [[aqua]] - The availability heuristic is judging the frequency or probability of an event by the ease with which examples of the event come to mind. ([Location 290](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=290)) - Tags: [[aqua]] - In real life, you’re unlikely to ever meet Bill Gates. But thanks to selective reporting by the media, you may be tempted to compare your life success to his—and suffer hedonic penalties accordingly. The objective frequency of Bill Gates is 0.00000000015, but you hear about him much more often. Conversely, 19% of the planet lives on less than $1/day, and I doubt that one fifth of the blog posts you read are written by them. ([Location 305](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=305)) - Tags: [[aqua]] - Using availability seems to give rise to an absurdity bias; events that have never happened are not recalled, and hence deemed to have probability zero. When no flooding has recently occurred (and yet the probabilities are still fairly calculable), people refuse to buy flood insurance even when it is heavily subsidized and priced far below an actuarially fair value. ([Location 309](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=309)) - Tags: [[aqua]] - “Cognitive biases” are those obstacles to truth which are produced, not by the cost of information, nor by limited computing power, but by the shape of our own mental machinery. ([Location 330](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=330)) - Tags: [[aqua]] - In cognitive science, “biases” are distinguished from errors that arise from cognitive content, such as learned false beliefs. These we call “mistakes” rather than “biases,” and they are much easier to correct, once we’ve noticed them for ourselves. (Though the source of the mistake, or the source of the source of the mistake, may ultimately be some bias.) ([Location 339](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=339)) - Tags: [[aqua]] - A bias is an obstacle to our goal of obtaining truth, and thus in our way. ([Location 354](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=354)) - Tags: [[aqua]] - Which is to say: Adding detail can make a scenario sound more plausible, even though the event necessarily becomes less probable. ([Location 370](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=370)) - Tags: [[aqua]] - they would need to be shocked by the word “and.” Moreover, they would need to add absurdities—where the absurdity is the log probability, so you can add it—rather than averaging them. They would need to think, “Reagan might or might not cut support to local governments (1 bit), but it seems very unlikely that he will support unwed mothers (4 bits). Total absurdity: 5 bits.” Or maybe, “Reagan won’t support unwed mothers. One strike and it’s out. The other proposition just makes it even worse.” ([Location 395](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=395)) - Tags: [[aqua]] - They would need to feel a stronger emotional impact from Occam’s Razor—feel every added detail as a burden, even a single extra roll of the dice. ([Location 409](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=409)) - Tags: [[aqua]] - What Do I Mean By “Rationality”? I mean two things: 1. Epistemic rationality: systematically improving the accuracy of your beliefs. 2. Instrumental rationality: systematically achieving your values. ([Location 436](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=436)) - Tags: [[aqua]] - Instrumental rationality, on the other hand, is about steering reality—sending the future where you want it to go. It’s the art of choosing actions that lead to outcomes ranked higher in your preferences. I sometimes call this “winning.” ([Location 445](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=445)) - Tags: [[aqua]] - Similarly, “Rational agents make decisions that maximize the probabilistic expectation of a coherent utility function” is the kind of thought that depends on a concept of (instrumental) rationality, whereas “It’s rational to eat vegetables” can probably be replaced with “It’s useful to eat vegetables” or “It’s in your interest to eat vegetables.” We need a concept like “rational” in order to note general facts about those ways of thinking that systematically produce truth or value—and the systematic ways in which we fall short of those standards. ([Location 456](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=456)) - Tags: [[aqua]] - this phenomenon is known as the “planning fallacy.” The planning fallacy is that people think they can plan, ha ha. A clue to the underlying problem with the planning algorithm was uncovered by Newby-Clark et al., who found that Asking subjects for their predictions based on realistic “best guess” scenarios; and Asking subjects for their hoped-for “best case” scenarios . . . . . . produced indistinguishable results.5 When people are asked for a “realistic” scenario, they envision everything going exactly as planned, with no unexpected delays or unforeseen catastrophes—the same vision as their “best case.” Reality, it turns out, usually delivers results somewhat worse than the “worst case.” ([Location 550](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=550)) - Tags: [[aqua]] - The outside view is when you deliberately avoid thinking about the special, unique features of this project, and just ask how long it took to finish broadly similar projects in the past. This is counterintuitive, since the inside view has so much more detail—there’s a temptation to think that a carefully tailored prediction, taking into account all available data, will give better results. But experiment has shown that the more detailed subjects’ visualization, the more optimistic (and less accurate) they become. ([Location 566](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=566)) - Tags: [[aqua]] - So there is a fairly reliable way to fix the planning fallacy, if you’re doing something broadly similar to a reference class of previous projects. Just ask how long similar projects have taken in the past, without considering any of the special properties of this project. Better yet, ask an experienced outsider how long similar projects have taken. ([Location 579](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=579)) - Tags: [[aqua]] - To seek truth merely for its instrumental value may seem impure—should we not desire the truth for its own sake?—but such investigations are extremely important because they create an outside criterion of verification: if your airplane drops out of the sky, or if you get to the store and find no chocolate milk, its a hint that you did something wrong. You get back feedback on which modes of thinking work, and which dont. ([Location 618](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=618)) - Tags: [[aqua]] - Curiosity, as a human emotion, has been around since long before the ancient Greeks. But what set humanity firmly on the path of Science was noticing that certain modes of thinking uncovered beliefs that let us manipulate the world —truth as an instrument. As far as sheer curiosity goes, spinning campfire tales of gods and heroes satisfied that desire just as well, and no one realized that anything was wrong with that. ([Location 628](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=628)) - Tags: [[aqua]] - For my part, I label an emotion as “not rational” if it rests on mistaken beliefs, or rather, on mistake-producing epistemic conduct. “If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm.” Conversely, an emotion that is evoked by correct beliefs or truth-conducive thinking is a “rational emotion”; and this has the advantage of letting us regard calm as an emotional state, rather than a privileged default. ([Location 649](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=649)) - Tags: [[aqua]] - Becoming more rational—arriving at better estimates of how-the-world-is—can diminish feelings or intensify them . Sometimes we run away from strong feelings by denying the facts, by flinching away from the view of the world that gave rise to the powerful emotion. If so, then as you study the skills of rationality and train yourself not to deny facts, your feelings will become stronger. ([Location 669](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=669)) - Tags: [[aqua]] - Ever since I adopted the rule of “That which can be destroyed by the truth should be,” I’ve also come to realize “That which the truth nourishes should thrive.” When something good happens, I am happy, and there is no confusion in my mind about whether it is rational for me to be happy. ([Location 679](https://readwise.io/to_kindle?action=open&asin=B07LDF7J3Q&location=679)) - Tags: [[aqua]]