Why Can't People Solve These Simple Problems?

The cognitive reflection test was meant to measure one thing: the tendency for someone to stop and think before giving a quick answer. The simplicity of the test made it a widely used measure for critical thinking. But some clever experiments illustrate that the CRT is not so simple after all.
00:00 Introduction
00:19 The CRT problems
1:20 The structure of CRT problems
3:04 Some hints about what's going wrong
4:53 The structure of insight problems
6:27 Explanation Number One
7:57 Explanation Number Two
8:57 Explanation Number Three
I'm launching a membership site. If you want in, go here: forms.gle/JoMA1Xa5Nnnwm8VU9
Recommended reading: www.benjaminkeep.com/recommen...
Sign up to my free (infrequently published) newsletter here: www.benjaminkeep.com/
REFERENCES
This video is mostly based off of Patel, N., Baker, S. G., & Scherer, L. D. (2019). Evaluating the cognitive reflection test as a measure of intuition/reflection, numeracy, and insight problem solving, and the implications for understanding real-world judgments and beliefs. Journal of Experimental Psychology: General, 148(12), 2129-2153. doi.org/10.1037/xge0000592
This is the (classic) paper that introduced the CRT as a measure of reflection. Frederick, S. (2005). Cognitive reflection and decision making. The Journal of Economic Perspectives, 19, 25- 42. dx.doi.org/10.1257/ 089533005775196732. It also includes correlations between the math section of the SAT and the CRT (.46) vs the verbal section of the SAT and the CRT (.24).
On the CRT’s correlation to tests measuring numeracy being about .5, see Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R. (2012). Measuring Risk Literacy: The Berlin Numeracy Test. Judgment and Decision Making, 7(1), 25-47. doi.org/10.1017/S193029750000...
On the associations between reflection/analytical thinking and various beliefs, see Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015). Everyday Consequences of Analytic Thinking. Current Directions in Psychological Science, 24(6), 425-432. doi.org/10.1177/0963721415604610
This mentions some of the earlier results correlating CRT performance to rational thinking, but also extends the CRT to a seven-question test. The four added questions are probably even more reliant on numerical ability than the first three. Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & cognition, 39(7), 1275-1289.

Пікірлер: 58

  • @jamesarthurkimbell
    @jamesarthurkimbell17 күн бұрын

    It kills me to see kids taught "word problems" as this adversarial game, best approached by refusing to read and relying on tricks. Scan for numbers, scan for verbs, apply formula, ignore content and context... you know, the opposite of a good mindset.

  • @NandoPr1m3
    @NandoPr1m316 күн бұрын

    I came for the learning... but stayed for all the facial expressions, hilarious

  • @ApatheticPerson
    @ApatheticPerson17 күн бұрын

    I'm not a native speaker, but even if I were, I don't think I would have gotten the "marry" question right

  • @pyepye-io4vu

    @pyepye-io4vu

    8 күн бұрын

    It's a stupid trick question. It relies on using "married" both passively and actively, or the verb "to marry" both intransitively and transitively. The sentence is constructed deliberately so that both transitive usage and intransitive usage is possible.

  • @Saad-le4bm
    @Saad-le4bm5 күн бұрын

    while am not naive or fluent in English I did not get the word "married" in the insight problem, but I did recognize that the problem says "he never divorced" So by assuming that women can divorce in that town the answer will be (19 or all the women who married that man divorced him) which means that man did actually married 20 women without breaking the Law

  • @willguggn2
    @willguggn217 күн бұрын

    Some dude wanted to hire me on the spot because I snap-answered a question of this type correctly. It was for a job I didn't have a clue about, though. He asked me how much a cucumber would weigh after its water content went from a fresh 99% down to 98% after an unspecified time. Spoiler: It's half as much as the fresh one. Yuck ^^

  • @avimir8805

    @avimir8805

    15 күн бұрын

    Reread this problem 8 times and still haven't understood it. I suppose I would not get the job😂 Edit: after ten more attempts I have got it. 99/100 = 99% (by definition), and 49/50 = 98%. My brain hurts...

  • @willguggn2

    @willguggn2

    15 күн бұрын

    @@avimir8805 :D My thought process was more like "double the dry content == half the weight" because the dry weight of the cucumber doesn't magically grow by leaving it on the counter.

  • @jenHry-ng3pw

    @jenHry-ng3pw

    15 күн бұрын

    ​@@avimir8805it usually helps to put there real world numbers than trying to solve it abstractly. Let's start with 200g of cucumber. That is 198g of water and 2g dry part at 99%. If it loses water, the dry part starts the same. To get 98% water with 2g of dry part, you need 98g of water. Total 100g from the original 200g, or 1/2

  • @blobbowo

    @blobbowo

    14 күн бұрын

    99 apples, 1 orange this is 99% apples and 1% orange. to get to 98% apples by decreasing the number of apples.. 49 apples, 1 orange. this is 98% apples and 2% orange. Technically not half the number of apples, but close enough.

  • @user-oy8vu3xb2y
    @user-oy8vu3xb2y17 күн бұрын

    Thank you! This has been an interesting watch

  • @ameyakale3205
    @ameyakale32055 күн бұрын

    Hi Dr Keep can you please make a video on learning techniques which can be helpful for someone with dyslexia

  • @kianwawa4731
    @kianwawa473117 күн бұрын

    I blame my vocabulary for not knowing that married had that meaning.

  • @blackatbelladona3069
    @blackatbelladona306917 күн бұрын

    Sir Benjamin, I have watched your older videos, especially the part about taking notes, visualizing, and learning from a book. You have communicated the idea of using visualization to try to figure out how the pieces of information interact with one another-a visual perspective on how the information really is used to reflect the mental representation of an idea on the paper itself. How well would you recommend Justin Sung's mindmap strategy with it? considering some of the examples of visualization notes you have done before in your videos might be seen as level 2 note-taking strategies based on his recent note-taking videos, or is it really depending on the nature of the information you are consuming, or maybe he's idea is a bit more relevant if we are trying to get the individual pieces together on a holistic view? I really tried your idea of using such notes to reflect what is in your mind and using them to converse with myself to have a dialectic discussion with clarity, along with using them to make a mental schema about the whole subject that I'm learning about. Your ideas are really helpful, but I wanted to know how your own visualization strategy in note-taking and Justin Sung preference in a holistic view of his mindmap can be come to terms in some sort of middle ground. I'm very thankful that your videos exist, as they are my go-to source for getting ideas on how to learn generally.

  • @benjaminkeep

    @benjaminkeep

    12 күн бұрын

    Thanks for the kind words. Unfortunately, I just don't know enough about Justin's strategy's to say. Maybe there's another opportunity for a collaborative video with him. There's certainly no harm in trying some variations out and trying to judge what's working.

  • @blackatbelladona3069

    @blackatbelladona3069

    11 күн бұрын

    @@benjaminkeep Thanks for the advice Mr. Benjamin, and also I might seem a bit little pushy but can I ask you to take a look of Nuvak's concept map? I mean it's pretty much similar to Justin Sungs in a way he makes his mind map except it's a well documented one that is available to the public considering its use on an academic setting. With the principle of blooms taxonomy, hierarchical thinking and constructivism principles. Which you can connect an (concept)-- premise --(concept 2) in a non linear format. Thank you once more.

  • @kinru1259
    @kinru125917 күн бұрын

    For the first one I couldn't find an intuitive answer that worked. 10 cent ball- no, 1 cent ball- no. So i made a whole system of equations 💀

  • @SimGunther

    @SimGunther

    17 күн бұрын

    Same conclusion Philippos came to in the comments of this video :)

  • @jawadoumar

    @jawadoumar

    17 күн бұрын

    Lol

  • @avimir8805

    @avimir8805

    17 күн бұрын

    I use them all the times😂 For me it is much easier to actually get an answer, and then it is simple to check if the answer isn't appropriate.

  • @ChilliDuck

    @ChilliDuck

    16 күн бұрын

    I am still confused to how to answer that question when you dont know intuitively that its 5 cent lol

  • @SimGunther

    @SimGunther

    16 күн бұрын

    @@ChilliDuck take a dollar off 1.10 to get 10 cents. Now it's just a question of the average cost per item to get 5 cents per item. Finally, you add a dollar to the bat so the bat is now $1.05 and the ball is $0.05

  • @jenHry-ng3pw
    @jenHry-ng3pw15 күн бұрын

    Well one intuitive thing that came to my mind is that the lake must be really super large if it can take 2^48 plants (not even considering we start with multiple plants). Feels more like Texas-size sea than a lake...

  • @SuhelParwez-jp2lg
    @SuhelParwez-jp2lg13 күн бұрын

    I solve all of these question very easily because I am preparing for a comptative exam SSC CGL. India

  • @amandaashmead5770
    @amandaashmead577014 күн бұрын

    I teach AP English to juniors, and I do a lot of SAT prep, as well. The deciding-the-test-is-wrong thing is real, and pretty frustrating. I compare it to doing a jigsaw puzzle: once you decide a piece is missing, you can't find any pieces. The whole time you are looking for it, you are half convinced it is not there to be found, and that just ruins your ability to find it, or you give up too soon. In the same way, once kids are convinced the test is wrong, or incomplete, they can't find the right answer, even though it's well within their abilities. In English, they usually decide that there is more than one correct answer. Teachers in younger grades make this worse: a kid is arguing about an answer and the teacher lobs out "well, they are both right, in a way, but you need to pick the one that is the best answer. Stop overthinking", which is just BS designed to humor a stubborn student, or cover up the fact that the teacher doesn't know (and, to be fair, many teachers don't really get training in close reading). This creates a monster because now the kid has the idea that they are smarter than the test, and they want to prove that over and over again. They burn energy looking for the answer choice that could be true, but only smart kids would see it, ignoring the one that just is true. What is the line? "Man is not rational, he is rationalizing". It's also "so sharp he cuts himself". All this is even more reasons why transactional education sucks. When every piece of feedback is evaluated as an opening negotiation in a haggling session over a grade, there's not much thinking going on. Thinking gets in the way of the rhetoric.

  • @hughcaldwell1034

    @hughcaldwell1034

    12 күн бұрын

    I think part of the problem is that a lot of context is actually left out of questions, either by an assumed shared intuition, or because of unquestioned assumptions. This is practically inevitable, as being precise enough in every single question to remove all possible ambiguity and reinterpretation would be exhausting, if not downright impossible. But it leads to misunderstandings when the student and the teacher have different assumptions. I remember a question on a test in year 2 or 3 asked for the smallest three digit number with all the same digits. The desired answer was 111, but I remember putting either 0.00 or 1.11. I wasn't trying to outsmart the test, but rather wanted to get to the bottom of what "three digit number" meant. They could have said "whole number", but that still doesn't remove all ambiguity. Seven is then a valid answer, as it is represented as 111 in binary, or three, as it is III in Roman numerals. But including a specification of base would just confuse most kids that age. Another example is the question of how many knights one can place on a chess board such that none can take each other. Though it isn't specified in the wording, my mind interprets this as a combinatorial rather than a strategic question, and so considerations such as which colours the knights are don't matter, and it's more about how they move. But many people don't share this tacit assumption and so come up with a different answer. I used to think they were deliberately avoiding the point of the question, but the truth is they don't share my intuition of what the point is.

  • @amandaashmead5770

    @amandaashmead5770

    12 күн бұрын

    @@hughcaldwell1034 I think it starts like that, but then it turns into an identity issue: "I need to find the arguable ambiguity, because that's who i am". It's a huge impediment to learning, because while it's important to approach life critically, that is vastly different than being contrarian.

  • @hughcaldwell1034

    @hughcaldwell1034

    11 күн бұрын

    @@amandaashmead5770 I agree that contrarianism can get in the way of learning. I've just seen a lot of instances where someone is being interpreted as contrarian when they are, in fact, very confused. Some teachers (not saying you) seem to approach students with the expectation of being tricked, to the point where one teacher didn't believe my brother's friend was colour blind and thought he'd coloured the sky purple to be silly. But yeah, I don't work with kids at all, and I'm definitely not trying to tell you what your own job is like.

  • @nicholasfigueiredo3171
    @nicholasfigueiredo317114 күн бұрын

    The first question I got "10 cents" then I summed and thought "wait that's 1.20 and that's wrong". So I got 1.05 and 0.05. The other 2 I didn't even consider 100 minutes or 24 days wtf. They weren't even options I was like "I have 5 machines in 5 minutes they make 5 stuff so each machine takes 5 minutes to make stuff, so 100 machines make 100 stuff in 5 minutes". The second well it's double every day if it fills in 48 then 47. I didn't get the married question tho. Not sure if it is because the others were numbers or because english is not my first language but I didn't get that one.

  • @zackhzb
    @zackhzb17 күн бұрын

    Inspiring as always, and it seems we are updating with more regularity. ^_^

  • @Ash.Phoenix
    @Ash.Phoenix10 күн бұрын

    Hi, Dr Keep! I'm a long-time viewer and commenter on your channel. I have not had the chance to sit down and watch KZread for some time as I was going through the law school examination period. Today, I finally caught up on your last few videos... and gosh, they are absolutely fantastic! As always, your videos are a testament to scientific rigour and better learning. Your channel is a true respite from the rehashed poor-quality content polluting much of the online world. Not only are your videos deeply educational, they are highly engaging and well-made. Thank you again for sharing your insights and your knowledge. I truly feel lucky to be one of your subscribers. I eagerly await your website and being a subscriber to your membership. Keep being fantastic & wishing you well, Ash

  • @notgate2624
    @notgate262416 күн бұрын

    Great video! I like the theme of recognizing that there can be many competing explanations for something and each might have different levels of validity for different reasons. KZread can be full of people oversimplifying so it's refreshing. Unrelated, but how would you like people to contact you? I couldn't find an email on your KZread page or website. Answering viewer questions could be an opportunity for video ideas, if you ever feel the need.

  • @benjaminkeep

    @benjaminkeep

    12 күн бұрын

    Thanks! Re: contacting - although I do answer video comments in the first week or two of a video being published, I stopped posting contact information after being inundated with questions and comments. I will post my contact info again in the future, but for now I'm focused on building a course and community where it's easier to have meaningful conversations.

  • @notgate2624

    @notgate2624

    11 күн бұрын

    @@benjaminkeep Understandable! Looking forward to the course 🙂

  • @xXJ4FARGAMERXx
    @xXJ4FARGAMERXx13 күн бұрын

    5 machines need 5 minutes to make 5 widges 100 machines need (?) minutes to make 100 widgets. The workforce was boosted 20x, but the workload was also boosted 20x, so the time should stay the same (i.e. the time is 5 minutes)

  • @rashedulkabir6227

    @rashedulkabir6227

    12 күн бұрын

    But the amount of work has also grown.

  • @CentralTokyo
    @CentralTokyo17 күн бұрын

    I recognized these questions before from when they were trying to determine if I have ASD. I got all 3 right. Probably because it turns out I have ASD.

  • @harshvardhan4771

    @harshvardhan4771

    16 күн бұрын

    What's ASD? And how did answering these questions correctly, mean that you have ASD?

  • @Choco794

    @Choco794

    16 күн бұрын

    Autistic spectrum disorders, but I don't see how these questions help in diagnosis.

  • @CentralTokyo

    @CentralTokyo

    16 күн бұрын

    @@harshvardhan4771 Autism Spectrum Disorder. It doesn’t prove you have it, there are a number of things that they consider, but there was an academic study that indicated that people with ASD score higher on average on that particular test.

  • @CentralTokyo

    @CentralTokyo

    16 күн бұрын

    @@Choco794 I was getting diagnosed in Japan. They had a bunch of tests in English so that I could understand them clearly. It was one of the tests. It wasn’t the only one, but it was the shortest one.

  • @valentinrafael9201
    @valentinrafael920115 күн бұрын

    The lily one might require some more math knowledge ( exponential scaling is not intuitive for humans ), but getting the first 2 wrong is a bit ridiculous. The second "problem" is just a scaling thing, with the time interval staying constant, because both variables are scaled by the same number. First problem is just a substitution, and I guess you can get it wrong because you have to add the ball twice, otherwise, you are saying that 1.1+0.1 = 1.1

  • @SimGunther
    @SimGunther17 күн бұрын

    Am I weird for knowing all the answers super quickly without looking up the answer before the video started? Maybe that's a problem with word problems and their structure that's not intuitive for most. Then again, the nature of the video might have made the questions easier than they should've than if they were mixed in with regular questions? Tip for those running into these kinds of wacky questions during an interview: don't feel like you need to answer the question quickly; emphasize with their needs and draw a good picture of the problem as the problem is hashed out iteratively. Remember, employees and clients want humans that are emotionally smart with some actual intelligence, not laborers that can easily be replaced with calculators and server farms.

  • @willguggn2

    @willguggn2

    17 күн бұрын

    Have you studied anything STEM-related?

  • @notgate2624

    @notgate2624

    16 күн бұрын

    1) This video primes use to think twice about the problem and reject intuitive answers. The people in the studies probably weren't primed that way. 2) The type of people to stumble upon a channel like this might've seen questions like this already. I had already seen all 3, so maybe other people are in a similar boat. They're very popular questions nowadays.

  • @rashedulkabir6227
    @rashedulkabir622712 күн бұрын

    How do you know that one machine takes five minutes to make one widget?

  • @barkmark

    @barkmark

    11 күн бұрын

    Because it’s not possible for one machine to make one widget per minute

  • @blobbowo
    @blobbowo14 күн бұрын

    My answer is that people can solve these problems, but our brains are lazy and take an easy shortcut which doesn't make sense. The below answers are pretty much the exact same as in the video. *A bat and a ball cost $1.10, and the bat costs $1.00 more.* It's tempting to just say the bat is a dollar and the ball is ten cents, but that would mean the ball costs 90 cents more, not a dollar. So the answer is the bat is $1.05, and the ball is five cents. *5 machines make 5 widgets in 5 minutes, how long does it take 100 machines to make 100 widgets?* Here it's easy to just see a pattern of 5-5-5 and extend it to 100-100-100, so that it takes 1 minute per machine. This could be correct, but using normal world reasoning, the machines don't have to work in sequence, it could be that the machines each take 5 minutes to make one widget, so 100 machines all starting up at the same time would still each make 1 widget in 5 minutes, meaning it takes 5 minutes for 100 machines to make 100 widgets. *In a lake, a lilypad patch doubles in size every day. If it takes 48 days to cover the whole lake, how long does it take to cover half the lake?* Again, a possible temptation is just dividing 48 by 2, and getting 24. But the patch doubles each day. What is the difference between the day number which covers half the lake and the day number which covers all? It's only 1 day, because when the patch covers half the lake and doubles after a day, it'll cover the other half of the lake. Covering half the lake is one day before completion; covering the whole lake, so we can ignore trying to calculate anything else in between and say 48-1=47 and say that the lilypad patch takes 47 days to cover half the lake.

  • @sebastianM
    @sebastianM16 күн бұрын

    That's a nice plant there, Benjamino, what's its name?

  • @yuvrajsingh-gm6zk

    @yuvrajsingh-gm6zk

    13 күн бұрын

    money plant I guess!

  • @benjaminkeep

    @benjaminkeep

    12 күн бұрын

    It's some variety of pothos - not sure the exact one.

  • @dadsonworldwide3238
    @dadsonworldwide323817 күн бұрын

    It's in the format that rarly will even a machinest tool & die maker approach that way. We got these in elementary school in my era 70s & 80s it was always as bad as greece not being able to ask the right question about the right steam engine system to bring about the right predictable industrial revolution .more richard finneman less Socrates plz. Everything starts in Greece revisionist history curriculum and since 1945s Smith_mundt act in America it's really been etymological and math introduction that I find to be alien and more foreign. Textualism methodology objectivism get shit done like study a tree while also aware if the forest is on fire. 1900s structuralism movement wants us to wear earphones study a tree but play musical chairs of super position to get any answer other than that the forest is on fire lol It's a long list of how their They're and there has been treated a dualistic umbrella on par with our most taught ordering skill biological population species where it's OK on paper but can argue over slightest variations. Because it defines taxonomy like a city by Walls or ethnicity personal actors etc etc You need a marduk basisn mind for their divided in / individual soul agency or whatever eqaulibrium. Still gen x we knew our 1890s born great grandparents before logic was all messed up . How we triangulate thermodynamical systems was mostly corrupted until more recently as evidence once again narrows the opposition of the greater world on our xyz manmade time hierarchy knowledge of good evil equations dualistic brain primordial self soul agency

  • @dadsonworldwide3238

    @dadsonworldwide3238

    17 күн бұрын

    Time and place need and demands of knowledge and tools for applications to watch out for but they have no business in curriculum imo. You did & probably still do get forced to play by those rules in academic participatory lifestyles, but the problem is not in cognitively correct answers it's recognizing the fraud of the question lol

  • @ApatheticPerson
    @ApatheticPerson17 күн бұрын

    The third one was the easiest

  • @chronosbat
    @chronosbat16 күн бұрын

    1) 5 cents 2) 20 minutes 3) 47 days

  • @AsperaNonEs

    @AsperaNonEs

    15 күн бұрын

    How tf did you come up with 20 minutes?

  • @xXJ4FARGAMERXx
    @xXJ4FARGAMERXx13 күн бұрын

    Bat + ball = $1.10 Bat = ball + $1.00 ball = ? Bat + ball = $1.10 (ball + $1.00) + ball = $1.10 2ball = $1.10 - $1.00 2ball = $0.10 ball = $0.05 Let's check: If Bat = $1.00 and ball = $0.10 then Bat + ball = $1.10 $1.00 + $0.10 = $1.10 True Bat = ball + $1.00 $1.00 = $0.10 + $1.00 FALSE That's why the second solution doesn't work

  • @rashedulkabir6227

    @rashedulkabir6227

    12 күн бұрын

    Did you mean that the answer does not equal $0.05?