MapKAI

Day 5 of 5

Compare Evidence, Form Advice

How should leaders combine value, risk, strategy, and judgment?

Estimated time: 25 minutes

Today you will learn to
  • compare evidence on a fair basis
  • state the limits of a benchmark
  • present a recommendation that can be challenged
  • name the evidence that would change the decision
Start Lesson

Watch the lesson

Subtitles available where provided

Chinese

Unmasking the High-Performance Illusion

English

The 92 Percent Illusion

Listen to the podcast

The Driving School with the Highest Pass Rate

01

Opening story or business situation

Xiaolin was preparing to learn to drive.

She searched online for nearby driving schools. The first advertisement said: “92% first-exam pass rate, number one in the area for three consecutive years.”

Below it were photographs of students standing outside the test centre with newly issued driving licences. Many reviews said the school was demanding but very good at preparing learners for the test.

Xiaolin decided quickly. “I will choose this one. The highest pass rate should mean it is the most reliable.”

Her colleague Anna was learning to drive too. When she heard this, she asked, “Do you know how that pass rate is calculated?” Xiaolin shook her head. “People who pass divided by people who take the test—that is a pass rate, isn’t it?”

Anna opened the school’s website again. At the top it said 92%, but far below was a small line: “Data include only students who completed all courses and were scheduled by their instructor for a first examination.”

“What is wrong with that?” Xiaolin asked.

“Perhaps nothing,” Anna said. “But people who did not finish the course are not included. Neither are people the instructor judged unready and did not allow to register.”

Xiaolin phoned the school. The person who answered openly explained that it controlled exam registration very tightly: an instructor would help book the official test only after a student passed internal mock tests.

“Isn’t that good?” Xiaolin said. “At least they do not send unprepared students to an exam.”

“It has benefits,” Anna replied. “But it means 92% is not the success rate of everyone who enrolled. It is the rate among people selected by the instructor to sit their first test.”

The next day Xiaolin looked at two other schools. The second had only a 68% first-exam pass rate, but accepted many students who had learned elsewhere and failed multiple times, and let students decide when to take the test.

The third did not publish one combined rate. It listed rates separately for manual and automatic learners, first exams and retakes, regular and intensive courses.

Its largest number did not stand out. But Xiaolin could at least see the conditions that produced each number.

She then compared prices. The 92% school’s basic package looked inexpensive: twenty lessons and one mock test. Yet many students later needed ten to fifteen additional lessons. Weekend and evening lessons cost extra, and use of the instructor’s car on test day was not included.

The 68% school charged a little more per lesson, but did not add a weekend fee and was more flexible about rescheduling. If Xiaolin compared only advertised package prices, the first school was clearly cheaper. Once probable lessons, test-day car use, and extras were included, the difference was much smaller.

She also discovered that the schools did not serve identical students. The first mainly offered intensive courses: at least three lessons a week, usually aiming for a test within two or three months. That suited people with flexible time who could practise intensively.

Xiaolin had just started a new job and could attend only one weekend lesson a week. She learned reasonably fast but became nervous at complex intersections and needed longer to build confidence.

Anna said, “The school with the highest rate may genuinely be excellent, but its programme may not suit your time or way of learning.”

Xiaolin still booked a trial lesson with the first school. The instructor was skilful and precise, but the entire lesson was tightly paced. As soon as she completed one movement, she had to begin the next exercise.

Afterward, the instructor advised thirty intensive lessons, preferably on weekdays. “What if I can attend only once a week?” Xiaolin asked. “Progress will be slower,” he said. “And we cannot guarantee a fixed weekend slot. We have many learners and every instructor’s day is full.”

Only then did Xiaolin understand that the earlier high rate arose under specific conditions: instructors strictly controlled who could take an exam; students trained frequently; courses were compact; most students could fit the school’s schedule; and the statistic included only people who completed the course and formally took a first exam.

None of those conditions was wrong. They simply did not fully match Xiaolin’s situation.

In the end, she chose the third school. Its advertisement did not say “number one in the area,” and its instructor did not promise that she would pass after a certain number of lessons.

After the trial lesson, the instructor said clearly, “With your current situation, start with one lesson a week. Once your road judgment becomes steadier, decide whether to increase the frequency. We cannot accurately predict how many lessons you need yet.”

Six months later, Xiaolin took her first test and did not pass. At a busy junction she judged too slowly and the examiner intervened. She was naturally disappointed.

The instructor did not rush to book another test. Instead, he put the test report beside her training records from the previous months and identified the common difficulty she had in complex traffic.

Six weeks later, Xiaolin took the test again and passed.

The day she received her licence, she did not appear in an advertisement photograph for “first-exam passes.” But she did not think she had chosen the worse school.

She had chosen a school that could honestly explain its conditions, costs, and uncertainty—and fit her pace of learning.

Later another colleague asked, “Which driving school is best?” Xiaolin did not name one immediately. She first asked, “Have you driven before? Do you want manual or automatic? How many lessons can you take each week? Do you want to test quickly, or are you more concerned with building confidence gradually?”

The colleague laughed. “I only asked which one has the highest pass rate.”

Xiaolin replied, “Of course the rate matters. But you also need to know who took the test, who did not, how much they paid, and whether they were learners like you.”

02

The decision problem

03

Core concepts

Integration

compare evidence on a fair basis

Why it matters

Are the options being compared on the same basis?

Leadership judgment

state the limits of a benchmark

Why it matters

What evidence would make you reverse your recommendation?

Decision presentation

present a recommendation that can be challenged

Why it matters

Are the options being compared on the same basis?

04

Visual framework

01Begin with a business situation
02Reveal the financial logic
03Connect the idea to a practical framework
04Apply it through reflection and decision questions

Five-step knowledge chain | From “highest pass rate” to professional advice

Identify who will decide from the comparison → check which samples are absent → standardise comparison and calculation → explain differences after cost and risk → judge who the product suits.

01Decide whom the comparison serves

A fund is not automatically best outside an investor. Start with objective, holding period, investor currency, liquidity need, risk tolerance, available products, and real constraints. Then align asset class, fund category, share class, active or passive status, geography, and strategy. If the decision set differs, “which is best?” has no meaning.

02Before comparing results, inspect missing samples

Funds that closed, merged, changed name, stopped selling, or disappeared from a database can make surviving averages look better: survivorship bias. Check source, inclusion and exclusion rules, whether closed funds remain, backfill, start dates, and whether the period avoids difficult years. Repair the comparison foundation before using statistics.

03Put returns on one risk ruler

Higher return can come from equity or interest-rate risk, geography, style, factors, currency, concentration, or liquidity—not only skill. Match benchmarks to asset class, geography, style, factors, and investable universe; use fair periods and state local, investor-currency, or hedged returns. Only then can skill be separated from environment, style, and currency.

04Inspect the investor’s net result and its sources

Consider expense ratio, sales load, transaction cost, tax, liquidity cost, and turnover, as well as volatility, drawdown, concentration, and capacity. Small-scale success may not scale as assets grow. Industry structure, distribution, pension systems, regulation, and investor habits can explain cost and competition, but correlation is not causation.

05Translate analysis into bounded suitability advice

A recommendation does not announce a permanent champion. It says why a product fits this investor, what evidence supports that view, which risks remain, what assumptions are untested, what limitations apply, and what change triggers review. The recommendation is a match between the product’s risk, return, and limits and the investor’s goals, horizon, and capacity.

05

Practical example

Business example

Take AI-generated conclusions back to their evidence, benchmark, and conditions of use before deciding whether and how to act.

06

Common misunderstanding

07

Key takeaways

  • compare evidence on a fair basis
  • state the limits of a benchmark
  • present a recommendation that can be challenged
  • name the evidence that would change the decision

08

Knowledge check

How should leaders combine value, risk, strategy, and judgment?

09

Reference notes

  1. Otten, Roger & Martin Schweitzer (2002). “A Comparison between the European and the U.S. Mutual Fund Industry.” Managerial Finance, 28(1), 14–36. DOI: 10.1108/03074350210767627.

    Comparison of the European and US mutual-fund industries.

    Supports comparison discipline: industry structure, distribution, institutions, benchmarks, risk, fees, sample selection, and suitability. Its historical data illustrate a method and are not direct conclusions about today’s industry.