Practices to reduce overconfidence ── Calibration training and the pre-mortem

Now comes the most practical part of this series. In Parts 1 to 7, we traced the structure of overconfidence, the signs to observe, the mechanisms that hinder learning, and the ethical core of moderation. This time, we will discuss these theories Specific techniques to reduce overconfidence It will be turned into a tool box.calibration training ── Lichtenstein et al. established a method to increase the accuracy of probability judgments in the 1980s, Brier (1950) quantified the accuracy of predictions, and Mellers et al.'s Good Judgment Project (2015) demonstrated the effectiveness of training.Pre-mortem --A technique presented by Gary Klein (2007) to write down failure scenarios before making a decision.Reference Class Forecasting -- Kahneman and Lovallo (1993)'s method of anchoring judgments to the statistics of similar cases.Devil's Advocate、10/10/10 rules ── Institutional and temporal tools for resisting overconfidence. This is the most "tool box"-like episode in this series, where we organize techniques that can be used in the field starting tomorrow.

01Why "Practice" needs to be a separate chapter

Why is there a need for a separate "Practice" chapter after Part 7 deals with moderation? The reason is directly related to Kahneman's warning cited in Part 1.Knowing about overconfidence and reducing overconfidence are two different things.。理論的理解は実践を支えるが、置き換えはしない。具体的な技法 ── 紙とペンと時間さえあれば使える工具 ── を持たない限り、知識は判断の現場には届きません。

This article deals with what half a century of judgment research has shown. です。重要なのは、これらの技法のすべてが「過信そのものを直接攻撃する」のではなく、「判断のプロセスに過信が入り込みにくい構造を作る」ものだということです。第 3 回で見た防衛機制の議論と整合します ── 無意識を直接見るのではなく、無意識の作動条件を変える。

"The effectiveness of an intervention to reduce overconfidence is proportional to the extent to which it forces reflection on judgment. Vague attempts to be careful are ineffective. Overconfidence is measurably reduced only when explicit techniques - numbers, notes, scenarios, third-person perspectives - are introduced." - J. Edward Russo & Paul J. H. Schoemaker, Sloan Management Review (1992) Purpose

The first half (sections 2 to 4) covers calibration theory and training, the middle (sections 5 to 8) explains four different tools (Pre-mortem, Reference Class Forecasting, Devil's Advocate, and the 10/10/10 rule), and the second half (sections 9 to 10) moves on to the field practice of materials inspection. This episode is set up as the episode with the most "take home" character in this series.

02What is calibration? Research by Lichtenstein et al.

The starting point for practicing reducing overconfidence iscalibration の概念です。これは、自分が表明する確信度と、実際の正確さのあいだのずれを測り、調整する作法を指します。判断研究におけるキャリブレーション概念の体系化は、1970 年代後半から始まりました。中心人物は、米国の判断研究者 Sarah Lichtenstein、Baruch Fischhof (The same researcher who conducted the hindsight bias research covered in Part 6)Lawrence Phillips There are three people.

"This is what happens when you are calibrated: About 70% of the judgments you say you're 70% sure about are actually correct. About 90% of the judgments you say you're 90% sure about are actually correct. Many people, especially experts, have large deviations in calibration in areas of high confidence (overconfidence)." -- Lichtenstein, Fischhoff & Phillips “Judgment under Uncertainty” (1982) Purpose from included papers

A graph to visualize calibration has self-reported confidence on the horizontal axis and actual accuracy on the vertical axis. If the calibration is perfect, the points will line up diagonally. Empirical research has consistently shown that for both experts and amateurs, points are under に位置する ── 確信度が実際の正確さを上回る ── ということでした。これが過信の数値化された姿です。

The theoretical importance of calibration is that it Making overconfidence measurable It is located in Rather than a vague "be careful," you can understand your own deviations in concrete numbers. Brier score, which will be discussed in the next section, is a standard tool for quantifying this.

03Brier score ─ Quantification of prediction accuracy

確率予測の正確さを単一の数値として捉える指標を、1950 年に米国の気象学者 glenn briar (1913–1998) proposed. today Brier score This metric, known as ``Probability Forecasting,'' began with the validation of weather forecasts and is now used in all areas of probabilistic forecasting, including medical diagnostics, business forecasting, policy evaluation, and artificial intelligence evaluation.

The formula for Brier score is simple.For each prediction, take the square of (predicted probability − actual outcome) and average over all predictions.. The results range from 0 (perfect prediction) to 1 (worst prediction). The important properties areThe Brier score evaluates not only the accuracy of predictions but also the level of confidence. That's it. Predictions that are wrong with confidence are penalized more than predictions that are conservatively wrong.

"The advantage of the Brier score is that it gives the forecaster an incentive to express ``true confidence.'' If you are too confident, you lose a lot of points when you miss. If you are too conservative, you score poorly when you are right. The best scores are obtained when the forecaster expresses his true beliefs at the time." - From Glenn W. Brier, Monthly Weather Review (1950)

As a practical example of this article, we will propose a minimum unit of self-training using the Brier score. Each time you make a material review decision, (1) self-report your confidence in 5% increments, (2) cross-check the results after a certain period of time, and (3) calculate the Brier score. If you collect this data from 30 to 50 cases, you will be able to quantitatively see the bias in your calibration. A pattern often emerges in which experts are more overconfident in areas of high confidence (90-100%).

04Demonstration of calibration training ── Good Judgment Project

Calibrate Can be improved with training The largest empirical study to demonstrate this was the one mentioned in Part 1. philip tetlock and barbara mellers has been implemented since 2011. Good Judgment Project (GJP) is. In a prediction tournament sponsored by the US intelligence agency IARPA, their team gathered thousands of forecasters to collect predictions about geopolitical and economic events over a four-year period.

GJP's findings come directly to the subject of this episode. Firstly,Calibration training (about 1 hour online education) Predictors who received the test were significantly more accurate than those who did not receive the test. Second,Habits of probability thinking Forecasters who are conscious of this will continue to improve their calibration year by year. Third, the most successful forecasters -- super-forecasters -- are not special knowledge special customs distinguished from others by

"What superforecasters had in common was habit, not intelligence. They didn't talk about their predictions in terms of absolute values, but in terms of probabilities. They considered multiple scenarios in parallel. They quickly updated their predictions as information updated. When they were wrong, they reflected on their decision-making process based on their records. These are all behaviors that can be trained." - From Philip Tetlock & Dan Gardner's ``Superforecasting'' (2015)

These four habits have been discovered through modern judgment research. The most practical way to reduce overconfidence です。資材審査員にとっての含意は実用的です。判断を確率言語で記述し、複数シナリオを書き出し、情報更新を素早く反映し、結果を季節ごとに点検する ── これだけで、Brier score の改善が期待できる。

05Pre-mortem ── Counting failure backwards from the future

of judgment before Among the techniques to reduce overconfidence, the most practical one is Pre-mortem is. US decision-making researchers Gary Klein (1944–) became widely known for his 2007 Harvard Business Review article ``Performing a project premortem.''

The pre-mortem process is simple. Before making any important decisions, tell your team:"Suppose that the decision you are about to make now turns out to be a huge failure in a year's time. Make a list of possible reasons why it failed.". This simple shift in perspective can have a dramatic effect on overconfidence.

"Pre-mortem is a technique that leverages prospective hindsight. Our 1989 study found that imagining future outcomes as having happened led people to identify 30% more possible causes of failure. Writing out failure scenarios before making decisions is one of the most effective interventions to reduce overconfidence." -- Gary Klein, Harvard Business Review (2007) more purpose

The psychological mechanism of pre-mortem is based on the hindsight bias discussed in Part 6. in the opposite direction 活用するところにあります。後から振り返ると "予測可能だった" と見える、という Fischhoff の発見を、判断の前に意識的に利用する。"もし失敗していたら、何が原因か" を仮想的に問うことで、判断の前提に潜むリスクが前景化されます。

The biggest advantage of Pre-mortem is that it takes only 15-20 minutes and does not require any special expertise. Projects where material review is difficult, risks are unclear, and decisions are divided between teams -- the introduction of Pre-mortem in these cases functions as an organizational resistance to overconfidence.

06Reference Class Forecasting ─ Mooring to statistics of similar cases

Another powerful technique to reduce overconfidence is toReference Class Forecasting (RCF) is. In his 1993 paper ``Timid choices and bold forecasts,''Daniel Kahneman and Dan Lovallo systematized this technique. later Bent Flubure has applied it to the prediction of large-scale public works projects, and has produced many demonstration cases such as the Copenhagen subway.

The RCF process has three stages. Firstly,類似する過去のケースの集合 (参照クラス) Identify. Second,statistics for that class (average time taken, average cost, success rate, etc.). Thirdly,Using those statistics as a starting point, make limited adjustments to the special circumstances of the current case.。

"People tend to look at decisions from the inside out -- the particularities of this project, the capabilities of this team, the unique context of this situation. But truly accurate predictions start by looking from the outside in -- how many times this type of thing has been attempted in similar situations in the past, and with what results. The outside view is the most powerful deterrent against overconfidence." -- From Daniel Kahneman & Dan Lovallo, Management Science (1993)

The implications of RCF tie directly into Part 4 of this series: Kelley's covariation principle. The core of RCF is the systematic collection of information on "agreementability (will other people get the same outcome in the same situation)" as described by Kelley from past cases. In material examination,Past review results and discovered risk statistics for similar projects, media, and areas This is the organizational foundation of RCF.

07Devil's Advocate ── 制度化された反対者

One of the oldest techniques for combating collective overconfidence is toDevil's Advocate (悪魔の代弁者) is. Historically, it has been institutionalized since the 16th century as a position in the Roman Catholic Church's canonization process to deliberately raise objections to candidate saints. Although it was abolished as a formal position in 1983, the concept continues to deeply influence the theory of organizational decision making.

In modern organizational decision-making research, the most systematic demonstration of the Devil's Advocate effect isshirley schweiger (Schweiger) and colleagues in the 1980s. We have demonstrated that the quality of decisions can be improved by intentionally placing ``pro-side,'' ``opposition,'' and ``Devil's Advocate'' in decision-making meetings.

“The most effective system for resisting groupthink is to institutionalize That's true. Deliberately assigning the role of Devil's Advocate to members treats dissent as a role rather than a persona. This makes it easier to express dissent and improves the quality of group judgment.'' - From Charles R. Schwenk, Strategic Management Journal (1988)

Implementing the Devil's Advocate on a material review team can take several forms. During the meeting, assign a role to a specific member, saying, "Today you are the Devil's Advocate." Before making a judgment, ask students to spend 10 minutes writing out arguments against the subject of their judgment. The important thing is thatPeople who object don't have to "really object" thing. Resistance to collective overconfidence works simply by framing counterarguments as roles. This overlaps with Bohm's reservation that we covered in Part 7.

0810/10/10 Rule ── Change the time scale

Another simple technique to change the context of judgment is the American journalist/author Susie Welch (1959–) proposed in his 2009 book “10-10-10” 10/10/10 rules There is. Although it is not based on a large body of organizational research, it has been widely adopted by practitioners as a practical tool for reducing overconfidence.

The procedure is simple. Before making an important decision, imagine the outcome on three time scales. Me in 10 minutes、Me in 10 months、Me in 10 years. How is this judgment evaluated on each time scale?

“Most decisions are driven by the emotions and interests of the moment. However, important decisions should be evaluated over a long time frame. The 10/10/10 rule consciously stretch out in time It is a simple tool. 10 minutes, 10 months, 10 years -- looking at the same decision on three scales often reveals different perspectives.'' -- From Suzy Welch's ``10-10-10'' (2009)

The value in the context of this episode of 10/10/10 is that Time-limited nature of overconfidence is to clarify. Overconfidence tends to be strongly linked to a short-term perspective. The feeling that ``I'm right now'' is likely to take on a different meaning when viewed from the perspective of 10 months or 10 years from now. In materials review decisions, especially those involving long-term responsibilities (large-scale campaigns, first materials for new drugs, regulatory gray areas decisions), a 10/10/10 perspective shift can be a useful check against overconfidence.

09In the pharmaceutical field ─ 4 practical situations

The seven techniques from the previous eight sections -- Calibration Training, Brier Score, Four GJP Habits, Pre-mortem, Reference Class Forecasting, Devil's Advocate, and the 10/10/10 Rule -- are brought to bear on the material examiner's field. You don't need to use everything every day. Choose the appropriate technique depending on the situation. This is the subject of this section.

Scenes 1 and 3 are group judgments, and Scenes 2 and 4 are judgments at the boundary between individuals and groups. It is not necessary for an organization to implement all seven techniques.Pick the 2-3 most useful things for your team and make them a habit. ことが、現実的な実装の出発点です。

104 etiquette for individuals and small teams

As the final practice of this article, we will pick out the four easiest manners to implement from the seven techniques. These are practices that individuals and small teams can start today without special institutional support.

The common principles of the four manners are:Rather than directly fighting overconfidence, incorporate changes in numbers, writing, third-person perspective, and time axis into the judgment process. That's it. These are techniques that do not directly counter unconscious defense mechanisms (Part 3), but rather change the operating conditions of the unconscious by consciously changing the structure of judgment. This overlaps with Vaillant's transition to a mature defense (Part 3).

We've compiled a toolbox of seven proven techniques for reducing overconfidence. Lichtenstein et al.'s Calibration, Brier score, Mellers et al.'s Good Judgment Project, Gary Klein's Pre-mortem, Kahneman's Reference Class Forecasting, Devil's Advocate, Suzy Welch's 10/10/10 -- seven viewpoints have pointed to the same place using different words.Overconfidence is not something to attack directly, but to be reduced indirectly by incorporating numbers, writing, a third-person perspective, and a time axis into the decision-making process.。

The remaining two parts will focus on the internal and integrative aspects of this series. In the next installment (part 9), we will return to deep psychology for the first time in this series. Between the ideal self and the real self -- Overconfidence caused by the discrepancy between Jungian self-images. By weaving together three perspectives: Jung's persona and shadow, Rogers' self-concordance, and Winnicott's false self and true self, I will illustrate how discrepancies in self-image support overconfidence. This is the episode where we return to the psychological core of this series.