Method

Rules 1.1.0 · as of 14 Sep 2026

Every rule was written before an outcome was checked; the tables come from the rule files, so the page and the score cannot disagree.

Headline

Each number is a Brier score on claims we could check.

Each person's number is a Brier score, computed on every prediction that could be checked against a public source we named before we looked. A prediction that named a deadline is checked at that deadline; one that named none is checked 24 months after it was made, a fixed window that nobody chooses per claim. Each prediction becomes a probability that the event happens by the deadline: the number the person gave, or, when they used words, a fixed table (will = 0.90, probably = 0.70, may = 0.50, unlikely = 0.30, never = 0.10). When the deadline passes, we record 1 if the event happened and 0 if it did not. The score for one prediction is the squared gap between the probability and the outcome: a confident claim that comes true scores 0.01, a confident claim that fails scores 0.81, and 'may' always scores 0.25. A person who predicted the same event several times gets one vote for that event. The headline is the average over events: 0 is perfect, 0.25 is a coin flip, 1 is confidently wrong every time. The range next to it shows how much the number could move if we redrew the events at random; we do not rank two people whose ranges overlap. Beside it we show what the published base rate would have scored on the same events, how often the person was simply on the right side, and the share of their claims that were too vague or undated to check. Plans for the person's own company and inside tips are not counted. Every line links to the quote, the source, the rule we wrote before checking, and the evidence.

lib/content/methodology.ts

Rules

The rules were fixed before we looked, and each has one reason.

#	stage	rule	why
R1	intake	We read the person's own archive in full first, then search the rest with fixed search strings.	A fixed corpus stops us from picking the claims we happen to remember.
R2	intake	A statement enters only if it is sincere, the person's own claim, forward-looking, and inside the five areas.	Jokes, reports, and other people's claims are not forecasts.
R3	intake	Plans for the person's own company (CONTROL) and inside tips (REPORT) are listed but never scored.	A person should not be scored on what they can decide or already know.
R4	intake	Every event is written from one of eight templates, on the asset, with a named public source, before we look at the outcome.	A criterion written after the outcome bends toward it.
R5	intake	'A and B' splits into two events; 'A or B' is one event.	One vote per event keeps a double claim from counting once.
R6	intake	Only the person's words, read through the anchor table, set a deadline; the coder never invents a date.	An invented deadline is the coder's forecast, not the person's.
R7	intake	A claim with no deadline in the person's words takes one uniform window of 24 months from the statement date.	A fixed window scores an undated claim without letting anyone pick its date.
R8	intake	A stated number is the probability; a phrase maps to the five-bin lexicon; a denial takes 1 minus the bin value.	The same word gets the same number for everyone.
R9	intake	The strongest phrase in the claim sentence decides the bin; coders never read tone.	Tone is where hindsight leaks in.
R10	intake	A base rate is fixed at intake from an external table by event class and stage; it is halved when the window is shorter than the class median time to decision; with no class there is no base rate.	We never learn a base rate from the items we score.
R11	scoring	An item is scored only when its deadline is on or before the as-of date and the outcome is true or false.	A claim that is already true but not yet due is still open to a change of criterion.
R12	scoring	Outcome is 1 if the event happened by the deadline, else 0; the item score is (p minus outcome) squared.	The Brier score rewards confidence only when it is right.
R13	scoring	All of one person's items on one event form one cluster; the cluster score is the mean of its scored items; every metric counts a cluster as one observation.	Repeating a claim six times is not six pieces of evidence.
R14	scoring	The headline scores every admitted claim. The dated-only and undated-only views are shown beside it, and the undated window is re-run at 36 months.	One score, with its parts in view. Rules 1.0 scored dated claims only; the change is logged in the corrections log.
R15	scoring	Void and pending items are never scored; a void reason is logged on the item.	An unscored item stays visible so the reader can count what was left out.
R16	scoring	The base rate and the market are scored on the same clusters as the person, with their own probability in place of p; the coin flip is the constant 0.25, not a row.	A score means little without a reference on the same events.
R17	scoring	Two people are ranked only when both are at least T1, share a coverage tier, and their intervals do not overlap.	A rank that the data cannot support is a false claim.
R18	reporting	Below 10 resolved clusters we show counts only; from 10 to 29 the score is provisional; from 30 it is full.	A mean of six items is a story, not a score.
R19	reporting	Items on an entity where the person has a disclosed role are scored and flagged; the headline is shown with and without them.	The reader decides how much an insider's call is worth.
R20	reporting	Intake files carry no outcome fields; the resolver never sees p; release files are hashed before scoring.	Blindness has to be structural, not a promise.
R21	reporting	A rule change re-scores everyone, bumps the rules version, and is logged in the corrections log.	A number that changed without a log is a number nobody can check.

data/rules

Lexicon

The same word gets the same number for everyone.

bin	p	label	phrases
A	0.90	will	will, is going to, certain, certainly, definitely, obviously, no doubt, guaranteed, inevitable, is coming, shall, won't be long
B	0.70	probably	probably, likely, I expect, expect, I think, I believe, I bet, my bet, on track, should (predictive, not normative), I suspect, my guess, I anticipate, more likely than not, substantially more than 50/50
C	0.50	may	may, could, might, possible, possibly, maybe, perhaps, 50-50, coin flip, would not surprise me, it is conceivable
D	0.30	unlikely	unlikely, I doubt, doubtful, do not expect, probably not, I am skeptical that, hard to see
E	0.10	never	never, will not, won't, no chance, impossible, not going to, there is no way, cannot

Five symmetric bins applied to what the forecaster asserts. A denial of E takes 1 minus the bin value. A stated number overrides the map (a range takes its midpoint; 'at least x' takes x). A qualified phrase around a number ('substantially more than 50/50') is a phrase and takes the qualifier's bin. The strongest phrase in the claim sentence decides; coders never read tone.

Alternative maps used by the sensitivity panel: kent (A 0.93, B 0.75, C 0.50, D 0.25, E 0.07); ends85 (A 0.85, B 0.70, C 0.50, D 0.30, E 0.15); flat75 (A 0.75, B 0.75, C 0.50, D 0.25, E 0.25).

Lichtenstein S, Newman JR (1967). Empirical scaling of common verbal phrases associated with numerical probabilities. Psychonomic Science 9:563-564.
Kent S (1964). Words of estimative probability. Studies in Intelligence 8(4).
Budescu DV, Broomell S, Por HH (2009). Improving communication of uncertainty in the reports of the IPCC. Psychological Science 20:299-308.
Reagan RT, Mosteller F, Youtz C (1989). Quantitative meanings of verbal probability expressions. Journal of Applied Psychology 74:433-442.
Wallsten TS, Budescu DV, Rapoport A, Zwick R, Forsyth B (1986). Measuring the vague meanings of probability terms. JEP: General 115:348-365.

data/rules/lexicon.json

Anchor table

Only the person's words set a deadline.

the person wrote	deadline	origin	note
an explicit calendar date	that date	stated	–
a month ('in March 2025', 'by March')	the last day of that month	anchor	–
a quarter ('Q2 2026')	the last day of that quarter	anchor	–
a year ('in 2026', 'by 2026', 'this year', 'next year')	31 December of that year	anchor	–
H1 / first half	30 June of that year	anchor	–
H2 / second half	31 December of that year	anchor	–
an annual predictions list published on or near 1 January	31 December of that year	anchor	–
'by the PDUFA date' or 'at the PDUFA date'	the published PDUFA date at the statement date	anchor	–
'by the end of the decade' / 'this decade'	31 December of the decade's last year	anchor	–
'within N months/years' / 'in N years'	statement date plus N months or years	anchor	–
'in a decade' / 'in ten years'	statement date plus 10 years	anchor	–
a trial result claim with no date ('when the trial reads out')	the ClinicalTrials.gov primary completion date on the statement date plus 12 months	table	tag table_dated
a named future meeting or event ('at APA in May', 'at JPM in January')	the last day of that meeting	anchor	–
'soon', 'coming', 'eventually', 'obviously', 'will follow', 'imminent', 'shortly', 'in the near future', 'time will tell'	none	undated panel	not a deadline; the item goes to the undated panel

Only the forecaster's words, read through this table, set a headline deadline. The coder never invents a date. Items with no deadline go to the undated panel with one uniform window of 24 months from the statement date.

data/rules/anchors.json

Quantity rules

A number claim resolves on the named series only.

#	rule
Q1	A quantity claim resolves on the named public series only; no substitute series is accepted.
Q2	A direction claim ('more', 'fewer', 'up', 'down') resolves as a binary at the threshold in the person's words.
Q3	A bare point number takes a fixed band of plus or minus 20 percent; inside the band is true.
Q4	A range takes its endpoints; 'at least x' resolves at x; 'about x' takes the band of Q3.
Q5	If the series has not published 12 months after the deadline, the event is unresolvable and the item is void.
Q6	Realized values are stored on the outcome and shown; they are never scored as a distance.

Template criterion: Resolved on the named data series only. A direction resolves as binary at the threshold in the forecaster's words. A bare point number takes a fixed band of plus or minus 20 percent. If the series has not published 12 months after the deadline, the event is unresolvable. Realized values are stored, not scored.

data/rules/templates.json · quantity_threshold

Reason codes

Vague, controlled and reported claims never enter the score.

code	label	test
VAGUE	Vague or promotional	No entity, observable event, benchmark or threshold can be written from the text and its immediate context. Slogans ('game changer', 'once-in-a-lifetime opportunity', 'favourite stock') and quantifiers with no countable set ('more', 'first', 'all') without a defined set.
CONTROL	Commitment (own venture)	The forecaster's own organization can make the outcome happen by its own decision: clinic openings, product launches, hires, trial starts, filings by their own company. Listed in the commitments table, never scored.
REPORT	Report or scoop	The statement passes on information rather than an expectation: cites sources ('I am told', 'sources say'), or names a date for another organization's internal action (layoffs, hires, closures) 30 days or less ahead. Listed in the reports table with a confirmation rate, never scored.
NORMATIVE	Normative	'Should', 'must', 'ought to', 'needs to' with no predictive reading.
THIRD_PARTY	Third-party claim	A quoted or paraphrased claim by someone else that the author does not adopt as their own.
SATIRE	Satire or irony	Marked or evident satire, hyperbole or irony.
KNOWN	Outcome already public	The outcome had been publicly determined or announced by the deciding body or sponsor on the statement date.
OUT_OF_AREA	Outside the five areas	Outside the five area definitions in areas.json: not a regulatory decision, trial outcome, company or market outcome, payer or policy decision, or practice adoption about drugs, devices or treatments in psychiatry, or about payment for and access to them. The registry applies the same test when it consolidates proposed events.
NOT_FORECAST	Not a forecast	Descriptive, retrospective, or a statement of preference; no future state asserted.
DUPLICATE	Duplicate extraction	The same statement in the same post was captured twice; the first row stands.
PARAPHRASE	Paraphrase, not own words	The only record of the claim is a third party's paraphrase or a secondhand report; the forecaster's own words are not available. Shown as a row, never coded or scored.

Void after intake

code	label	test
AMBIGUOUS	Coders disagree	The two coders wrote different registry events or different deadlines for the same statement. No adjudication toward a reading.
CONDITION_UNMET	Condition unmet	'If A then B': A did not occur by the deadline.
UNRESOLVABLE	Unresolvable	The named resolution source has not published 12 months after the deadline, or the source no longer exists.

Tags (scored, but flagged)

tag	test
affiliated	A third party decides, but the forecaster has a disclosed role, stake, employment, board seat or trial site in an entity named in the event. Scored and flagged; the headline is shown with and without.
retrospective	The item was coded after its outcome was public (all 2021 to mid-2026 items in release 1.0).
prospective	The item was coded and frozen before its outcome was public.
table_dated	The deadline came from the ClinicalTrials.gov table rule, not from the forecaster's words.
denial	The forecaster denied the proposition; p(E) = 1 minus the bin value.
stated_number	The forecaster gave a numeric probability; it overrides the lexicon.

data/rules/reason-codes.json

Event templates

Every event is written on the asset, from one of eight templates.

Every registry event is written from one template. The entry is written on the asset (drug, device, program, rule), never on the corporate name, so mergers and renames do not void it. 'A and B' splits into two events; 'A or B' is one event. The same entry resolves every forecaster's items.

template	proposition	criterion	source
drug_approval	<Regulator> approves <drug> for <indication>.	TRUE on the date the regulator's approval letter or database entry (or the sponsor's approval announcement) is dated. A complete response letter, a positive advisory vote or a filing acceptance is not an approval. A label narrower than the indication named in the claim counts if the claim's indication reasonably includes it and the reading is logged at intake.	Drugs@FDA or the regulator's database; sponsor press release as a secondary source.
device_authorization	<Regulator> grants <clearance, De Novo, PMA or breakthrough designation> for <device> for <indication>.	TRUE on the decision date in the regulator's database (510(k), De Novo, PMA) or the sponsor's announcement of a designation. Intended-use and setting (home use, clinic) must match the claim.	FDA 510(k), De Novo or PMA database; sponsor release for designations.
trial_result	<Program> reports <primary endpoint met or not met> in <trial> (or <effect size threshold>).	TRUE on the date of the sponsor's topline release or the publication that first reports the result. 'Positive' means the prespecified primary endpoint met significance as reported by the sponsor; an effect-size threshold in the claim is resolved on the reported estimate.	Sponsor topline press release; ClinicalTrials.gov results; peer-reviewed publication.
rule_or_policy	<Body> adopts, extends, rescinds or issues <rule, guidance, order, law or program>.	TRUE on the publication date in the Federal Register, the agency site, the state register or the enacted bill's effective or signing date as named in the claim. A proposal is not adoption unless the claim was about the proposal.	Federal Register; agency press release; state legislature record.
court_outcome	<Court or agency> <convicts, sentences, rules, dismisses or settles> in <case>.	TRUE on the docket date of the verdict, sentence, order or filed settlement named in the claim.	Court docket (CourtListener, PACER), DOJ press release.
company_event	<Company or program> <is acquired, files for bankruptcy, lays off staff, raises, lists, delists, shuts down, launches a product, or misses or hits a number>.	TRUE on the date of the definitive public announcement (SEC filing, press release, court filing) of the event named in the claim; rumors and letters of intent do not count unless the claim was about them.	SEC EDGAR; company press release; bankruptcy docket.
coverage_decision	<Payer or program> covers, stops covering, or changes payment for <treatment or code>.	TRUE on the effective date of the published coverage policy, fee schedule, NCD or LCD, or the payer's public policy bulletin named in the claim.	CMS Medicare Coverage Database; payer policy bulletins; Federal Register fee schedule.
quantity_threshold	<Metric> is at or beyond <threshold> (or moves in <direction>) by <period end>, per <named series>.	Resolved on the named data series only. A direction resolves as binary at the threshold in the forecaster's words. A bare point number takes a fixed band of plus or minus 20 percent. If the series has not published 12 months after the deadline, the event is unresolvable. Realized values are stored, not scored.	The named public series (IQVIA, SEC filings, FDA database counts, a named investment tracker).

data/rules/templates.json

Reference rows

The base rate and the market are scored on the same events as the person.

row	how	min n
Base rate	Mean over resolved clusters, all persons pooled, of the cluster score computed with the base rate in place of p; only clusters where every item carries a base rate.	10
Market	Same as the base rate with the market price in place of p: the last quote up to 7 days before the statement, on a public question whose deadline is within one quarter of the item's.	5
Coin flip	Not a row. The constant 0.25 is drawn as a reference line: the score of 0.50 on every item.	–

The halving rule

Base rate b is fixed at intake from this table by event class and the asset's stage on the statement date. If the item's window (statement date to deadline) is shorter than the class median time to decision, b is halved: P(E in window) = P(E | stage) x P(T <= window | E) <= 0.5 x P(E | stage) when the window is below the median, so halving is an upper bound that favours the forecaster. If no class fits, the item has no b and the reference row abstains on it; 0.5 is never substituted. External studies only; the ledger never derives a base rate from its own items.

Base-rate classes

class	p	median months	source
Phase 3 program to FDA approval, psychiatry (phase3_to_approval_psychiatry)	0.563	36	Wong CH, Siah KW, Lo AW (2019) Biostatistics 20(2):273-286; BIO/Informa/QLS Clinical Development Success Rates 2011-2020 (2021)
Phase 3 program to approval, all indications (phase3_to_approval_all)	0.578	36	BIO/Informa/QLS 2021 (phase 3 to NDA/BLA 57.8%)
Phase 2 program to approval, all indications (phase2_to_approval_all)	0.150	72	BIO/Informa/QLS 2021 (phase 2 to phase 3 28.9% x phase 3 to NDA 57.8% x NDA to approval 90.6%)
NDA/BLA submitted to approval (nda_submitted_to_approval)	0.906	10	BIO/Informa/QLS 2021 (NDA/BLA to approval 90.6%); PDUFA standard review 10 months, priority 6 to 8 months
First-cycle approval of a novel drug at the PDUFA date (first_cycle_nme_approval)	0.500	10	Sacks LV et al (2014) JAMA 311(4):378-384 (about half of NME applications approved on the first cycle)
Eventual approval after a complete response letter (post_crl_eventual_approval)	0.640	14	Analyses of CRLs 2013-2023 report 47 to 64 percent eventual approval with a median of about 14 months to resubmission approval
Approval after a negative advisory committee vote (negative_adcomm_to_approval)	0.330	12	FDA-AdComm concordance about 88 percent; roughly one third of negative votes are followed by approval
Breakthrough therapy designation to approval (btd_to_approval)	0.720	48	Analyses of BTD programs 2012-2022
PDUFA date extended by three months (pdufa_extension)	0.120	0	Share of NDA reviews with a major-amendment extension
DEA telemedicine flexibility extended again (dea_telemedicine_extension)	0.830	0	Laplace rule (4 extensions plus 1) / (4 plus 2) on the 2023, 2024, 2025 and 2026 extensions
510(k) clearance after submission (device_510k_clearance_after_submission)	0.850	5	FDA 510(k) decision statistics (about 85 percent substantially equivalent)
Announced corporate action completes as announced (generic_binary_company_event)	–	–	Abstain: no external study

data/rules/base-rates.json

Metrics

Eleven metrics, each with its formula and its minimum n.

#	metric	formula	min n	unit	note
M1	Mean Brier (headline)	B_c = mean over scored items i in cluster c of (p_i - o_i)^2. B = mean over resolved clusters c of B_c, over every admitted item (dated items at their deadline, undated items at the fixed window).	10 · min_clusters_headline	0 perfect · 0.25 coin flip · 1 confidently wrong	Also per area (null below 10 clusters in the area) and as a cumulative mean by calendar quarter, clusters ordered by resolution date.
M2	Interval and stability	Interval = percentile 2.5 and 97.5 of B over 2000 resamples of clusters with replacement (mulberry32, seed 20260913, clusters sorted by id first). LOO = max over clusters of \|B without c - B\|.	10 · min_clusters_headline	Brier points	The same resample indices serve M4 and M5, so a ratio is bootstrapped jointly with its parts.
M3	Hit rate	hit_i = 1 if (p > 0.5 and o = 1) or (p < 0.5 and o = 0), else 0; items with p = 0.5 are excluded. H_c = mean of hit_i in c. H = mean over contributing clusters. Wilson 95 percent interval with n = contributing clusters.	10 · min_clusters_headline	share of clusters on the right side	Hit rate ignores how confident the person was; M1 does not.
M4	Base-rate skill (paired)	S_b = clusters whose items all carry a base rate b. BS_b = mean over S_b of the cluster mean of (b_i - o_i)^2. BSS_base = 1 - B(S_b) / BS_b. Interval from the M2 resamples restricted to S_b.	10 · min_clusters_headline	positive beats the base rate · 0 equals it	Reported with n = \|S_b\| and BS_b as the difficulty of the paired set.
M5	Market skill (paired)	As M4 with the market price m_i in place of b_i, on S_m = clusters whose items all carry a market price.	5 · market_min_clusters	positive beats the market	Markets are thin in this field; n is usually small.
M6	Calibration by bin	Items with a stated or registered p go to the nearest bin. Per bin: forecast = bin p; observed = weighted mean of o with item weight 1 / (items in its cluster); n = clusters touching the bin. A bin with n < 5 merges into its neighbour toward 0.50.	20 · calibration_min_clusters	observed share of true outcomes per bin	A cluster contributes one unit across the bins it touches.
M7	Murphy decomposition	B = REL - RES + UNC over the M6 bins and weights. REL = sum_k w_k (f_k - obs_k)^2. RES = sum_k w_k (obs_k - obs_bar)^2. UNC = obs_bar (1 - obs_bar).	30 · murphy_min_clusters	Brier points	REL is how far the words were from the truth; RES is how much the words sorted outcomes; UNC is how hard the events were.
M8	Non-affiliated block	M1 to M3 recomputed on clusters where no item carries the affiliated flag.	10 · min_clusters_headline (both sides)	as M1 to M3	Null unless the affiliated side and the non-affiliated side each have at least the minimum number of clusters.
M9	Composition	scoreable share = admitted headline items / sincere statements (Wilson, n = sincere). undated share = undated items / admitted. affiliated share = affiliated items / admitted. extreme share = items in bin A or E, or stated p >= 0.85 or <= 0.15, / admitted. concentration = max over events of scored items on that event / all scored items. median lead = median months from statement date to deadline over headline items. max re-dating = max distinct deadlines in one cluster. prospective share = clusters whose items all carry the prospective flag / resolved clusters. Not-admitted counts by reason code.	– · none (counts)	shares and counts	Composition tells the reader what the score is made of before they read it.
M10	Timing	For scored items with state false and a recorded outcome date: months(outcome date) - months(deadline). Median, first and third quartile, n.	– · none (reported with n)	months late	A right call that came late scores as a miss in M1; M10 shows by how much.
M11	Shared events and paired comparison	For every registry event with items from two or more people: each person's deadline, p, state, and cluster score. For a pair (a, b): d_j = B_a,j - B_b,j over shared resolved events j; mean d with a bootstrap interval over events; counts of d < 0, d = 0, d > 0.	5 · shared_event_min	Brier points, negative favours a	The only comparison that holds the events fixed between two people.

Ranking rule: a helper canRank(a, b) returns true only when both rows are T1 or better, both persons share a coverage tier, and the two intervals do not overlap. The leaderboard shows rank only when canRank is true.

data/rules/thresholds.json

Evidence tiers

Below 10 clusters we show counts, not a score.

tier	label	range	shown
T0	counts only	fewer than 10 resolved clusters	counts of true, false, pending, void; no Brier score, no interval, no rank
T1	provisional	10 to 29 resolved clusters	score and interval, marked provisional; can be ranked against another T1 or T2 person in the same coverage tier when intervals do not overlap
T2	full	30 or more resolved clusters	score, interval, calibration and decomposition when their own minimums are met

data/rules/thresholds.json

Coverage tiers

We rank only people whose archives were read the same way.

tier	label	definition
A	complete archive	Every item in the person's own channel was read in full; a coverage file lists the posts that were missing or stubs.
B	archive plus search	The person's own channel was read in full and secondary sources were added with fixed search strings; secondary coverage is not complete.
C	search-built	No complete archive exists; items come from fixed search strings, transcripts and press quotes. The absence of a claim is not evidence.

A rank is shown only between people in the same coverage tier. A C-tier corpus under-counts claims, and the missing claims are not random.

data/forecasters.json · corpus_tier

Hindsight controls

Coders never see outcomes, and resolvers never see probabilities.

#	control	where
H1	Stage files: statements and items carry no outcome fields, so a coder cannot see an outcome while coding.	lib/data/schema.ts (Statement, Item)
H2	Registry before look: the event proposition, criterion, source and readings are written at intake and versioned.	data/rules/templates.json; registry event fields created_at, version
H3	Blind resolver: the resolver pass writes the outcome and its evidence without seeing p or the person's words.	Outcome records (resolver, evidence, checked_through)
H4	Anchor table: a deadline comes only from the person's words; 'soon' and 'eventually' send the item to the undated panel.	data/rules/anchors.json
H5	Lexicon: the strongest phrase decides the bin; tone, context and later posts are never read into p.	data/rules/lexicon.json
H6	KNOWN code: a statement made after the deciding body announced the outcome is not admitted.	data/rules/reason-codes.json
H7	Freeze: every release file is hashed (sha256) at freeze; scores are recomputed from the frozen files only.	Freeze record (release, files, sha256)
H8	Retrospective flag: every item coded after its outcome was public carries the retrospective tag; the prospective share is reported next to the score.	Item tags retrospective, prospective
H9	Hindsight scan: each item is marked clean, flagged or reviewed after a second read for outcome words in the coding.	Item field hindsight_scan
H10	Second coder: coder B codes admit, event, deadline, bin and asserts without seeing coder A; agreement is reported as kappa; a disagreement on event or deadline voids the item (AMBIGUOUS) and is never adjudicated toward a reading.	CoderB records; lib/score/kappa.ts
H11	Rechecks: any outcome can be challenged on state, date, criterion, evidence or leakage; the verdict and whether it was applied are logged.	Recheck records
H12	Corrections log: a change to any published number or rule is logged with the version it moved from and to.	Corrections records; this page
H13	Registry scope gate: when proposed events are merged into the registry, an event outside the five area definitions gets no entry, and its statement returns to not admitted (OUT_OF_AREA). The gate reads subject area only, never outcomes or odds.	data/areas.json; prompts/registry-consolidate.md; Statement coders.gate

Coder agreement (kappa) on 2,622 double-coded statements. Event and deadline on 197 admitted items; bin on 196; 44 outcomes rechecked, 29 upheld.

field	kappa
admit	0.81
event	1.00
deadline	1.00
bin	0.79
asserts	–

Prefilter check: the census finder set 342 statements aside as not forecasts and sent 2,153 to the coders. Coder B re-read a seeded sample of 69 of the set-aside statements and admitted 0, an estimated 0 missed forecasts in the whole set-aside group.

lib/data/schema.ts · data/rules

Sensitivity panel

The headline is shown again under seven alternative rules.

variant	label	what changes
ends_085_015	Softer ends	Lexicon map A 0.85, B 0.70, C 0.50, D 0.30, E 0.15: 'will' and 'never' lose some confidence; the middle bins do not move.
kent	Kent words	Lexicon map A 0.93, B 0.75, C 0.50, D 0.25, E 0.07: the Kent (1964) scale for words of estimative probability.
flat_075	Flat 0.75	Lexicon map A 0.75, B 0.75, C 0.50, D 0.25, E 0.25: every phrase is 0.75 on the side it leans to; 'may' stays 0.50.
non_affiliated	Non-affiliated only	Clusters with any affiliated item are removed.
prospective_only	Prospective only	Only clusters whose items all carry the prospective flag.
dated_only	Dated only	Only items with a deadline in the person's own words; undated items removed. This was the rules 1.0 headline.
undated_36	Undated window 36 months	Every item again, with undated items at a 36-month window instead of 24.
loo_max_change	Leave one out	The largest change in the headline when one cluster is left out (the M2 stability number).

A lexicon variant changes only items whose p came from the lexicon: p = variant[bin] when the item asserts the event, else 1 - variant[bin]. Stated and registered probabilities never change. Each variant is null below 10 resolved clusters.

variant	Muir	Angermayer	Doblin
headline	0.201 (n 21)	0.520 (n 10)	0.361 (n 13)
Softer ends	0.194 (n 21)	0.486 (n 10)	0.334 (n 13)
Kent words	0.202 (n 21)	0.564 (n 10)	0.370 (n 13)
Flat 0.75	0.191 (n 21)	0.412 (n 10)	0.334 (n 13)
Non-affiliated only	0.201 (n 21)	–	–
Prospective only	–	–	–
Dated only	–	–	–
Undated window 36 months	0.164 (n 12)	–	0.346 (n 13)
Leave one out	0.030 (n 21)	0.053 (n 10)	0.037 (n 13)

data/rules/lexicon.json · sensitivity_maps

Worked examples

One event gives one vote, however often it was predicted.

A repeated claim: one event, six deadlines, one vote

Rick Doblin · six statement dates

Six statements that FDA would approve MDMA-assisted therapy for PTSD, each with its own date.

MAPS bulletins, conference talks and press interviews; each item links to its own quote.

Admit. Each statement is sincere, his own, forward-looking and regulatory. Each is admitted.
Register. One registry event, template drug_approval: 'FDA approves MDMA-assisted therapy (midomafetamine) for PTSD.' Resolution source: Drugs@FDA; sponsor release as a secondary source.
Date. The anchor table reads six deadlines from his words, from 2021-12-31 to 2024-08-31. Every deadline is on or before the as-of date, so every item is in the scoring window.
Probability. Five statements use 'will' (bin A, p 0.90). One uses 'probably' or 'likely' (bin B, p 0.70). No stated numbers.
Resolve. The event had not occurred by any of the six deadlines (a complete response letter, not an approval, in August 2024). Outcome 0 for all six.
Score items. (0.90 - 0)^2 = 0.81 for each 'will' item; (0.70 - 0)^2 = 0.49 for the 'probably' item.
Score the cluster. (5 x 0.81 + 0.49) / 6 = 0.757. This cluster is one observation in his headline mean.
Hit rate. Every item had p above 0.5 and outcome 0, so the cluster hit is 0.
Timing. If the event occurs later, each of the six items records the months between its deadline and the outcome date in M10.

items	phrase bin	p	outcome	item Brier	subtotal
5	A 'will'	0.90	0 (false)	0.81	4.05
1	B 'probably'	0.70	0 (false)	0.49	0.49
6				sum / 6	0.757

Cluster Brier 0.757: one vote in the headline mean, whether he said it once or six times.

A compound claim: two events, one deadline, pending

Owen Scott Muir · 2026-01-01

COMP360 and KetaFree (IV Ketamine) will be approved for MDD by FDA.

The Frontier Psychiatrists, 'Ten 2026 Predictions on Which to Hang Your Hat.', 2026-01-01.

Admit. Sincere, his own claim, forward-looking, regulatory. Neither drug is made by an entity in his affiliations table, so no affiliated flag.
Split. 'A and B' is two claims. Two registry events from the drug_approval template: FDA approves COMP360 (psilocybin) for major depressive disorder; FDA approves KetaFree (IV ketamine) for major depressive disorder. Each event is its own cluster.
Date. The post is an annual predictions list published on 1 January, so the anchor table sets the deadline to 2026-12-31 for both events (origin: anchor).
Probability. 'will' is bin A, so p = 0.90 for each event (origin: lexicon). No stated number.
Tags. Coded before the outcome was public, so both items carry the prospective flag.
State. The deadline is after the as-of date, so it has not passed. Both items are pending and count as zero observations in every metric.
Known true. If FDA approves one drug before 2026-12-31, the row shows 'known true, enters on 2026-12-31'. It is still pending in every metric until that date.
What it will score. On 2026-12-31 each event scores (0.90 - 1)^2 = 0.01 if approved by then, or (0.90 - 0)^2 = 0.81 if not. The two events are two votes.

event	deadline	p	origin	state	enters
FDA approves COMP360 for MDD	2026-12-31	0.90	lexicon A	pending	2026-12-31
FDA approves KetaFree for MDD	2026-12-31	0.90	lexicon A	pending	2026-12-31

Two pending clusters. They add to the pending count now and to the score after 2026-12-31.

lib/score/brier.ts

Known limits

The score is honest about what it cannot show.

#	limit	effect
L1	Small n.	Below 30 resolved clusters the interval is wide and the score is provisional; one cluster can move it (see the leave-one-out number).
L2	Retrospective coding in release 1.0.	Items from 2021 to mid-2026 were coded after their outcomes were public. The controls reduce hindsight; they do not remove it. The prospective share says how much of the score is clean.
L3	Coverage differs by person.	A C-tier corpus under-counts claims. Ranks are shown only inside one coverage tier.
L4	The lexicon is a convention.	'will' is 0.90 because the literature puts it there, not because the person meant it. The sensitivity panel shows how much the choice matters.
L5	Window-close scoring punishes a late right call.	A claim that came true after its deadline is a miss. The timing metric reports the lag so the reader can judge it.
L6	Base-rate classes come from other fields and stages.	A class may fit an event loosely. Items with no fitting class have no base rate, and the reference row abstains on them.
L7	Markets are thin.	Few events have a public market question with a matching deadline; market skill usually has a small n.
L8	Splits and readings are coder decisions.	How a compound claim splits and which label counts as the claimed indication are logged at intake, but they are choices.
L9	One event can dominate.	A person who wrote often about one program has a score that leans on that program. Concentration reports the share.
L10	The 24-month undated window is a convention.	An undated claim is scored at a window nobody chose per claim. The 36-month run and the dated-only view show how much the window matters.
L11	The registry is generous on indication.	An approval narrower than the claimed indication counts as true when the reading was logged at intake. This favours the person.

lib/content/methodology.ts

Corrections

Every change to a published number is logged here.

date	scope	change	reason	from	to
14 Sep 2026	Coder agreement (kappa) on admission	The coder-B match file now holds coder B's decision for every double-coded statement (2,153), not only the 77 statements that reached the admitted path. Admission kappa 0.75 replaces the -0.07 shown for a few minutes on 2026-09-14. Items, statements and the registry are byte-identical to release 1.0.0; the freeze record 1.0.1 hashes the new match file.	The merge script wrote the match file only for admitted-path statements, which biased the admission denominator.	1.0.0	1.0.1
14 Sep 2026	Denial tag on two inverted items (owen-0454, owen-0457)	The denial tag now follows the asserts flag after the registry polarity flip; both items keep p 0.70 and 0.90 on the extension event E-0014. Only the sensitivity variants that read the tag change.	The merge script added the denial tag from the coder record but did not remove it when the consolidation inverted the proposal.	1.0.1	1.0.2
14 Sep 2026	Headline panel (rules 1.1.0)	The headline Brier score now covers every admitted claim: dated claims at the deadline in the person's words, undated claims at the fixed 24-month window. Under rules 1.0 the headline covered dated claims only and the undated claims sat in a separate panel; that dated-only score stays visible as the 'dated only' view and sensitivity variant. No item, event or outcome changed.	With fewer than ten dated claims resolved per person, the rules 1.0 headline could not show a score before 2027. The 24-month window was already a fixed, pre-registered rule and the pooled score was already published as a sensitivity variant; making it the headline shows a score that the data supports while keeping the dated-only number in view.	1.0.2	1.1.0

data/corrections.json

Versions

Rules version 1.1.0 is in force.

version	date	as of	note
1.1.0	14 Sep 2026	14 Sep 2026	Rules 1.1.0: the headline scores every admitted claim; a claim with no stated deadline takes the fixed 24-month window. Rules 1.0 scored dated claims only. Rules, lexicon, anchor table, templates, base-rate table and thresholds share this version. A change re-scores everyone and is logged in the corrections log.
1.0.0	13 Sep 2026	13 Sep 2026	First release. The headline scored dated claims only; undated claims sat in a separate 24-month panel that was never pooled.

data/rules/version.json · data/VERSION

Glossary

Each term keeps one meaning across the site.

term	definition	see
admitted	A statement that passed intake: sincere, the person's own claim, forward-looking, inside the five areas, and not excluded by a reason code. Only admitted statements become items.	reason code, item
affiliated	A flag on an item where a third party decides the event but the person has a disclosed role, stake, employment, board seat or trial site in an entity named in the event. Scored and flagged; the headline is shown with and without.	non-affiliated block
anchor table	The fixed table that turns the person's own words ('by 2026', 'Q2', 'within two years') into a deadline. The coder never invents a date.	deadline, undated panel
area	One of five topic classes: regulatory decisions, clinical trial outcomes, company and market outcomes, payer and policy, practice adoption. Every event has one area.
as-of date	The date the scores were computed for, read from data/VERSION. An item is scored only when its deadline is on or before this date.	window-close scoring
asserts	Whether the statement says the event will happen (true) or will not (false). Items store p on the side of the event happening; asserts is informational.	denial
base rate	The probability a published external study gives an event of that class and stage, fixed at intake. Halved when the item window is shorter than the class median time to decision. Never learned from the ledger's own items.	halving rule, reference row
bin	One of five lexicon steps: A 'will' 0.90, B 'probably' 0.70, C 'may' 0.50, D 'unlikely' 0.30, E 'never' 0.10. A stated number goes to the nearest bin for calibration only.	lexicon
bootstrap interval	The 2.5 and 97.5 percentiles of a statistic over many resamples of the clusters with replacement, drawn with a seeded generator so every run gives the same interval. It shows how much the number could move if the events were redrawn.	cluster
Brier score	The squared gap between a probability and the outcome, (p - o)^2. 0 is perfect, 0.25 is the constant coin flip, 1 is confidently wrong. Lower is better.	cluster Brier, mean Brier
calibration	For each bin, the share of events that happened against the probability the bin claims. A person is calibrated when 'will' events happen about 90 percent of the time and 'may' events about half the time.
cluster	All of one person's items on one registry event. A cluster is one observation in every metric, so a claim repeated six times still gets one vote.	cluster Brier
cluster Brier	The mean of the item Brier scores of the scored items in a cluster.	cluster, Brier score
coin flip	The constant 0.25, the Brier score of saying 0.50 on every item. Drawn as a reference line; not a row on the leaderboard.
compound split	'A and B' becomes two registry events and two clusters; 'A or B' stays one event. Each split is logged at intake.
concentration	The largest share of a person's scored items that sit on one event. High concentration means the score leans on one program.
corrections log	The public list of every change to a published number or rule, with the version it moved from and to.
coverage tier	How completely a person's claims were collected: A complete archive, B archive plus search, C search-built. Ranks are shown only inside one tier.
dated view	The headline panel restricted to items with a deadline in the person's own words. Shown beside the headline; it was the rules 1.0 headline.	headline panel
deadline	The date by which the event must happen for the item to be true. It comes only from the person's words through the anchor table.	anchor table
denial	A statement that the event will not happen. Its probability on the event side is 1 minus the bin value.	asserts
evidence tier	How much resolved evidence a person has: T0 fewer than 10 clusters (counts only), T1 10 to 29 (provisional), T2 30 or more (full).	cluster
extreme share	The share of admitted items in bin A or E, or with a stated probability at or beyond 0.85 or 0.15. A measure of how bold the person's words are.
freeze	A release of the data files with a sha256 hash of each. Scores are recomputed from the frozen files only, so nothing can change after the outcomes are looked up.
halving rule	When the item window is shorter than the base-rate class's median time to decision, the base rate is halved. The halved value is an upper bound that favours the person.	base rate
headline panel	Every admitted item: dated items checked at the deadline in the person's words, undated items checked at the fixed 24-month window. The headline score is computed on this panel.	dated view, undated panel
hindsight scan	A second read of each item for outcome words in the coding. The item is marked clean, flagged or reviewed.
hit rate	The share of clusters where the person was on the right side of 0.50. Items at exactly 0.50 are left out. Reported with a Wilson interval.
item	An admitted statement with its intake fields: event, deadline, probability, tags. An item carries no outcome fields; the outcome lives on the registry event.	admitted, registry event
kappa	Cohen's kappa: agreement between two coders beyond chance, from -1 to 1. Reported for admit, event, deadline, bin and asserts from the second-coder match test.
known true	An item whose event has already happened but whose deadline has not passed. Shown as 'known true, enters on <deadline>'; pending in every metric until then.	pending, window-close scoring
leave-one-out change	The largest change in the mean Brier when one cluster is removed. A stability check next to the interval.
lexicon	The fixed five-bin table that turns a phrase into a probability. The strongest phrase in the claim sentence decides; coders never read tone.	bin
market reference	The last public prediction-market price up to 7 days before the statement, on a question whose deadline is within one quarter of the item's. Used for the market row and market skill.	reference row
mean Brier	The mean of the cluster Brier scores over a person's resolved clusters. The headline number.	cluster Brier
Murphy decomposition	The mean Brier split into reliability (how far the words were from the truth), resolution (how much the words sorted outcomes) and uncertainty (how hard the events were): B = REL - RES + UNC.
non-affiliated block	The headline metrics recomputed on clusters with no affiliated item. Shown only when both sides have enough clusters.	affiliated
not admitted	A statement that failed intake for a logged reason code. Listed and counted, never scored.	reason code
outcome	1 if the registry event happened on or before the item's deadline, 0 if it did not. Written by a resolver who never sees p.
paired comparison	For two people, the difference in cluster Brier on the events they both predicted, averaged over those events with a bootstrap interval. The only comparison that holds the events fixed.
panel	A pool of items scored together. The headline panel holds every admitted item; the dated view and the undated panel are its two parts.	headline panel, undated panel
pending	An item whose deadline has not passed. Counted, never scored.	known true
prospective	An item coded and frozen before its outcome was public. The prospective share says how much of a score is free of hindsight.	retrospective
re-dating	The same person naming a new deadline for the same event. Each deadline is its own item inside one cluster; the maximum number of distinct deadlines in one cluster is reported.
reason code	The logged reason a statement was not admitted: VAGUE, CONTROL, REPORT, NORMATIVE, THIRD_PARTY, SATIRE, KNOWN, OUT_OF_AREA, NOT_FORECAST, DUPLICATE.	not admitted
reference row	A leaderboard row that is not a person: the base rate or the market, scored with its own probability on the same clusters. Shows what a rule of thumb would have scored.	base rate, market reference
registry event	A proposition written from a template on the asset, with a resolution criterion and a named public source, before the outcome is looked up. The same event resolves every person's items on it.	template
resolved	A cluster with at least one scored item. Only resolved clusters count toward n.
retrospective	An item coded after its outcome was public. All release 1.0 items from 2021 to mid-2026 carry this tag.	prospective
scoreable share	Admitted headline items divided by sincere statements, with a Wilson interval. The share of what the person said that could be checked.
sensitivity panel	The headline mean Brier recomputed under alternative rules: softer lexicon ends, Kent words, flat 0.75, non-affiliated only, prospective only, dated only, undated window at 36 months, leave one out.
skill score	1 minus the person's Brier divided by a reference Brier on the same clusters. Positive beats the reference; 0 equals it.	base rate, market reference
template	One of eight fixed forms for a registry event (drug approval, device authorization, trial result, rule or policy, court outcome, company event, coverage decision, quantity threshold), each with its own criterion and source type.	registry event
timing	For items scored false whose event later happened: the months between the deadline and the outcome date. Median and quartiles are reported.
undated panel	Items with no deadline in the person's words, scored on a uniform 24-month window from the statement date. Part of the headline panel since rules 1.1.0; also shown on its own.	headline panel, dated view
void	An item that cannot be scored: the coders disagreed (AMBIGUOUS), the condition was unmet, or the source never published (UNRESOLVABLE). Counted, never scored.
Wilson interval	A 95 percent interval for a share that stays inside 0 and 1 and behaves well at small n. Used for hit rate and scoreable share.
window-close scoring	An item enters the score only when its deadline has passed and the outcome is true or false. A claim already known true still waits for its deadline.	as-of date, known true

lib/content/glossary.ts