domains / families / research / 09-search-procedures

09-search-procedures

separate

Model how a specification universe is actually walked, rather than assuming it is enumerated. Implements the search procedures a p-hacker or a pressured agent uses (first-significant stopping, random trial within a budget, greedy coordinate descent from the pre-registered analysis, random hill-climbing, and the two-stage split-sample walk with an optional continuation rule, after Adda, Decker and Ottaviani 2020), replays each on null data to measure its false-positive rate and the inflation of what it reports, and records the path so the write-up can be checked against it. Use when asked what a realistic p-hacking session looks like, how likely a given way of searching is to manufacture significance on a specific design, how to calibrate a result that came from a sequential rather than exhaustive search, or to simulate an agent's search behaviour on known-null data.

When to use

• simulate a realistic p-hacking session
• measure how likely a search method is to manufacture significance
• calibrate a result from a sequential search
• simulate an agent's search behaviour on known-null data
• measure false-positive rate of a specific procedure
• record the path of a search for audit

When not to use

• exhaustive enumeration of all specifications

Skill page

Source