domains / families / research / 03-specification-search

03-specification-search

separate

Run an instrumented walk of a specification universe and recover honest inference from it. Estimates every specification in a design card (or walks it with a realistic search procedure), logs a complete ledger, flags pathological specifications, calibrates the search against an enforced null, runs the Simonsohn joint tests on the whole curve, measures how far the finding sits from the pre-registered analysis, attributes significance to the choices that produced it, and writes a publishable honest report. Use to run a multiverse or specification-curve analysis, to audit a search someone else ran, to compute multiplicity-corrected or null-calibrated p-values for a selected specification, or to demonstrate how far a search can move a result under a known-zero effect.

When to use

• run a multiverse or specification-curve analysis
• audit a search someone else ran
• compute multiplicity-corrected or null-calibrated p-values
• demonstrate how far a search can move a result under a known-zero effect
• estimate every specification in a design card
• walk a grid with a realistic search procedure
• flag pathological specifications
• run Simonsohn joint tests on the whole curve

When not to use

• produce the winner without the ledger

Skill page

Source