domains / families / research / 01-phack-taxonomy

01-phack-taxonomy

separate

Name and classify p-hacking strategies, and quantify what each one does to the false-positive rate. Covers the twelve-strategy compendium of Stefan and Schoenbrodt (2023), thirteen econometrics-specific degrees of freedom (clustering doctrine, fixed-effect structure, RDD bandwidth, kernel and inference mode, IV instrument sets and first-stage screening, staggered-DiD estimator and comparison-group choice, synthetic-control donor pools), the search procedures that turn a strategy into a session, and the two between-stage strategies of Adda, Decker and Ottaviani (2020): selective continuation from a pilot to a confirmatory study, which is not p-hacking until the pilot is pooled, and selective reporting between stages. Use when asked what p-hacking is, which strategy a particular analytical choice corresponds to, how much a given researcher degree of freedom inflates type I error, or to enumerate the ways a specific result could have been obtained.

When to use

• Naming and classifying p-hacking strategies
• Quantifying false-positive rate inflation
• Mapping analytical choices to numbered strategies
• Enumerating ways a specific result could have been obtained
• Simulating strategies to calculate fpr_hacked
• Classifying selective continuation and reporting

When not to use

• Determining if a single isolated choice is p-hacking
• Concluding that high false-positive rates imply individual misconduct

Skill page

Source