domains / families / science / scientific-data-preprocessing

scientific-data-preprocessing

separate

⚠️ CRITICAL USER EXPERIENCE-BASED SKILL - ALWAYS CONSULT BEFORE DATA PREPROCESSING ⚠️ Prevents catastrophic errors (88.9% error rate in V1.0 case study) through multi-level feature analysis, data leakage detection, and semantic validation. MANDATORY for: data preprocessing, feature engineering, standardization, normalization, interpolation, missing value handling, feature selection, or ANY data transformation task. Covers grouped time-series, cross-sectional, panel data. Detects: time travel leakage, causal inversion, ID misuse, semantic-numeric fallacies, distribution blindness. User's hard-won lessons from real project failures.

When to use

• Any data cleaning, transformation, or preparation work
• Loading and preparing data for modeling
• Creating training/test splits
• Handling missing values (imputation, deletion)
• Feature scaling/normalization/standardization
• Encoding categorical variables
• Feature engineering or construction
• Feature selection or dimensionality reduction

When not to use

• purely cross-sectional data
• domain-specific feature engineering decisions
• choosing ML models
• distributed/big data infrastructure

Skill page

Source