domains / families / research / ray-data

ray-data

separate

Scalable data processing for ML workloads. Streaming execution across CPU/GPU, supports Parquet/CSV/JSON/images. Integrates with Ray Train, PyTorch, TensorFlow. Scales from single machine to 100s of nodes. Use for batch inference, data preprocessing, multi-modal data loading, or distributed ETL pipelines.

When to use

• Processing large datasets (>100GB) for ML training
• Need distributed data preprocessing across cluster
• Building batch inference pipelines
• Loading multi-modal data (images, audio, video)
• Scaling data processing from laptop to cluster
• **Streaming execution**: Process data larger than memory
• **GPU support**: Accelerate transforms with GPUs
• **Framework integration**: PyTorch, TensorFlow, HuggingFace

When not to use

• Small data (<1GB) on single machine
• Tabular data with SQL-like operations
• Enterprise ETL with SQL queries

Skill page

Source