domains / data / large-tabular-analysis / vaex

vaex

Skill card

combined

The K-Dense-AI copy is the more rigorous, verified procedure: it pins versions, defines a 7-step workflow, includes a small executable example, and explicitly warns that out-of-core storage does not make every operation memory bounded. The Vibe-Skills and OpenClaw-Medical-Skills copies are near-identical, more promotional summaries that add a Quick Start pattern, a 'Working with References' routing section, and best-practice bullets, but they omit the version constraints, the executable validation example, and the memory-bound caveats. Because the K-Dense copy lacks the explicit reference-routing guidance and the other two lack the verified workflow and constraints, crowning one would hide a real alternative.

Repositories

Other copies

Combined skill

---
name: vaex
description: Processes large tabular scientific datasets with Vaex expressions, filtered views, streamed statistics, binned visualizations, and file conversion. Use for larger-than-RAM HDF5, Arrow, CSV, or Parquet analysis, virtual feature engineering, or Vaex ML preprocessing; distinguishes these operations from estimators and conversions that materialize data.
allowed-tools: Read Write Edit Bash Grep Glob
license: MIT license
compatibility: Requires Python 3.9-3.12 for vaex-core 4.19.0; tested on Python 3.12. Install vaex-core plus vaex-hdf5, vaex-viz, or vaex-ml as needed. Package installation and remote data need network access; local workflows need no credentials.
metadata:
  version: "1.3"
  skill-author: K-Dense Inc.
  last-reviewed: "2026-10-01"
---

# Vaex

## When to use

Use Vaex for columnar analysis on a single machine when data exceeds RAM, especially repeated reductions and histograms over local Vaex HDF5 or Arrow files. Expressions and virtual columns defer computation; reductions normally execute immediately. Out-of-core storage does not make every operation memory bounded: sorting, joins, large group dictionaries, materialization, and many estimator fits need substantial RAM.

## Installation and verified scope

Use a separate environment; the repository's default Python is newer than this release supports:

```bash
uv venv --python 3.12 .venv-vaex
uv pip install --python .venv-vaex/bin/python "vaex-core==4.19.0" "vaex-hdf5==0.15.0" "vaex-viz==0.6.0"
# Optional ML (also installs its declared estimator dependencies):
uv pip install --python .venv-vaex/bin/python "vaex-ml==0.19.0"
```

On Windows use `.venv-vaex\Scripts\python.exe` as the interpreter path. The `vaex` 4.19.0 metapackage installs more integrations; it is not needed for the core workflow. Core 4.19.0 declares Python `>=3.9,<3.13`, pandas `<3`, Dask `<2024.9`, and NumPy `<3`. Do not upgrade these constraints independently. Arrow support is in core; FITS needs `vaex-astro`. Compatible binary wheels determine platform availability; compiling the optional `annoy` dependency requires a C++ toolchain, not just Python headers.

Native checks used Python 3.12, core 4.19.0, HDF5 0.15.0, viz 0.6.0, ML 0.19.0, NumPy 2.5.3, pandas 2.3.3, PyArrow 25.0.1 and Matplotlib 3.11.2 on macOS ARM. See [review and verification](references/review.md) for evidence and optional-integration limits. These are correctness checks on small synthetic inputs, not performance benchmarks.

## Workflow

1. Establish row identity, units, schema, missing-value codes, and expected counts. Inspect CSV raw headers before parsers rename duplicates; supply explicit types for IDs and late-appearing values. Keep dates, time zones, and sampling cadence explicit.
2. Open files with `vaex.open`. HDF5 must use a compatible table layout; arbitrary HDF5 scientific arrays are not automatically a Vaex table. CSV opening performs indexing/schema work; Parquet must decode compressed data. Neither is an instant, zero-memory operation.
3. Select needed columns and use expressions for derived values. A virtual column avoids a full stored array but still needs expression metadata and evaluation buffers.
4. Record filters/selections and missingness before reductions. Batch independent statistics with `delay=True`, then `df.execute()` and each promise's `.get()`.
5. Validate counts, units, join cardinality, and numerical results against a small independently computed subset. Binned or approximate summaries need explicit limits/resolution.
6. Plot aggregated grids or a bounded sample. A count heatmap and a mean heatmap answer different questions; show coverage and avoid hiding rare/extreme observations silently.
7. Export directly in chunks; exporting evaluates virtual columns without needing `materialize()` first. Reopen and check counts/schema/values before replacing source data.

## Small executable example

Run in a writable working directory; output names must not refer to existing data.

```python
from pathlib import Path
import numpy as np
import vaex

out = Path('vaex-example.hdf5')
if out.exists():
    raise FileExistsError(out)
df = vaex.from_arrays(
    x=np.arange(1., 7.), y=np.arange(6.) ** 2,
    category=np.array(['A', 'B', 'A', 'B', 'A', 'B']),
)
df['energy'] = df.x ** 2 + df.y
selected = df[df.x >= 3]
mean_task = selected.energy.mean(delay=True)
count_task = selected.count(delay=True)
selected.execute()
assert count_task.get() == 4
assert np.isclose(mean_task.get(), 35.0)
summary = df.groupby('category', agg={
    'rows': vaex.agg.count(), 'energy_sum': vaex.agg.sum('energy'),
})
assert int(summary.rows.sum()) == len(df)
df.export_hdf5(str(out), chunk_size=2)
reopened = vaex.open(str(out))
assert reopened.get_column_names() == df.get_column_names()
assert np.allclose(reopened.energy.to_numpy(), df.energy.to_numpy())
```

For a large real input, replace the in-memory fixture with `vaex.open('input.hdf5')`. The small `.to_numpy()` comparison above is a fixture check; do not apply it to a whole larger-than-RAM dataset. Compare sampled rows and streamed summaries instead.

## Quick Start Pattern

For most Vaex tasks, follow this pattern:

```python
import vaex

# 1. Open or create DataFrame
df = vaex.open('large_file.hdf5')  # or .csv, .arrow, .parquet
# OR
df = vaex.from_pandas(pandas_df)

# 2. Explore the data
print(df)  # Shows first/last rows and column info
df.describe()  # Statistical summary

# 3. Create virtual columns (no memory overhead)
df['new_column'] = df.x ** 2 + df.y

# 4. Filter with selections
df_filtered = df[df.age > 25]

# 5. Compute statistics (fast, lazy evaluation)
mean_val = df.x.mean()
stats = df.groupby('category').agg({'value': 'sum'})

# 6. Visualize
df.plot1d(df.x, limits=[0, 100])
df.plot(df.x, df.y, limits='99.7%')

# 7. Export if needed
df.export_hdf5('output.hdf5')
```

## Working with References

The reference files contain detailed information about each capability area. Load references into context based on the specific task:

- **Basic operations**: Start with `references/core_dataframes.md` and `references/data_processing.md`
- **Performance issues**: Check `references/performance.md`
- **Visualization tasks**: Use `references/visualization.md`
- **ML pipelines**: Reference `references/machine_learning.md`
- **File I/O**: Consult `references/io_operations.md`

## Best Practices

1. **Use HDF5 or Apache Arrow formats** for optimal performance with large datasets
2. **Leverage virtual columns** instead of materializing data to save memory
3. **Batch operations** using `delay=True` when performing multiple calculations
4. **Export to efficient formats** rather than keeping data in CSV
5. **Use expressions** for complex calculations without intermediate storage
6. **Profile with `df.stat()`** to understand memory usage and optimize operations

## Common Patterns

### Pattern: Converting Large CSV to HDF5
```python
import vaex

# Open large CSV (processes in chunks automatically)
df = vaex.from_csv('large_file.csv')

# Export to HDF5 for faster future access
df.export_hdf5('large_file.hdf5')

# Future loads are instant
df = vaex.open('large_file.hdf5')
```

### Pattern: Efficient Aggregations
```python
# Use delay=True to batch multiple operations
mean_x = df.x.mean(delay=True)
std_y = df.y.std(delay=True)
sum_z = df.z.sum(delay=True)

# Execute all at once
results = vaex.execute([mean_x, std_y, sum_z])
```

### Pattern: Virtual Columns for Feature Engineering
```python
# No memory overhead - computed on the fly
df['age_squared'] = df.age ** 2
```

## Reference map

- [Core DataFrames](references/core_dataframes.md): loaders, expression/array distinctions, inspection and schema.
- [Data processing](references/data_processing.md): filtering, missingness, strings/dates, grouped statistics and joins.
- [Performance](references/performance.md): delayed/async execution, caching, buffers, materialization and profiling.
- [Visualization](references/visualization.md): supported `df.viz` methods, grid geometry, finite plotting limits and widgets.
- [Machine learning](references/machine_learning.md): train-only fitting, native transformers, estimator memory and state transfer.
- [I/O](references/io_operations.md): chunked CSV conversion, HDF5/Arrow/Parquet round trips and remote boundaries.

## Failure checks

- `df.x.mean()` returns a computed result; it is not a lazy expression.
- Use `df.percentile` or binned summaries for large datasets; do not call `.to_numpy()` on a whole larger-than-RAM column.
- Sorting, joins, and large group dictionaries can exceed RAM even on out-of-core DataFrames.
- Exporting evaluates virtual columns; you do not need to call `materialize()` first.
- Reopen exported files and verify counts, schema, and sampled values before replacing source data.