domains / data / polars-dataframe-operations / polars

polars

Skill card

combined

The copies share a common core (expressions, lazy vs eager, basic I/O, group_by) but diverge significantly in scope and specific guidance. The K-Dense copy is a version-pinned (1.44.2) reference with specific installation commands and detailed mapping strategy explanations. The Vibe-Skills and agentic-awesome-skills copies are nearly identical, focusing on 'in-memory' use cases and providing a slightly different (simpler) explanation of window function mapping strategies. The brycewang-stanford copy is a 'meta-skill' that provides a decision tree and a table of reference files rather than inline code examples, and it includes specific version migration notes (e.g., `apply` to `map_elements`). Because the K-Dense copy has unique version-specific details and the brycewang copy has a unique structural approach (decision trees) and migration notes, no single copy is the 'authentic' procedure that the others merely reproduce. They represent different levels of abstraction and specific constraints.

Repositories

Other copies

Combined skill

---
name: polars
description: High-performance DataFrame library for Python ETL, analytics, and pandas migration. Supports expression-based manipulation, lazy query optimization, parallel execution, and Arrow interoperability.
---

# Polars

## Overview
Polars is a columnar DataFrame library for Python and Rust with Arrow interoperability. Use it for efficient data processing, pandas migration, and data pipeline optimization.

## When to Use
- **In-Memory Performance**: When pandas is too slow but data fits in RAM (1-100GB).
- **ETL Pipelines**: When building transformations that benefit from lazy evaluation and parallel execution.
- **Pandas Migration**: When seeking a faster, more expressive alternative to pandas.
- **Note**: For larger-than-RAM data, consider Dask or Vaex, though Polars supports streaming.

## Installation
Install the current stable release. Pin versions for reproducibility in production.
```bash
uv pip install polars
# Optional extras for specific integrations:
uv pip install "polars[excel,database,fsspec,pandas,numpy]"
```

## Core Concepts

### Expressions
Expressions are the fundamental building blocks. They describe transformations and are lazy until executed in a context (`select`, `with_columns`, `filter`, `group_by`).
- Use `pl.col("column_name")` to reference columns.
- Chain methods to build complex transformations.

### Lazy vs Eager
- **Eager (DataFrame)**: Operations execute immediately. Good for small data or interactive exploration.
- **Lazy (LazyFrame)**: Operations build a query plan optimized before execution. Use for large datasets, complex pipelines, or when performance is critical.
  - Benefits: Predicate pushdown, projection pushdown, parallel execution.

## Common Operations

### Select & Filter
```python
# Select columns
df.select("name", "age")

# Filter rows
df.filter(pl.col("age") > 25)

# Multiple conditions
df.filter(
    pl.col("age") > 25,
    pl.col("city") == "NY"
)
```

### With Columns
Add or modify columns. Computations are parallel.
```python
df.with_columns(
    age_plus_10=pl.col("age") + 10,
    name_upper=pl.col("name").str.to_uppercase()
)
```

### Group By & Aggregations
```python
df.group_by("city").agg(
    pl.col("age").mean().alias("avg_age"),
    pl.len().alias("count")
)
```

### Window Functions
Apply aggregations while preserving row count using `.over()`.
```python
df.with_columns(
    avg_age_by_city=pl.col("age").mean().over("city"),
    rank_in_city=pl.col("salary").rank().over("city")
)
```
**Mapping Strategies:**
- `group_to_rows` (default): Maps results back to rows; scalar aggregates broadcast.
- `explode`: Changes row layout/count; use in `select`.
- `join`: Joins grouped values back as lists.

## Data I/O
Polars supports CSV, Parquet, JSON, Excel, Databases, and Cloud Storage.

**Parquet (Recommended for Performance):**
```python
df = pl.read_parquet("file.parquet")
df.write_parquet("output.parquet")
```

**CSV (Lazy for Large Files):**
```python
lf = pl.scan_csv("file.csv")
result = lf.filter(...).select(...).collect()
```

## Version Notes & Migration
- **1.x Changes**: `apply` renamed to `map_elements`, `groupby` to `group_by`, `melt` to `unpivot`.
- **Types**: `pl.Utf8` is now `pl.String` (Utf8 is an alias).
- **Pandas Interop**: Use `pl.from_pandas(df)` and `df.to_pandas()` for conversion.

## Performance Tips
- Use Lazy evaluation for large data.
- Avoid row iteration; use vectorized expressions.
- Use Parquet for storage and I/O.
- Check query plans with `lf.explain()` to identify bottlenecks.