domains / science / ena-database-access / ena-database

ena-database

Skill card

combined

The copies overlap heavily on ENA data types, Portal vs Browser API roles, formats, and rate limits, but they disagree on operational details and each has material steps the others lack. The aipoch copy adds explicit dependencies, a User-Agent, a complete runnable script, and a concrete exponential-backoff implementation. The Vibe-Skills and OpenClaw copies add browser-based search modes (free text, BLAST, Rulespace), cross-reference and CRAM registry endpoints, and a bulk-download pattern. Crowning one copy would hide the runnable pipeline guidance in one and the broader endpoint/search coverage in the others.

Repositories

Other copies

Combined skill

---
name: ena-database
description: "Access European Nucleotide Archive via API/FTP. Retrieve DNA/RNA sequences, raw reads (FASTQ), genome assemblies by accession, for genomics and bioinformatics pipelines. Supports multiple formats."
---

# ENA Database

## Overview

The European Nucleotide Archive (ENA) is a public repository for nucleotide sequence data and associated metadata. Use REST APIs and FTP/Aspera to access DNA/RNA sequences, raw reads, genome assemblies, and functional annotations for genomics and bioinformatics pipelines.

## When to Use

Use this skill when you need to:

- Retrieve nucleotide sequences or raw sequencing reads by accession
- Search for samples, studies, runs, experiments, or assemblies by metadata criteria
- Download FASTQ files or genome assemblies for analysis
- Query taxonomic information for organisms
- Access sequence annotations and functional data
- Integrate ENA data into bioinformatics pipelines
- Perform cross-reference searches to related databases
- Bulk download datasets via FTP or Aspera

## Dependencies

- Python >=3.9
- requests >=2.31.0
- Optional: lxml >=4.9.0 for XML parsing when using the Browser API

## Data Types and Structure

ENA organizes data into hierarchical object types:

- **Studies/Projects**: Group related data and control release dates. Studies are the primary unit for citing archived data.
- **Samples**: Units of biomaterial from which sequencing libraries were produced.
- **Raw Reads**: Consist of Experiments (sequencing methods, library preparation, instrument details) and Runs (references to data files from a single sequencing run).
- **Assemblies**: Genome, transcriptome, metagenome, or metatranscriptome assemblies at various completion levels.
- **Sequences**: Assembled and annotated sequences in the EMBL Nucleotide Sequence Database.
- **Analyses**: Results from computational analyses of sequence data.
- **Taxonomy Records**: Taxonomic information including lineage and rank.

## Programmatic Access

Consult `references/api_reference.md` for detailed endpoint documentation.

### Key APIs

- **ENA Portal API** - Advanced search across all ENA data types. Use for complex queries and metadata searches. Documentation: https://www.ebi.ac.uk/ena/portal/api/doc
- **ENA Browser API** - Direct retrieval of records and metadata by accession. Returns XML. Documentation: https://www.ebi.ac.uk/ena/browser/api/doc
- **ENA Taxonomy REST API** - Query taxonomic information including lineage and rank.
- **ENA Cross Reference Service** - Access related records from external databases. Endpoint: https://www.ebi.ac.uk/ena/xref/rest/
- **CRAM Reference Registry** - Retrieve reference sequences by MD5 or SHA1 checksums. Endpoint: https://www.ebi.ac.uk/ena/cram/

### Rate Limiting

All APIs have a rate limit of 50 requests per second. Exceeding this returns HTTP 429 (Too Many Requests). Implement exponential backoff when receiving HTTP 429 responses. Batch requests when possible and use bulk download tools for large datasets instead of iterating API calls.

## Searching and Retrieving Data

### Browser-Based Search

- Free text search across all fields
- Sequence similarity search (BLAST integration)
- Cross-reference search to find related records
- Advanced search with Rulespace query builder

### Programmatic Queries

- Use Portal API for advanced searches at scale
- Filter by data type, date range, taxonomy, or metadata fields
- Download results as tabulated metadata summaries or XML records

### Data Retrieval Formats

**Metadata Formats:**
- XML (native ENA format)
- JSON (via Portal API)
- TSV/CSV (tabulated summaries)

**Sequence Data:**
- FASTQ (raw reads)
- BAM/CRAM (aligned reads)
- FASTA (assembled sequences)
- EMBL flat file format (annotated sequences)

**Download Methods:**
- Direct API download (small files)
- FTP for bulk data transfer
- Aspera for high-speed transfer of large datasets
- enaBrowserTools command-line utility for bulk downloads

## Example Usage

The following script searches ENA for runs in a study via the Portal API, fetches one run's record via the Browser API, and retrieves taxonomy lineage via the Taxonomy REST API.

```python
#!/usr/bin/env python3
import sys
import time
import requests

PORTAL_SEARCH = "https://www.ebi.ac.uk/ena/portal/api/search"
BROWSER_XML = "https://www.ebi.ac.uk/ena/browser/api/xml"
TAXONOMY = "https://www.ebi.ac.uk/ena/taxonomy/rest"

SESSION = requests.Session()
SESSION.headers.update({"User-Agent": "ena-database-skill/1.0"})

def get_with_backoff(url, params=None, max_retries=6, timeout=30):
    delay = 1.0
    for attempt in range(max_retries):
        r = SESSION.get(url, params=params, timeout=timeout)
        if r.status_code != 429:
            r.raise_for_status()
            return r
        time.sleep(delay)
        delay *= 2
    r.raise_for_status()

def search_runs_by_study(study_accession, limit=5):
    params = {
        "result": "read_run",
        "query": f"study_accession={study_accession}",
        "format": "json",
        "limit": limit,
        "fields": "run_accession,study_accession,sample_accession,experiment_accession,tax_id,scientific_name,fastq_ftp"
    }
    r = get_with_backoff(PORTAL_SEARCH, params=params)
    return r.json()

def fetch_run_xml(run_accession):
    url = f"{BROWSER_XML}/{run_accession}"
    r = get_with_backoff(url)
    return r.text

def fetch_taxonomy_lineage(tax_id):
    url = f"{TAXONOMY}/tax-id/{tax_id}"
    r = get_with_backoff(url)
    return r.json()

def main():
    if len(sys.argv) < 2:
        print("Usage: python ena_example.py <STUDY_ACCESSION>  (e.g., PRJEB1234)", file=sys.stderr)
        sys.exit(2)

    study = sys.argv[1]
    runs = search_runs_by_study(study_accession=study, limit=5)

    if not runs:
        print(f"No runs found for study {study}")
        return

    print(f"Found {len(runs)} runs for study {study}")
    first = runs[0]
    run_acc = first.get("run_accession")
    tax_id = first.get("tax_id")

    print("\nFirst run summary (Portal API JSON):")
    for k in ["run_accession", "sample_accession", "experiment_accession", "scientific_name", "tax_id", "fastq_ftp"]:
        print(f"  {k}: {first.get(k)}")

    if run_acc:
        xml = fetch_run_xml(run_acc)
        print("\nBrowser API XML (first 600 chars):")
        print(xml[:600])

    if tax_id:
        tax = fetch_taxonomy_lineage(tax_id)
        print("\nTaxonomy lineage (ENA Taxonomy REST API):")
        rec = tax[0] if isinstance(tax, list) and tax else tax
        print(f"  scientificName: {rec.get('scientificName')}")
        print(f"  rank: {rec.get('rank')}")
        print(f"  lineage: {rec.get('lineage')}")

if __name__ == "__main__":
    main()
```

Run:

```bash
python ena_example.py PRJEB1234
```

## Common Use Cases

**Retrieve raw sequencing reads by accession:**
```python
accession = "ERR123456"
url = f"https://www.ebi.ac.uk/ena/browser/api/xml/{accession}"
```

**Search for all samples in a study:**
```python
study_id = "PRJNA123456"
url = f"https://www.ebi.ac.uk/ena/portal/api/search?result=sample&query=study_accession={study_id}&format=tsv"
```

**Find assemblies for a specific organism:**
```python
organism = "Escherichia coli"
url = f"https://www.ebi.ac.uk/ena/portal/api/search?result=assembly&query=tax_tree({organism})&format=json"
```

**Get taxonomic lineage:**
```python
taxon_id = "562"
url = f"https://www.ebi.ac.uk/ena/taxonomy/rest/tax-id/{taxon_id}"
```

## Integration with Analysis Pipelines

**Bulk Download Pattern:**
1. Search for accessions matching criteria using Portal API
2. Extract file URLs from search results
3. Download files via FTP or using enaBrowserTools
4. Process downloaded data in pipeline

**BLAST Integration:**
Integrate with EBI's NCBI BLAST service (REST/SOAP API) for sequence similarity searches against ENA sequences.

## Best Practices

- Implement exponential backoff when receiving HTTP 429 responses
- Batch requests when possible to stay within 50 req/sec limit
- Use bulk download tools for large datasets instead of iterating API calls
- Always cite using Study/Project accessions when referencing archived data

## Conflicts

- aipoch/medical-research-skills specifies Python >=3.9, requests >=2.31.0, optional lxml >=4.9.0, and a User-Agent header; the other copies do not specify dependencies or User-Agent
- aipoch/medical-research-skills provides a concrete exponential-backoff implementation with max_retries=6 and timeout=30; the other copies only recommend exponential backoff without parameters
- Vibe-Skills and OpenClaw list browser-based search modes (free text, BLAST, Rulespace) and cross-reference/CRAM endpoints; aipoch omits these in the visible text
- Vibe-Skills and OpenClaw include a bulk-download pattern and BLAST integration section; aipoch focuses on a runnable search-fetch-taxonomy script