domains / families / science / batch-processing

batch-processing

separate

Process many sequence files in batch (count, merge, split, convert, summarize) with memory-safe streaming and on-disk indexing using Biopython, pysam, or pyfastx. Use when iterating over a directory of FASTA/FASTQ files, merging or splitting datasets, building random access across many or huge files, or automating per-file operations without exhausting RAM.

When to use

• iterating over a directory of FASTA/FASTQ files
• merging or splitting datasets
• building random access across many or huge files
• automating per-file operations without exhausting RAM
• counting sequences across files
• converting sequence file formats
• summarizing sequence data

When not to use

• loading entire large files into RAM
• using list() on SeqIO.parse for large datasets
• indexing plain gzip files

Skill page

Source