Bioinformatics Shop in a Box: A Reference for Team Leads

Foundations

How omics data move from sample to result

Every genomics project follows roughly the same path. A biological sample is prepared into a sequencing library, a sequencer reads it, and software turns those reads into files your team stores, processes, and analyzes. Most confusion for new leaders comes from not knowing which file belongs to which step, so the diagram below is the map for the rest of this resource. Colors follow the convention the Integrative Genomics Viewer (IGV) uses for the four DNA bases, and here they mark the kind of work: green for the wet lab and sequencer, blue for alignment, amber for variant calling, and red for analysis and interpretation.

Primary, secondary, and tertiary analysis

Vendors, platforms, and job descriptions divide the work into three phases, and knowing the terms helps when you read a quote from a sequencing provider or a platform contract.

Reference genomes and coordinates

A reference genome is the agreed assembly that reads are placed on. Two human builds are in wide use. GRCh38 (also called hg38) is the current default for new work; GRCh37 (hg19) still appears in older studies, older annotation sources, and many published variant lists. A newer complete assembly, T2T-CHM13, fills gaps both builds leave open and is used increasingly in research pipelines. The same variant has different coordinates on different builds, so every CRAM, VCF, BED file, and phenopacket must record its build, and files from different builds must never be combined without conversion (see liftover).

• Chromosome naming differs by source. UCSC-style files write chr1; Ensembl and many GRCh37 files write 1. Tools will silently find no overlap between files that disagree, so check naming before any merge or intersection.
• Coordinate systems differ by format. VCF and SAM positions start at 1. BED files start at 0 and exclude the end position. An off-by-one error between them is one of the most common bugs in genomics code.
• The build name alone is not enough. Pipelines usually use an "analysis set" of GRCh38 that masks or adds certain sequences (decoys, alternate contigs). Two teams can both say "GRCh38" and still have incompatible CRAM files, because CRAM decoding needs the identical reference FASTA. Record the file name and its checksum.

How this resource is organized

The jump bar at the top shows one section at a time and groups sections into stages. Foundations covers the science and the file formats. Pipelines and standards covers processing, joint calling, combining cohorts, and the Global Alliance for Genomics and Health (GA4GH) standards, including Phenopackets. Infrastructure covers cloud storage, computing, file transfer, GitHub, containers, and a command reference. Running the team covers onboarding and the generators, which produce onboarding plans, IT requests, protocols, harmonization plans, notebook entries, and transfer manifests that you can edit, copy, download, or print. Reference holds the glossary and the list of web sites and specifications.

Foundations

Assays and sequencing technologies

The assay decides what questions the data can answer, how large the files are, and which pipeline applies. The table compares the assays a genomics program is likely to encounter.

Short-read and long-read sequencing

Short-read sequencing (Illumina and similar platforms) produces reads of roughly 100 to 300 bases. Most human studies use paired-end reads, where both ends of each DNA fragment are read, and the distance between the pair (the insert size) helps place reads and detect rearrangements. Short reads are accurate and inexpensive, and nearly every standard pipeline assumes them. Their weakness is repetitive or duplicated regions, where a short read could fit in several places.

Long-read sequencing (PacBio HiFi and Oxford Nanopore) produces reads from thousands to hundreds of thousands of bases. Long reads span repeats and structural rearrangements, separate the two parental copies of a region (phasing), and can report methylation directly. They use different aligners (such as minimap2 or pbmm2) and callers, and cost more per sample, though the gap has narrowed.

Depth, coverage, and read quality

Depth is the number of reads overlapping a position. "30x WGS" means the genome is covered by about 30 reads on average; some positions will have far fewer. Exomes are often sequenced to 100x or more on target because capture is uneven. Every base in a FASTQ has a Phred quality score, where Q30 means an estimated one-in-a-thousand chance the base is wrong. Coverage summaries from tools such as mosdepth report the fraction of the genome or target covered at 10x, 20x, and so on, and those fractions are more informative than the average.

Library preparation, barcodes, and runs

Before sequencing, DNA or RNA is fragmented and ligated to adapters. Each sample gets a short index sequence (a barcode) so many samples can share one flow cell; demultiplexing uses the barcodes to split the run back into per-sample FASTQ files. PCR-free library preparation reduces duplicate reads and bias and is common for WGS. Samples sequenced on different runs, lanes, or kits can differ systematically, which is why pipelines record a read group for every set of reads and why run and kit belong in your sample manifest.

RNA-seq design choices

• Poly-A selection or ribosomal RNA depletion. Poly-A selection captures mature messenger RNA; ribosomal depletion also keeps non-polyadenylated RNA and tolerates degraded samples better.
• Stranded or unstranded. A stranded library records which DNA strand a transcript came from and is the usual default now. The quantification step must be told the strandedness, and a wrong setting quietly cuts counts.
• Depth. Bulk RNA-seq depth is described in millions of reads per sample rather than in "x" coverage; differential expression studies commonly use tens of millions.
• Units. Raw counts feed statistical models. TPM and similar normalized values help with display and comparison within a sample but are not inputs for tools such as DESeq2.
• Single-cell specifics. Each read has a cell barcode and a unique molecular identifier (UMI). The platform software (for example 10x Genomics Cell Ranger, or STARsolo) produces a cell-by-gene matrix that is then analyzed in Scanpy (Python) or Seurat (R).

Foundations

File formats and what each one holds

Each step in the pipeline writes a different file, and each file answers a different question. The short version: FASTQ holds raw reads, BAM and CRAM hold the same reads placed on the genome, a gVCF holds one sample's evidence at every position, and a VCF holds only the positions where someone differs from the reference. Most of the other formats are indexes, reference files, or tables derived from these.

Approximate file sizes for one 30x human genome

Sizes drive storage cost, transfer time, and which files a program keeps. The bars show rough planning ranges for one short-read, 30x whole genome as of September 2026. Actual sizes depend on depth, read length, quality score binning, and compression settings; to reproduce the figures for your own data, run ls -lh on local files or gcloud storage du -s (Google Cloud) or aws s3 ls --summarize --human-readable (Amazon Web Services) on a bucket path.

A joint-called VCF grows with the number of samples and variant sites rather than with depth; for large cohorts, teams often switch to formats built for scale, such as sparse matrix stores in Hail, or split the file by chromosome. An exome at 100x on target is roughly a tenth of these sizes. Many programs keep CRAM plus gVCF long term, move FASTQ to archival storage or delete it after confirming the CRAM can reproduce the reads, and never keep BAM once CRAM exists.

VCF, gVCF, joint VCF, and CRAM compared

Anatomy of a VCF

A VCF is a tab-separated text file, usually compressed with bgzip and indexed. Header lines start with ## and define the reference, contigs, and every field. The column header starts with #CHROM. Each data row is one site. The example below uses illustrative values.

##fileformat=VCFv4.2
##reference=GRCh38_full_analysis_set_plus_decoy_hla.fa
##contig=<ID=chr1,length=248956422>
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=AD,Number=R,Type=Integer,Description="Allelic depths for the ref and alt alleles">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read depth">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype quality">
#CHROM  POS      ID  REF  ALT  QUAL  FILTER  INFO   FORMAT       SAMPLE_0042
chr1    1014143  .   C    T    812   PASS    DP=41  GT:AD:DP:GQ  0/1:20,21:41:99

• CHROM, POS, REF, ALT locate the variant and give the reference and alternate alleles. Insertions and deletions include one anchoring base before the change.
• QUAL is the caller's confidence that a variant exists; FILTER is PASS or the names of failed filters.
• INFO holds site-level annotations such as total depth, allele frequency in the cohort, or annotations added later.
• FORMAT and sample columns hold per-sample values. GT is the genotype: 0/0 reference, 0/1 heterozygous, 1/1 homozygous alternate, ./. missing, and a pipe as in 0|1 means the alleles are phased. AD is reads supporting each allele, DP is depth, and GQ is confidence in the genotype.

Anatomy of a gVCF

A gVCF adds rows for stretches where the sample matches the reference. Each reference block uses the symbolic allele <NON_REF> (GATK) or <*> (DeepVariant, bcftools) and an END tag, with the lowest depth and quality inside the block. Joint genotyping reads these blocks to tell "confident reference" apart from "no data."

#CHROM  POS      ID  REF  ALT          QUAL   FILTER  INFO         FORMAT              SAMPLE_0042
chr1    1014100  .   G    <NON_REF>    .      .       END=1014142  GT:DP:GQ:MIN_DP     0/0:38:99:35
chr1    1014143  .   C    T,<NON_REF>  812.6  .       DP=41        GT:AD:DP:GQ         0/1:20,21,0:41:99
chr1    1014144  .   A    <NON_REF>    .      .       END=1014390  GT:DP:GQ:MIN_DP     0/0:12:27:4

The last block shows lower depth and quality. A downstream analysis might treat genotypes there as unreliable even though the sample is called reference.

Formats a team handles

Index files travel with their data

Indexes let tools jump to a region without reading the whole file, which is what makes it possible to look at one gene inside a 30 GB CRAM or to stream a region from cloud storage. An index is useless without its data file and must be regenerated if the data file changes. When transferring or sharing, move .cram with .crai, .bam with .bai, and .vcf.gz with .tbi or .csi, and move or document the reference FASTA used for CRAM.

Pipelines and standards

Analysis pipelines and workflow languages

A pipeline is a fixed sequence of tools with fixed versions, parameters, and reference files. Running every sample through the same pipeline is what makes samples comparable; changing a tool version partway through a study creates a batch effect. Most teams start from a published, maintained pipeline rather than writing one, and spend their effort on configuration, validation, and documentation.

Germline short-variant pipeline for DNA

This is the workflow behind most rare disease, natural history, and cohort genomics. It finds single-nucleotide variants (SNVs) and small insertions and deletions (indels) inherited in the germline.

1. Read quality control. FastQC reports per-base quality, adapter content, and duplication for each FASTQ; MultiQC collects those reports across all samples into one page. Fastp can trim adapters and low-quality bases if needed.
2. Alignment. BWA-MEM2 places each read pair on the reference. Every alignment needs a read group tag naming the sample, library, and sequencing lane, because later tools use it to model errors and to identify the sample.
3. Sorting and duplicate marking. Reads are sorted by position, and reads that come from the same original fragment (PCR or optical duplicates) are flagged so they do not count as independent evidence.
4. Base quality score recalibration (BQSR). GATK adjusts the sequencer's quality scores using known variant sites. DeepVariant documentation reports that it does not need this step.
5. CRAM output and alignment QC. Tools such as mosdepth (coverage), VerifyBamID2 (sample contamination), Picard metrics, and samtools stats generate the numbers used for pass or fail decisions. Checking genetic sex against the reported sex catches sample swaps.
6. Per-sample calling in gVCF mode. GATK HaplotypeCaller with -ERC GVCF, or DeepVariant with gVCF output, writes one gVCF per sample.
7. Joint genotyping. The gVCFs for the cohort are combined and genotyped together. The next section explains why and how.
8. Filtering. GATK's Variant Quality Score Recalibration (VQSR) learns what good variants look like from known sites; newer GATK releases add machine-learning filtering tools, and small cohorts often use fixed "hard filter" thresholds instead. Check current GATK documentation for the recommended method for your cohort size.
9. Normalization and annotation. bcftools norm splits multiallelic sites and left-aligns indels so the same variant is always written the same way. Annotation with Ensembl VEP, SnpEff, or ANNOVAR adds gene, transcript, predicted consequence, HGVS names, population frequency from gnomAD, ClinVar assertions, and scores such as CADD, REVEL, and SpliceAI.
10. Prioritization and analysis. Exomiser ranks variants using the individual's HPO terms and inheritance mode; family data allow segregation checks; cohort data allow association and burden tests.

Other DNA variant types

• Structural variants and copy number. Deletions, duplications, inversions, and translocations larger than about 50 bases need separate callers: Manta, DELLY, and the GATK-SV pipeline for short-read genomes; ExomeDepth or GATK gCNV for exome copy number; Sniffles and pbsv for long reads.
• Repeat expansions. ExpansionHunter estimates repeat lengths from short-read genomes, including the C9orf72 hexanucleotide repeat studied in ALS research and repeats in several neurological conditions. Long reads measure these more directly.
• Mitochondrial DNA. Mitochondrial variants can be present in only a fraction of copies (heteroplasmy) and need a mitochondria-aware mode, such as GATK's mitochondrial pipeline.
• Somatic variants. Cancer and mosaicism studies compare tumor and normal samples or look for low-fraction variants, using tools such as GATK Mutect2. The pipeline and filters differ substantially from germline calling.

Bulk RNA-seq pipeline

Differential expression tools model raw counts, estimate how variable each gene is, and test for group differences while adjusting for covariates such as sex, age, batch, and RNA quality. Batch, tissue source, and time from collection to freezing often explain more variation than the biology under study, so they belong in the sample manifest from the start. For single-cell data, analysts filter low-quality cells, normalize, cluster, annotate cell types, and usually test differences by summing counts per sample and cell type ("pseudobulk") so that individuals, not cells, are the unit of analysis.

Workflow languages and engines

A workflow language describes the steps, their inputs and outputs, and the container each runs in. An engine reads that description and runs the steps on a laptop, a cluster, or the cloud, retries failures, runs samples in parallel, and records what ran. Using one is what separates a pipeline from a folder of scripts.

Validating a pipeline

Before trusting a pipeline, and whenever a version changes, run it on reference samples with known answers. The Genome in a Bottle (GIAB) consortium at NIST publishes benchmark variant calls and high-confidence regions for well-studied samples such as HG001 (NA12878) and the HG002 trio. Comparing your calls with hap.py or rtg vcfeval reports precision and recall for SNVs and indels. Keep the report with the pipeline release so anyone can see what performance the version showed.

An example batch job on a cluster

On a shared cluster that uses the Slurm scheduler, a job array can run one sample per task. Workflow engines generate jobs like this automatically; the script shows what happens underneath.

#!/bin/bash
#SBATCH --job-name=hc_gvcf
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --array=1-40
#SBATCH --output=logs/%x_%A_%a.out
set -euo pipefail

# One sample ID per line in samples.txt
SAMPLE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" samples.txt)

apptainer exec containers/gatk.sif gatk --java-options "-Xmx12g" HaplotypeCaller \
  -R ref/GRCh38_analysis_set.fa \
  -I cram/${SAMPLE}.cram \
  -O gvcf/${SAMPLE}.g.vcf.gz \
  -ERC GVCF

Pipelines and standards

Joint calling and combining cohorts

The file people often call a "joint VCF" is more precisely a joint-called VCF, also called a multi-sample VCF, cohort VCF, or project-level VCF (pVCF). The process that produces it is called joint genotyping or joint calling. It takes one gVCF per sample and produces one file with a genotype for every sample at every site where anyone in the cohort has a variant.

Why merging plain VCFs is not the same thing

If you merge single-sample VCFs with a tool such as bcftools merge, a sample with no row at a site is either written as missing or, with some options, assumed to match the reference. Neither is reliably right, because a single-sample VCF does not say whether a position was well covered. The table shows the difference for the chr7 site in the diagram.

Joint genotyping also improves sensitivity at sites where one sample has weak evidence that others confirm, produces a consistent list of sites across the cohort, and allows cohort-level filtering methods such as VQSR, which need many samples to learn from.

Adding samples later and the N+1 problem

When new samples arrive, a joint-called VCF cannot simply be appended to. The N+1 problem is the cost of re-genotyping the whole cohort each time. The gVCF approach keeps that cost manageable: new samples are called once into gVCFs, and only the joint genotyping step reruns. GenomicsDB workspaces can be extended incrementally, and very large programs use sparse representations such as Hail's variant data set format. Plan cohort "freezes," named and dated versions of the joint call set, so that analyses cite a fixed version.

Combining data sets from different studies

Combining two natural history studies, or a study and a registry, is mostly a question of what each side can share. The decision diagram shows the usual paths, from strongest to weakest.

Steps for combining two natural history studies

1. Confirm permission. Compare consent language, data use agreements (DUAs), and any data use conditions coded with the GA4GH Data Use Ontology (DUO). Confirm that pooling, the intended research use, and the destination environment are allowed, and that each institution's IRB (institutional review board) has approved the combined analysis and lists the people who will access the data as approved study personnel.
2. Inventory the genomic data. For each study record the assay, capture kit and target BED for exomes, read length, sequencer, mean depth, reference build and exact FASTA, pipeline and tool versions, and which file types exist (FASTQ, CRAM, gVCF, VCF).
3. Choose the harmonization path using the diagram above, and write the choice and its reasons into the protocol.
4. Bring everything to one reference build. Realigning from reads is preferred. If only VCFs exist, lift them over with Picard LiftoverVcf, CrossMap, or the bcftools liftover plugin, and report how many variants failed to convert.
5. Process or joint genotype all samples together where the data allow, and freeze a named, dated call set.
6. Restrict to comparable regions. For exomes, intersect the capture targets of every kit involved; for all data, consider limiting analyses to regions with adequate coverage in every study.
7. Check samples across the combined set. Look for duplicates and relatives across studies with somalier, KING, or peddy, because the same participant may be enrolled in more than one study, especially in rare disease cohorts. Compare genetic sex with recorded sex, check contamination, and estimate genetic ancestry with principal components against a reference panel such as 1000 Genomes or HGDP.
8. Look for batch effects. Plot principal components colored by study, compare allele frequencies of common variants between studies, and compare per-sample metrics such as the Ti/Tv ratio, heterozygous-to-homozygous ratio, and call rate. Keep study or sequencing batch as a covariate.
9. Harmonize phenotypes. Map clinical features to HPO terms and diagnoses to MONDO or OMIM, unify units and age representations, reconcile visit schedules, and document differences in instrument versions or scoring rules. Phenopackets are a practical exchange format for case-level data; longitudinal clinical data are often harmonized in the OMOP Common Data Model.
10. Maintain an identifier crosswalk. Keep one controlled table linking study participant IDs, sample IDs in VCF headers, and file locations, stored separately from the genomic data with restricted access.
11. Analyze. Pooled individual-level analysis (mega-analysis) with study as a covariate and mixed models such as REGENIE or SAIGE to handle relatedness; gene-level burden and SKAT-O tests for rare variants; meta-analysis with METAL when data cannot be pooled; and genotype-stratified progression models for natural history questions.
12. Record every decision in a harmonization log: versions, dropped samples and why, liftover losses, and which covariates were used. The harmonization plan generator in this resource produces a starting document.

Pipelines and standards

GA4GH standards and Phenopackets

The Global Alliance for Genomics and Health (GA4GH) is an international standards body for genomic and health data. It maintains the file specifications your team already uses (SAM, BAM, CRAM, and VCF are maintained under the GA4GH umbrella by the htslib community), along with standards for describing patients, finding data, running analyses in the cloud, and controlling access. Knowing which standard covers which job helps when a platform, consortium, or funder says it "supports GA4GH."

What a phenopacket holds

A phenopacket is a structured JSON or YAML document for one individual (a Family or Cohort message groups several). Its fields use ontologies rather than free text so that computers can compare cases:

• subject: the individual's identifier, sex, age at last encounter, and vital status.
• phenotypicFeatures: Human Phenotype Ontology (HPO) terms, each with optional onset, severity, modifiers, and an excluded flag for features that were looked for and found absent. Recording absent features is one of the most useful things a phenopacket does.
• diseases: diagnoses coded with MONDO, OMIM, or Orphanet identifiers, with onset.
• measurements, biosamples, medicalActions: lab values (often LOINC-coded), sample descriptions, and treatments.
• interpretations: a diagnosis and the genomic interpretations supporting it, including each variant's VCF-style coordinates, HGVS expressions, gene, and zygosity.
• files: links to genomic files such as a VCF or CRAM, with the build and a mapping from the phenopacket's subject ID to the sample name inside the file.
• metaData: who created it, when, the schema version, and the exact ontology versions used.

How a phenopacket connects to a VCF

{
  "id": "study-a-P0042",
  "subject": {
    "id": "P0042",
    "sex": "FEMALE",
    "timeAtLastEncounter": { "age": { "iso8601duration": "P14Y" } }
  },
  "phenotypicFeatures": [
    { "type": { "id": "HP:0001250", "label": "Seizure" },
      "onset": { "age": { "iso8601duration": "P4M" } } },
    { "type": { "id": "HP:0001263", "label": "Global developmental delay" } },
    { "type": { "id": "HP:0000252", "label": "Microcephaly" }, "excluded": true }
  ],
  "diseases": [
    { "term": { "id": "MONDO:0005027", "label": "epilepsy" } }
  ],
  "interpretations": [
    { "id": "interpretation-1",
      "progressStatus": "SOLVED",
      "diagnosis": {
        "disease": { "id": "MONDO:0005027", "label": "epilepsy" },
        "genomicInterpretations": [
          { "subjectOrBiosampleId": "P0042",
            "interpretationStatus": "CAUSATIVE",
            "variantInterpretation": {
              "variationDescriptor": {
                "id": "variant-1",
                "vcfRecord": { "genomeAssembly": "GRCh38", "chrom": "chr1",
                               "pos": 1014143, "ref": "C", "alt": "T" },
                "allelicState": { "id": "GENO:0000135", "label": "heterozygous" }
              } } } ] } }
  ],
  "files": [
    { "uri": "gs://your-study-bucket/joint/cohort.2026-09.vcf.gz",
      "individualToFileIdentifiers": { "P0042": "SAMPLE_0042" },
      "fileAttributes": { "genomeAssembly": "GRCh38", "fileFormat": "vcf",
                          "description": "Joint-called cohort VCF, September 2026 freeze" } }
  ],
  "metaData": {
    "created": "2026-09-18T00:00:00Z",
    "createdBy": "study-a-data-team",
    "phenopacketSchemaVersion": "2.0",
    "resources": [
      { "id": "hp", "name": "human phenotype ontology", "namespacePrefix": "HP",
        "url": "http://purl.obolibrary.org/obo/hp.owl", "version": "record the release used",
        "iriPrefix": "http://purl.obolibrary.org/obo/HP_" }
    ]
  }
}

How phenopackets and genomic files are analyzed together

• Individual prioritization. Exomiser takes a phenopacket (for HPO terms and pedigree information) and a VCF, filters variants by frequency and predicted effect, and ranks genes by how well their known disease phenotypes match the individual's HPO profile.
• Cohort genotype and phenotype comparisons. Collections of phenopackets can be loaded into tables to compare feature frequencies between genotype groups, for example missense versus truncating variants in one gene. The Monarch Initiative's phenopacket-store and GPSEA tooling are examples of this approach.
• Matching and discovery. Structured HPO profiles make it possible to find similar cases across registries and to answer discovery queries through Beacon networks.
• Linking to longitudinal data. A phenopacket is a case summary at a point in time. Natural history analyses usually also need repeated visits, which many programs hold in a registry or in the OMOP Common Data Model, linked through the same identifier crosswalk used for genomic samples.

Infrastructure

Cloud storage, compute, and file transfer

Genomic files are too large to move casually, so the most important infrastructure principle is to bring the computing to the data. Files stay in one storage location; analysis machines start next to that storage, read what they need, write results back, and shut down. Analysts work through a browser, a remote notebook, or a remote terminal, and only small results such as tables, plots, and reports come back to a laptop.

What cloud object storage is

Cloud storage for genomics is usually object storage: Amazon S3, Google Cloud Storage, or Azure Blob Storage. Files (objects) live in buckets (containers on Azure) and are addressed by a path such as s3://bucket/path/file.cram, gs://bucket/path/file.cram, or https://account.blob.core.windows.net/container/path/file.cram. Object storage looks like folders but is not a regular file system: objects are written whole rather than edited in place, "folders" are just prefixes of names, and access is through command-line tools, software libraries, or mounting utilities rather than a normal drive letter.

• Storage classes. Each provider offers standard storage for active data and cheaper infrequent-access and archive classes (for example S3 Glacier or Google Archive) with retrieval fees and, for the deepest archives, retrieval delays of hours. Lifecycle rules can move files between classes automatically based on age.
• Costs. Programs pay for stored gigabytes per month, for operations such as listing and reading, and for egress when data leave a region or the provider. Egress often surprises new teams. "Requester pays" buckets charge the person downloading rather than the owner. Check each provider's pricing calculator for current rates, and set budgets and alerts.
• Protection. Turn on encryption at rest (default on major providers), object versioning or soft delete, access logging, and public-access blocking. Grant access through groups and roles rather than individual keys, and never place protected data in a publicly readable bucket.
• Organization. Many teams separate buckets or prefixes by lifecycle and permission: raw data that almost no one can change, working areas that pipelines write to, results for sharing, and reference files. The IT request generator includes this layout.

Where analysis takes place

Reading data from storage can happen in two ways. Most pipelines copy ("localize") input files onto a machine's disk before a step runs. Tools built on htslib can also stream a region directly from cloud storage using the index, for example samtools view gs://bucket/sample.cram chr1:1000000-1010000, which avoids copying a whole CRAM to look at one gene. Streaming requires credentials to be available to the tool and, for CRAM, access to the reference FASTA.

How file transfer works

Transfers happen when a sequencing provider delivers data, when a collaborator shares a cohort, when data move between an institution and the cloud, or when files move between clouds. Every transfer follows the same rules:

1. Confirm the transfer is allowed under the data use agreement, and record it in a transfer log. The data transfer generator produces a manifest and log template.
2. Build a manifest listing every file with its size and checksum (MD5 or a SHA-2 hash) before sending.
3. Send companion files together: indexes with their data files, and the reference FASTA or its exact name and checksum for CRAM.
4. Use a tool that resumes and verifies, since transfers of terabytes will be interrupted.
5. Verify on arrival by recomputing checksums and comparing file counts and sizes with the manifest. Note that Amazon S3 ETags are not plain MD5 checksums for files uploaded in parts, so compute checksums yourself or use the provider's additional checksum features.
6. Keep data encrypted in transit and at rest, and do not use email, consumer file sharing, or unencrypted portable drives.

Keeping cloud costs predictable

• Label every resource with a project or grant code so bills can be split, and set budget alerts at several thresholds.
• Use spot or preemptible machines for pipeline steps that can retry, and let the workflow engine shut machines down.
• Delete intermediate files after a run passes QC, and move raw data to cheaper storage classes with lifecycle rules once CRAMs are verified.
• Keep compute in the same region as the data, and review egress lines on the monthly bill.

Infrastructure

GitHub, containers, and reproducible code

Data live in storage; code lives in version control. Git records every change to code as a commit with an author, a date, and a message. GitHub (or GitLab, or Bitbucket) hosts Git repositories and adds review, issue tracking, automation, and access control. For a bioinformatics team, the repository is where pipelines, configuration files, analysis notebooks, environment definitions, and documentation are kept, and the commit identifier is how you later prove which code produced a result.

What GitHub does for a bioinformatics team

• Repositories hold pipelines, configuration, analysis code, and documentation. Private repositories inside an organization account keep work internal until it is ready to publish.
• Branches and pull requests let someone propose a change and have a colleague review it before it reaches the main branch. Branch protection rules can require review and passing tests.
• Issues and project boards track analysis requests, bugs, and decisions next to the code that addresses them.
• GitHub Actions runs automated checks on every pull request, such as linting, unit tests, or a full pipeline run on a tiny test data set.
• Releases and tags mark the exact code version used for a data freeze, report, or paper. Services such as Zenodo can archive a release with a DOI.
• Container registry (GitHub Packages, ghcr.io) can store the team's container images next to the code that builds them.
• Security features, including required two-factor authentication, secret scanning, and dependency alerts, reduce the chance that a credential or a vulnerable package slips through.

What never goes into a repository

Participant data of any kind (FASTQ, CRAM, VCF, phenotype tables, identifier crosswalks), credentials and keys, .env files, and large binary files. Git keeps every version forever, so a file committed by mistake stays in the history even after deletion and must be purged with specialized tools, and a public repository exposes it immediately. A starting .gitignore for a genomics repository:

# Genomic and phenotype data never go into Git
*.fastq.gz
*.fq.gz
*.bam
*.bai
*.cram
*.crai
*.vcf
*.vcf.gz
*.tbi
*.csi
*.bcf
*.h5ad
*.rds
data/
results/
# Credentials and local settings
.env
*.pem
*.json.key
# Workflow engine working files
work/
.nextflow*
cromwell-executions/
.snakemake/

A repository layout many teams use

study-pipeline/
  README.md              purpose, how to run, who maintains it
  CHANGELOG.md           what changed in each release
  main.nf or main.wdl    workflow definition
  conf/                  profiles for laptop, cluster, and cloud
  containers/            Dockerfiles or container references with digests
  envs/environment.yml   conda environment for local work
  assets/                small reference lists, target BED files (no participant data)
  tests/                 tiny test inputs and expected outputs
  docs/                  protocol, QC thresholds, validation reports
  notebooks/             analysis notebooks that read from storage paths
  .github/workflows/     automated tests

Containers and software environments

A container packages a tool with its operating system libraries so it runs the same on a laptop, a cluster, and the cloud. Docker builds and runs containers on laptops and cloud machines; Apptainer (formerly Singularity) runs them on shared clusters where Docker is not allowed. BioContainers publishes ready-made images for most bioinformatics tools. For interactive work, conda environments (created quickly with mamba, using the conda-forge and Bioconda channels) and R's renv lock package versions in a file that lives in the repository.

Refer to containers by an immutable digest (image@sha256:...) rather than a moving tag such as latest, so a rerun uses the identical software.

The minimum record for reproducing a result

• Git commit or release tag of the pipeline and analysis code.
• Container digests or environment lock files.
• Reference FASTA name and checksum, annotation release, and resource versions (gnomAD, ClinVar, VEP cache).
• Input manifest with storage paths and checksums, and the data freeze name.
• Parameters and configuration files, random seeds, and the workflow engine's run report.

Sharing a workflow outside the team is easiest through Dockstore, which registers WDL, Nextflow, CWL, and Galaxy workflows from GitHub and supports the GA4GH Tool Registry Service.

Infrastructure

Command reference

These are the commands a lead is most likely to see in notebooks, pull requests, and help requests. File names are placeholders. Options change between versions, so check tool --help or the documentation for the version your containers pin. Each group opens on its own.

Running the team

Setting up a new team member

A new analyst needs a modest computer, a standard set of software, several accounts, the right level of access to data, and completed compliance steps before touching participant data. Most heavy computing happens on a cluster or in the cloud, so the laptop is mainly a window into those systems. The onboarding plan generator turns this section into a checklist for a specific person, and the IT request generator turns it into a request your IT team can act on.

Computer hardware

Planning figures as of September 2026; confirm against your institution's standard builds.

• A current laptop with 16 to 32 GB of memory, an SSD of 512 GB or more, full-disk encryption, and institutional device management.
• An external monitor, since reading alignment views, notebooks, and documentation side by side is routine.
• A workstation with more memory is useful only if the team plans to run analyses locally on de-identified or public data; most genome-scale work belongs on a cluster or in the cloud.

Software for the laptop

Accounts and access

• Institutional identity with multi-factor authentication, and the email, chat, and ticketing tools the team uses.
• Membership in the GitHub organization and the right GitHub team, with two-factor authentication and an SSH key registered.
• Cloud access through group membership (for example a Google group or an AWS IAM Identity Center group), with read-only access to raw data and read-write access to working areas.
• An HPC account and project allocation, if the team uses a cluster.
• Analysis platform accounts (for example Terra or DNAnexus) and workspace membership.
• Controlled-access data approvals, such as dbGaP or AnVIL data access requests, which take their own review time.
• Container registry access if the team publishes images.

Principles that keep access manageable: grant through groups and roles; keep separate development and production projects; give raw data write access to very few people; review memberships on a schedule; and remove access on the last day as part of offboarding.

Compliance and training

• Human subjects research training required by your institution (for example CITI modules).
• Privacy and security training, including HIPAA training where identifiable health information is involved.
• Signature on each data use agreement that requires named users.
• Listing on each relevant IRB protocol as approved study personnel, which is typically what grants the institutional login to study information.
• Acknowledgment of the team's data handling rules: no participant data on laptops, in email, in Git, or in unapproved tools, including AI tools that are not approved for that data.

An early learning path

1. Set up the laptop, clone the team repositories, and run the shell and Git commands in the command reference on a practice repository.
2. Run an nf-core pipeline's test profile on the cluster or in the cloud to confirm the environment, containers, and permissions work.
3. Process a Genome in a Bottle sample (or a subset of it) through the team pipeline and compare the calls with the benchmark.
4. Open a CRAM and a VCF from that run in IGV and with samtools and bcftools, and trace one variant from reads to VCF row.
5. Read the team's protocol, QC thresholds, and harmonization log for the current study.
6. Write a notebook entry using the generator, and review a colleague's pull request.

Roles on a bioinformatics team

Running the team

Generators for plans, requests, and records

Choose a generator, fill in what you know, and select Build. The result appears as Markdown text you can edit in place, copy, download, or print. Entries are saved in this browser only, so you can return to them later; nothing is sent anywhere.

Reference

Glossary

Definitions are written for leaders and collaborators rather than for specialists. Type to filter by term or by any word in a definition, or jump by letter.

No terms match that filter. Try a shorter word or clear the box.

Reference

Web sites and specifications referenced

These are the project and specification sites behind the descriptions in this resource, listed as of September 2026. Tool options, recommended steps, and platform features change with new releases, so the linked documentation is the authority when it differs from a summary here.

File format specifications and core tools

• SAM, BAM, CRAM, VCF, BCF, htsget, refget, and Crypt4GH specifications (hts-specs)
• htslib, samtools, and bcftools
• bcftools manual
• BWA-MEM2
• GATK and GATK Best Practices
• Picard tools
• DeepVariant
• GLnexus
• Hail
• Ensembl Variant Effect Predictor (VEP)
• mosdepth
• somalier
• PLINK 2
• FastQC and MultiQC
• STAR, Salmon, and DESeq2
• Scanpy and Seurat
• ExpansionHunter and Manta
• Integrative Genomics Viewer (IGV)

Workflows, containers, and code

• nf-core pipelines and Nextflow
• Workflow Description Language (WDL) and miniwdl
• Snakemake
• Common Workflow Language (CWL)
• Dockstore
• Bioconda, conda-forge, and Miniforge
• BioContainers
• Docker documentation and Apptainer
• GitHub documentation and GitHub CLI
• Slurm workload manager

Standards, ontologies, and reference resources

• Global Alliance for Genomics and Health (GA4GH) and its GitHub organization
• Phenopacket schema documentation and repository
• phenopacket-store
• Exomiser
• Human Phenotype Ontology and MONDO Disease Ontology
• Data Use Ontology (DUO)
• Beacon v2 and Variation Representation Specification
• gnomAD and ClinVar
• Genome in a Bottle (NIST)
• OMOP Common Data Model (OHDSI)
• UCSC liftOver and CrossMap

Storage, compute, transfer, and platforms

• Amazon S3 documentation and AWS CLI
• Google Cloud Storage documentation and Google Cloud CLI
• Azure Blob Storage documentation
• rclone and Globus
• Terra and NHGRI AnVIL