Pipeline Steps

The AIND Ephys Pipeline consists of several key processing steps that are executed in sequence. Here’s a detailed look at each step:

Job Dispatch

The job-dispatch step:

  • Generates JSON files for parallel processing

  • Enables parallelization across:

    • Multiple probes

    • Multiple shanks (e.g., for NP2-4shank probes)

  • Creates independent processing jobs for parallel execution

Preprocessing

The preprocessing step handles several critical data preparation tasks:

  • Phase shift correction

  • Highpass filtering

  • Denoising

    • Bad channel removal

    • Common median reference (“cmr”) or highpass spatial filter (“destripe”)

  • Motion estimation and correction (optional)

Spike Sorting

The pipeline supports multiple spike sorting algorithms:

Each sorter can be selected based on your specific needs and data characteristics.

Postprocessing

The postprocessing step performs additional processing on the combined preprocessed recording and sorted data:

  • Removal of duplicate units

  • Computations of extensions:

    • Waveforms extraction

    • Templates

    • Spike amplitudes

    • Unit locations

    • Principal Component Analysis (PCA) projections

    • Spike locations

    • Correlograms

    • Template similarity

    • Template metrics

    • Quality metrics

Curation

The curation step applies quality control by:

  • Quality metrics-based filtering using thresholds on:

    • ISI violation ratio

    • Presence ratio

    • Amplitude cutoff

  • Unit classification as noise, MUA, or SUA using pretrained classifier (UnitRefine)

The recipe for quality metrics can be customized to suit your specific needs.

Visualization

The visualization step generates static figures and interactive Figurl links for each probe:

  • timeseries: including snippets of raw data, drift map, and motion visualizations

  • sorting_summary: for spike sorting results inspection and curation

Each plot of the timeseries is also saved as a static image in the visualization/ folder.

Result Collection

The result collection step:

  • Aggregates outputs from all parallel jobs

  • Copies output folders to the results directory

  • Organizes results in a standardized structure

NWB Export

The final step creates standardized NWB output files, including:

Features:

  • Supports multiple streams (e.g., probes) per file

  • Optional raw data and LFP data writing