Deployment Options

The pipeline can be deployed in several environments:

SLURM Deployment

Deploying on a SLURM cluster provides better performance and resource management.

Requirements

  • Access to a SLURM cluster

  • Nextflow installation

  • Singularity/Apptainer installation

  • Optional: Figurl setup for visualizations

Configuration

  1. Clone the repository:

git clone https://github.com/AllenNeuralDynamics/aind-ephys-pipeline.git
cd aind-ephys-pipeline
cd pipeline

  1. Copy and modify the SLURM configuration:

cp nextflow_slurm.config nextflow_slurm_custom.config

  1. Update the params.default_queue and params.gpu_queue parameters in nextflow_slurm_custom.config to match your cluster’s partitions. The latter is only needed if different than the default queue.

  2. Create a new or modify the existing a submission script (slurm_submit.sh):

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem=4GB
#SBATCH --partition={your-partition}
#SBATCH --time=2:00:00

# Load required environment (if nextflow is installed in a conda environment)
conda activate env_nf

PIPELINE_PATH="path-to-your-cloned-repo"
DATA_PATH="path-to-data-folder"
RESULTS_PATH="path-to-results-folder"
WORKDIR="path-to-large-workdir"

DATA_PATH=$DATA_PATH RESULTS_PATH=$RESULTS_PATH nextflow \
    -C $PIPELINE_PATH/pipeline/nextflow_slurm_custom.config \
    -log $RESULTS_PATH/nextflow/nextflow.log \
    run $PIPELINE_PATH/pipeline/main_multi_backend.nf \
    -work-dir $WORKDIR \
    -resume

  1. (Optional) Pre-build required Apptainer/Singularity/Apptainer images for faster startup:

./pull_pipeline_images.sh --sorter kilosort4

This will pull and build the necessary Apptainer/Singularity images for the Kilosort4 sorter. Adjust the --sorter argument as needed (e.g., to kilosort25 or spykingcircus2 or all). Note that this step requires you to set the NXF_APPTAINER_CACHEDIR/NXF_SINGULARITY_CACHEDIR environment variable to a directory with enough space to store the images. Images used by the nextflow script will be cached automatically if not pre-built.

  1. Submit the pipeline job:

sbatch slurm_submit.sh

Local Deployment

Warning

While local deployment is possible, it’s recommended to use SLURM or batch processing systems for better performance. Local deployment limits parallelization of resource-intensive processes to avoid system overload.

Requirements

See the Installation page for detailed setup instructions.

Running Locally

  1. Clone the repository:

git clone https://github.com/AllenNeuralDynamics/aind-ephys-pipeline.git
cd aind-ephys-pipeline
cd pipeline

  1. Run the pipeline:

DATA_PATH=$PWD/../data RESULTS_PATH=$PWD/../results \
    nextflow -C nextflow_local.config -log $RESULTS_PATH/nextflow/nextflow.log \
    run main_multi_backend.nf \
    --n_jobs 8 -resume

Code Ocean Deployment (AIND)

For AIND internal use, the pipeline is deployed on Code Ocean with different branches for various configurations:

Main Branches

  • main/co_kilosort4: Kilosort4 sorter

  • co_kilosort25: Kilosort2.5 sorter

  • co_spykingcircus2: Spyking Circus 2 sorter

Optogenetics Branches

  • co_kilosort25_opto: Kilosort2.5 with opto artifact removal

  • co_kilosort4_opto: Kilosort4 with opto artifact removal

  • co_spykingcircus2_opto: Spyking Circus 2 with opto artifact removal