Merge pull request #25 from Tauffer-Consulting/compiled-kilosort-all

Add Matlab Compiled Kilosort - All versions

Author: alejoe91

kilosort_no_license/README.md

@@ -0,0 +1,49 @@
+# Kilosort Compiled Docker Images
+
+This documentation is intended to show how to create a Docker image with Matlab compiled implementation of Kilosort sorter. The main goal of this project is to avoid the requirement of Matlab Licenses and also abstract the installation and setup steps to run kilosort
+
+## Requirements
+- Packaging a MATLAB Docker image is supported on Linux only
+- Docker
+- Matlab
+- NVIDIA GPU and CUDA capabilities
+
+### Matlab Requirements
+- MATLAB Compiler
+- Parallel Computing Toolbox
+- Signal Processing Toolbox
+- Statistics and Machine Learning Toolbox
+
+## Creating Docker Image
+
+There are four main steps to generate a functional kilosort-compiled docker image:
+
+1. Kilosort Setup
+2. Compile Kilosort as Standalone Application
+3. Create a (base) docker image with Matlab Runtime and the compiled application from step 2
+4. Extend the docker image from step 3 for improvements and fixes
+
+Detailed descriptions for each step are documented in `README.md` for each supported kilosort version folder (`kilosort-compiled`, `kilosort2-compiled` and so on)
+
+Licenses for Matlab and toolboxes are needed only for compiling kilosort as Standalone Application and to generate the base Docker image. After this process, no license will be required, either to extend the base image or to run the sorter.
+
+## Running a container
+
+After generating kilosort-compiled image, [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#setting-up-nvidia-container-toolkit) is required to run a docker with GPU capabilities
+
+The base syntax to run dockerized kilosort is:
+
+```bash
+docker run -v <host-data-folder>:<docker-data-folder> <image>:<tag> <ks_command> [ARGS]
+```
+- A volume has to be binded to a folder were all the data to run kilosort are stored
+- `<image>`: Image name given by `build.sh` script
+- `<tag>`: Image Tag given by `build.sh` script
+- `<ks_command>`: Depends on kilosort version, options are `ks_compiled`, `ks2_compiled`, `ks2_5_compiled` or `ks3_compiled`
+- `[ARGS]`: Currently the only ARG option is the folder where the data are stored (\<docker-data-folder\>)
+
+This kilosort compiled version assumes that there are also two `.mat` files in this data folder:
+1. `ops.mat`: Configurations (options) data
+2. `chanMap.mat`: ChannelMap data
+
+Both files are automatically generated by spikeinterface when running `run\_sorter`. If you want to run compiled kilosort independently you need to generate them manually (HOW TO still needs to be done)

kilosort_no_license/kilosort-compiled/Dockerfile

@@ -0,0 +1,16 @@
+FROM ks-matlab-base
+
+RUN chmod 755 /usr/bin/mlrtapp/ks_compiled
+ENV PATH="/usr/bin/mlrtapp:${PATH}"
+
+RUN apt-get update -y
+RUN apt-get install software-properties-common -y
+RUN add-apt-repository ppa:deadsnakes/ppa -y
+RUN apt-get install git python3.8 python3.8-dev -y
+RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
+RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2
+RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
+RUN apt-get install python3-pip -y
+RUN update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1
+RUN pip install -U pip
+

kilosort_no_license/kilosort-compiled/README.md

@@ -0,0 +1,56 @@
+# Generating Compiled Kilosort Docker Image
+
+## 1. Kilosort Setup 
+- Git clone or Download Kilosort [source code](https://github.com/cortex-lab/KiloSort)
+- Open Matlab
+- Compile Kilosort mexfiles:
+  - Set Matlab's workspace folder to `<git-cloned-path>/KiloSort/CUDA`
+  - In Matlab console run:
+    ```matlab
+    >> mexGPUall
+    ```
+
+## 2. Compiling Kilosort as Matlab's Standalone Application
+- Set Matlab's workspace folder to `/path/to/spikeinterface-dockerfiles/kilosort_no_license`
+- Run `mcc` command with `utils` folder and kilosort path:
+```matlab
+>> mcc -m kilosort-compiled/ks_compiled.m -a utils -a <git-cloned-path>/KiloSort
+```
+
+## 3. Generating Base Docker Image
+- To generate the base docker image (called `ks-matlab-base`) with the compiled application, run the following command in Matlab console:
+```matlab
+>> compiler.package.docker('ks_compiled', 'requiredMCRProducts.txt', 'ImageName', 'ks-matlab-base')
+```
+
+- [Optional] Files generated by Matlab Compiler can be deleted:
+  - In your terminal, go to the `kilosort_no_license` folder in this project:
+  ```bash
+  $ cd /path/to/spikeinterface-dockerfiles/kilosort_no_license
+  ```
+  - Run `rm` command:
+  ```bash
+  $ rm -r \
+    includedSupportPackages.txt \
+    ks-matlab-basedocker/ \
+    ks_compiled \
+    mccExcludedFiles.log \
+    readme.txt \
+    requiredMCRProducts.txt \
+    run_ks_compiled.sh \
+    unresolvedSymbols.txt
+  ```
+
+## 4. Extending Base Image and creating final image
+
+The Dockerfile in this folder applies some fixes and updates to the base image generated automatically by Matlab in order to properly run kilosort:
+
+- In your terminal, go to the folder for this project:
+```bash
+$ cd /path/to/spikeinterface-dockerfiles/kilosort_no_license/kilosort-compiled
+```
+
+- Run build script:
+```bash
+$ source build.sh
+```

kilosort_no_license/kilosort-compiled/build.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+docker build -t spikeinterface/kilosort-compiled-base:0.1.0 .
+

kilosort_no_license/kilosort-compiled/ks_compiled.m

@@ -0,0 +1,31 @@
+function ks_master(fpath)
+    try
+        set(groot,'defaultFigureVisible', 'off');
+
+        % Load channel map file
+        load(fullfile(fpath, 'chanMap.mat'));
+
+        % Load the configuration file, it builds the structure of options (ops)
+        load(fullfile(fpath, 'ops.mat'));
+
+        useGPU = ops.GPU;
+
+        % load predefined principal components (visualization only (Phy): used for features)
+        dd                  = load('PCspikes2.mat'); % you might want to recompute this from your own data
+        ops.wPCA            = dd.Wi(:,1:7);   % PCs
+
+        % This part runs the normal Kilosort processing on the simulated data
+        [rez, DATA, uproj] = preprocessData(ops); % preprocess data and extract spikes for initialization
+        rez                = fitTemplates(rez, DATA, uproj);  % fit templates iteratively
+        rez                = fullMPMU(rez, DATA);% extract final spike times (overlapping extraction)
+
+        % save python results file for Phy
+        rezToPhy(rez, fullfile(fpath));
+    catch
+        fprintf('----------------------------------------');
+        fprintf(lasterr());
+        quit(1);
+    end
+    quit(0);
+end
+

kilosort_no_license/kilosort-compiled/push.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+docker push spikeinterface/kilosort-compiled-base:0.1.0
+

kilosort_no_license/kilosort2-compiled/Dockerfile

@@ -0,0 +1,16 @@
+FROM ks2-matlab-base
+
+RUN chmod 755 /usr/bin/mlrtapp/ks2_compiled
+ENV PATH="/usr/bin/mlrtapp:${PATH}"
+
+RUN apt-get update -y
+RUN apt-get install software-properties-common -y
+RUN add-apt-repository ppa:deadsnakes/ppa -y
+RUN apt-get install git python3.8 python3.8-dev -y
+RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
+RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2
+RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
+RUN apt-get install python3-pip -y
+RUN update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1
+RUN pip install -U pip
+

kilosort_no_license/kilosort2-compiled/README.md

@@ -0,0 +1,60 @@
+# Generating Compiled Kilosort2 Docker Image
+
+## 1. Kilosort Setup 
+- Git clone or Download Kilosort [source code](https://github.com/MouseLand/Kilosort)
+- Git checkout to the correct version:
+```bash
+$ git checkout tags/v2.0
+```
+- Open Matlab
+- Compile Kilosort mexfiles:
+  - Set Matlab's workspace folder to `<git-cloned-path>/Kilosort/CUDA`
+  - In Matlab console run:
+    ```matlab
+    >> mexGPUall
+    ```
+
+## 2. Compiling Kilosort as Matlab's Standalone Application
+- Set Matlab's workspace folder to `/path/to/spikeinterface-dockerfiles/kilosort_no_license`
+- Run `mcc` command with `utils` folder and kilosort path:
+```matlab
+>> mcc -m kilosort2-compiled/ks2_compiled.m -a utils -a <git-cloned-path>/Kilosort
+```
+
+## 3. Generating Base Docker Image
+- To generate the base docker image (called `ks2-matlab-base`) with the compiled application, run the following command in Matlab console:
+```matlab
+>> compiler.package.docker('ks2_compiled', 'requiredMCRProducts.txt', 'ImageName', 'ks2-matlab-base')
+```
+
+- [Optional] Files generated by Matlab Compiler can be deleted:
+  - In your terminal, go to the `kilosort_no_license` folder in this project:
+  ```bash
+  $ cd /path/to/spikeinterface-dockerfiles/kilosort_no_license
+  ```
+  - Run `rm` command:
+  ```bash
+  $ rm -r \
+    includedSupportPackages.txt \
+    ks2-matlab-basedocker/ \
+    ks2_compiled \
+    mccExcludedFiles.log \
+    readme.txt \
+    requiredMCRProducts.txt \
+    run_ks2_compiled.sh \
+    unresolvedSymbols.txt
+  ```
+
+## 4. Extending Base Image and creating final image
+
+The Dockerfile in this folder applies some fixes and updates to the base image generated automatically by Matlab in order to properly run kilosort2:
+
+- In your terminal, go to the folder for this project:
+```bash
+$ cd /path/to/spikeinterface-dockerfiles/kilosort_no_license/kilosort2-compiled
+```
+
+- Run build script:
+```bash
+$ source build.sh
+```

kilosort_no_license/kilosort2-compiled/build.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+docker build -t spikeinterface/kilosort2-compiled-base:0.1.0 .
+

kilosort_no_license/kilosort2-compiled/ks2_compiled.m

@@ -0,0 +1,42 @@
+function ks2_compiled(fpath)
+    try
+        set(groot,'defaultFigureVisible', 'off');
+
+        % Load channel map file
+        load(fullfile(fpath, 'chanMap.mat'));
+
+        % Load the configuration file, it builds the structure of options (ops)
+        load(fullfile(fpath, 'ops.mat'));
+
+        % preprocess data to create temp_wh.dat
+        rez = preprocessDataSub(ops);
+
+        % time-reordering as a function of drift
+        rez = clusterSingleBatches(rez);
+
+        % main tracking and template matching algorithm
+        rez = learnAndSolve8b(rez);
+
+        % final merges
+        rez = find_merges(rez, 1);
+
+        % final splits by SVD
+        rez = splitAllClusters(rez, 1);
+
+        % final splits by amplitudes
+        rez = splitAllClusters(rez, 0);
+
+        % decide on cutoff
+        rez = set_cutoff(rez);
+
+        fprintf('found %d good units \n', sum(rez.good>0))
+
+        fprintf('Saving results to Phy  \n')
+        rezToPhy(rez, fullfile(fpath));
+    catch
+        fprintf('----------------------------------------');
+        fprintf(lasterr());
+        quit(1);
+    end
+    quit(0);
+end