Skip to content

Commit b7e37fa

Browse files
alejoe91alejoe91
authored
Merge pull request #35 from Tauffer-Consulting/compiled-hdsort
Added hdsort-compiled
2 parents 910f54d + 7762f3d commit b7e37fa

5 files changed

Lines changed: 148 additions & 0 deletions

File tree

‎hdsort-compiled/Dockerfile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
FROM hdsort-matlab-base
2+
3+
RUN chmod 755 /usr/bin/mlrtapp/hdsort_compiled
4+
ENV PATH="/usr/bin/mlrtapp:${PATH}"
5+
6+
RUN apt-get update -y
7+
RUN apt-get install software-properties-common -y
8+
RUN add-apt-repository ppa:deadsnakes/ppa -y
9+
RUN apt-get install git python3.8 python3.8-dev -y
10+
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
11+
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2
12+
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
13+
RUN apt-get install python3-pip -y
14+
RUN update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1
15+
RUN pip install -U pip
16+

‎hdsort-compiled/README.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# HDsort Compiled Docker Images
2+
3+
This documentation is intended to show how to create a Docker image with Matlab compiled HDsort sorter. The main goal of this project is to avoid the requirement of Matlab Licenses and also abstract the installation and setup steps to run the sorter
4+
5+
There are three main steps for generating a functional HDsort docker image:
6+
1. Compile HDsort as Matlab's Standalone Application
7+
2. Create a (base) docker image with Matlab Runtime and the compiled application from step 1
8+
3. Extend the docker image from step 2 for improvements and fixes
9+
10+
## Requirements
11+
- Packaging a MATLAB Docker image is supported on Linux only
12+
- Docker
13+
- Matlab
14+
15+
### Matlab Requirements
16+
- MATLAB Compiler
17+
- More requirements to be checked and listed...
18+
19+
## Compiling HDsort as Matlab's Standalone Application
20+
- Git clone or Download HDsort [source code](https://git.bsse.ethz.ch/hima_public/HDsort)
21+
- Open Matlab
22+
- Set Matlab's workspace folder to `/path/to/spikeinterface-dockerfiles/hdsort-compiled`
23+
- Run `mcc` command with `hdsort_compiled.m` as base and adding path to git cloned HDSort:
24+
```matlab
25+
>> mcc -m hdsort_compiled.m -a <git-cloned-path>/HDsort
26+
```
27+
28+
## Generating Base Docker Image
29+
- To generate the base docker image (called `hdsort-matlab-base`) with the compiled application, run the following command in Matlab console:
30+
```matlab
31+
>> compiler.package.docker('hdsort_compiled', 'requiredMCRProducts.txt', 'ImageName', 'hdsort-matlab-base')
32+
```
33+
34+
35+
- [Optional] Files generated by Matlab Compiler can be deleted:
36+
- In your terminal, go to the `hdsort-compiled` folder in this project:
37+
```bash
38+
$ cd /path/to/spikeinterface-dockerfiles/hdsort-compiled
39+
```
40+
41+
- Run `rm` command:
42+
```bash
43+
$ rm -r \
44+
hdsort-matlab-basedocker \
45+
hdsort_compiled \
46+
includedSupportPackages.txt \
47+
mccExcludedFiles.log \
48+
readme.txt \
49+
requiredMCRProducts.txt \
50+
run_hdsort_compiled.sh \
51+
unresolvedSymbols.txt
52+
```
53+
54+
55+
## Extending Base Image and creating final image
56+
The Dockerfile in this folder applies some fixes and updates to the base image generated automatically by Matlab in order to properly run waveclus:
57+
58+
59+
- In your terminal, go to the folder for this project:
60+
```
61+
$ cd /path/to/spikeinterface-dockerfiles/hdsort-compiled
62+
```
63+
64+
- Run build script:
65+
```
66+
$ source build.sh
67+
```

‎hdsort-compiled/build.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#!/bin/bash
2+
3+
docker build -t spikeinterface/hdsort-compiled-base:latest -t spikeinterface/hdsort-compiled-base:0.1.0 .
4+

‎hdsort-compiled/hdsort_compiled.m‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
function hdsort_compiled(outputFolder)
2+
try
3+
% cding to outputFolder is needed to save output files in the correct place
4+
cd(fullfile(outputFolder))
5+
6+
%% Load sorter configs and params
7+
load(fullfile(outputFolder, 'configsParams.mat'))
8+
9+
if (strcmp(fileFormat, 'maxwell'))
10+
RAW = hdsort.file.MaxWellFile(rawFile);
11+
elseif (strcmp(fileFormat, 'mea1k'))
12+
RAW = hdsort.file.BELMEAFile(rawFile);
13+
end
14+
15+
RAW.restrictToConnectedChannels(); % This line is very important to not sort empty electrodes!
16+
17+
%% Create the object that performs the sorting:
18+
mainFolder = '.';
19+
HDSorting = hdsort.Sorting(RAW, mainFolder, sortingName)
20+
21+
%% Preprocess:
22+
% The preprocessor loads data from the file in chunks, filters it, and saves
23+
% the filtered data into a new hdf5 file that is standardized for all types
24+
% of input data.
25+
% It further performs a couple of operations for each local electrode group
26+
% (LEG) such as spike detection, spike waveform cutting and noise
27+
% estimation. The result is a folder named group000x for each LEG that
28+
% contains the data necessary to perform the parallel parts of the sorting.
29+
% This implementation minimizes the number of time-consuming file access
30+
% operations and thus speeds up the parallel processes significantly, even
31+
% allowing the parallel processes to be run on a small desktop computer or
32+
% laptop.
33+
34+
chunkSize = chunkSize; % This number depends a lot on the available RAM
35+
HDSorting.preprocess('chunkSize', chunkSize, 'forceFileDeletionIfExists', true);
36+
37+
%% Sort each LEG independently:
38+
HDSorting.sort('sortingMode', loopMode); % (default)
39+
% Alternative sorting modes are:
40+
% HDSorting.sort('sortingMode', 'local'); % for loop over each LEG
41+
% HDSorting.sort('sortingMode', 'grid'); % requires a computer grid architecture
42+
43+
%% Combine the results of each LEG in the postprocessing step:
44+
HDSorting.postprocess()
45+
46+
%% Export the results in an easy to read format:
47+
[sortedPopulation, sortedPopulation_discarded] = HDSorting.createSortedPopulation(mainFolder);
48+
49+
%% When the sorting has already been run before, open the results from a file with:
50+
% sortedPopulation = hdsort.results.Population(HDSorting.files.results);
51+
catch
52+
fprintf('----------------------------------------');
53+
fprintf(lasterr());
54+
quit(1);
55+
end
56+
quit(0);
57+
end

‎hdsort-compiled/push.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
#!/bin/bash
2+
3+
docker push --all-tags spikeinterface/hdsort-compiled-base
4+

0 commit comments

Comments
 (0)