MESmerize/community

Eric Thomson

@EricThomson

To run caiman motion correction in mesmerize without the piecewise motion correction, should it be enough to check the Use motion correction kwargs box and manually add in pw_rigid=False, and then just set the rigid correction params I want (and ignore the Elastic correction params)?

Kushal Kolar

@kushalkolar

Nope, I haven't added the option to perform just the rigid correction, is there a usecase where non-rigid doesn't work?

Eric Thomson

@EricThomson

There are cases where rigid is sufficient. Since it is a lot faster it is preferable in those cases (e.g., we have animals under anesthesia where there is not much movement, and rigid works well).

Eric Thomson

@EricThomson

To "delete" a project (e.g., a test project I was using to play with) I can just delete its associated directory? Or I guess my question is: is this the recommended way? :)

Kushal Kolar

@kushalkolar

Yes, and yup that's the recommended way

Irene Ayuso

@AyusoAa

Hi all! I am trying to use the Mesmerize VM. I successfully installed it and mesmerize is in the venvs folder. However, when I double click in the desktop icon, mesmerize doe snot start. A terminal window opens, no message pops up, and the it disappears. Mesmerize also does not start when typing mesmerize in the terminal. Did anyone encounter such issues? Cheers

Kushal Kolar

@kushalkolar

@AyusoAa When you say you installed it, do you mean that you imported the VM? You don't need to install Mesmerize within the VM, Mesmerize is pre-installed.

Irene Ayuso

@AyusoAa

Yes, I meant, I imported it. I did not install mesmerize.

Kushal Kolar

@kushalkolar

Can you provide screenshots and all the details for reproducing it? You're welcome to post an issue on the repo

Irene Ayuso

@AyusoAa

that's the terminal window that shows up, and then, dissapears, when I double click at the mesmerize icon. Alright, I will post an issue on the repo with all the details

Serapio M. Baca

@smbaca

Got mesmerize installed onto an ubuntu linux box but can't get caiman to install... any thoughts on this error message? Installing collected packages: caiman
Running setup.py develop for caiman
ERROR: Command errored out with exit status 1:
command: /home/serapio/Desktop/mesmer/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/serapio/Desktop/mesmer/CaImAn/setup.py'"'"'; file='"'"'/home/serapio/Desktop/mesmer/CaImAn/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps
cwd: /home/serapio/Desktop/mesmer/CaImAn/
Complete output (25 lines):
running develop
running egg_info
creating caiman.egg-info
writing caiman.egg-info/PKG-INFO
writing dependency_links to caiman.egg-info/dependency_links.txt
writing top-level names to caiman.egg-info/top_level.txt
writing manifest file 'caiman.egg-info/SOURCES.txt'
reading manifest file 'caiman.egg-info/SOURCES.txt'
adding license file 'LICENSE.txt'
writing manifest file 'caiman.egg-info/SOURCES.txt'
running build_ext
building 'caiman.source_extraction.cnmf.oasis' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/caiman
creating build/temp.linux-x86_64-3.6/caiman/source_extraction
creating build/temp.linux-x86_64-3.6/caiman/source_extraction/cnmf
/opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -g -fwrapv -O2 -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC -I/home/serapio/Desktop/mesmer/lib/python3.6/site-packages/numpy/core/include -I/home/serapio/Desktop/mesmer/include -I/usr/include/python3.6m -c caiman/source_extraction/cnmf/oasis.cpp -o build/temp.linux-x86_64-3.6/caiman/source_extraction/cnmf/oasis.o
In file included from /usr/include/python3.6m/Python.h:8:0,
from caiman/source_extraction/cnmf/oasis.cpp:26:
/usr/include/python3.6m/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory

 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1
----------------------------------------

ERROR: Command errored out with exit status 1: /home/serapio/Desktop/mesmer/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/serapio/Desktop/mesmer/CaImAn/setup.py'"'"'; file='"'"'/home/serapio/Desktop/mesmer/CaImAn/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

Kushal Kolar

@kushalkolar

@smbaca Can you please post an issue on the repo? It's easier to read when you put it in a code block. Thanks!

1 reply

Irene Ayuso

@AyusoAa

Hi all! I would like to ask for help with my mesmerize project. With a colleague, we're trying to set up the mesmerize pipeline in the lab, but we're encountering a number of issues. We're dealing with 1p recordings in tiff format, with their respective json files and 1D behavioral files. We have managed to run motion correction and do CNMF-E, and as well, we've managed to import our stimulus map containing the behavior file. However (1) DF/F is not calculated, nor displayed. (2) We are not managing to plot the data correlated to our stimuli (PSTH). (3) When trying to export the zscore values associated with each ROI in csv format for plotting elsewhere, we get a file which contains characters instead of values. Shall I provide more details here, or shall I post an issue in github? for now, see avobe. Any help is very much appreciated :D

Kushal Kolar

@kushalkolar

This node can give you df/f with 1p data http://docs.mesmerizelab.org/en/master/user_guides/flowchart/nodes.html#staticdfofo

Kushal Kolar

@kushalkolar

@AyusoAa and the tutorial series on youtube has instructions for doing stimulus/behavior mapped analysis.

Irene Ayuso

@AyusoAa

@kushalkolar thanks! managed the DF/F, and watched again all the tutorial series. I still don't manage to export zcore or df/f data for each ROI as csv files. Help with that would be super welcome. Thanks in advance!

Kushal Kolar

@kushalkolar

@AyusoAa You can use the Save node to save the data stream at any point in the flowchart. http://docs.mesmerizelab.org/en/master/user_guides/flowchart/nodes.html#save

You can then use the Transmission.from_hdf5 to load the data and access the pandas dataframe for any further analysis. http://docs.mesmerizelab.org/en/master/user_guides/flowchart/overview.html#transmission-files

Michelle Jin

@mjin1812

Hi Kushal! This software is absolutely amazing and thank you so much for all the time and effort you put into crafting it. I'm currently piloting using mesmerize on a 1-p dataset to use cnmfe. I'm also interested in exporting the processed cnmfe data as a pickle file to potentially analyze separately. I just have a quick questions about something I noticed: After exporting and reading the saved file, I saw there was a discrepancy in the fps metadata I set for this dataset and the fr stored in the data dictionary in the parameters of the cnmf_data_dict. I was wondering which frame rate is actually being used during cnmfe?

Kushal Kolar

@kushalkolar

@mjin1812 Eric Thomson made a very nice notebook that goes through how to explore/use the cnmfe hdf5 files that are produced through the mesmerize batch manager:
https://github.com/kushalkolar/MESmerize/blob/master/notebooks/demo_batch_ingestion.ipynb

@mjin1812 for the fps discrepancy that's weird, I'm assuming you're using the GUI to set the CNMFE params? The GUI fetches the fps from the current viewer meta data and sets it as fr. If you are using the batch manager through the mesmerize scripting interface you probably have to set the fr manually for each file if it differs.

Michelle Jin

@mjin1812

Thanks for the fast response! I will definitely check out the notebook. For reference here is a link to a screenshot showing the difference in frame rates https://imgur.com/a/HMLush2

perhaps I just entered the metadata wrong. I followed the instructions only on entering it manually

Kushal Kolar

@kushalkolar

what method are you using to create the cnmfe batch items?

Michelle Jin

@mjin1812

I used the batch manager gui

Or mean the cnmfe gui to add to the batch manager

For reference, I'm using mesmerize through the VM installation

Kushal Kolar

@kushalkolar

@mjin1812 just took a look, the GUI sets the framerate at the component evaluation stage which appears to be when CNMFE uses the framerate, though I don't know why this doesn't show up in the cnmf_data_dict

Michelle Jin

@mjin1812

Thanks for your help! Does the gui automatically set the frame rate using the metadata dictionary? If so, then maybe I’ll just go with that value

Kushal Kolar

@kushalkolar

@mjin1812 The GUI does set the fps but only at the evaluate_components() step which might be why it's not showing up in the cnmf_data_dict. Anyways I'll put in a fix so that it's set at the very beginning and then it should show up in cnmf_data_dict

Michelle Jin

@mjin1812

Hi Kushal! Can I clarify whether 'cnmf_idx' in the 'ROI_states' series of a transmission object dataframe is corresponding to the roi indices in a traditional cnmf estimates object?

Kushal Kolar

@kushalkolar

yes, it's the originating cnmf.estimates.idxvalue for that ROI

Michelle Jin

@mjin1812

Thank you!

Serapio M. Baca

@smbaca

Does anyone have any experience with using mesmerize on a cluster? We have to request number of cores, RAM, GPU or no but then we have a 10TB scratch space and can use a Desktop environment running CentOS below to access mesmerize. So far, I have been able to run the motion correction but then the cnmf-e fails with not being able to find a file.

I am using Eric Thomson's instructions on the windows installation so I know that it is not a problem with the tiff movie.

Kushal Kolar

@kushalkolar

@smbaca this will probably be easiest to solve if you post an issue with all the details :)

You will probably have to set a workdir and it will need to be mounted at the same location on the compute nodes and the submission machine, but if you post an issue I can help with the details

Serapio M. Baca

@smbaca

Ok, posted the error message in the other location. Program seems to want to create a file in /var/www/ood... and that is not an installed directory.

Michelle Jin

@mjin1812

Hi Kushal! I follow instructions online to export and open up a transmission object externally in python. I detected what I think may be an indexing issue in the transmission dataframe. The "_DFOF" traces and "_RAW_CURVE" traces aren't lining up for me. I plotted the calculated dfof and raw curves for three consecutive rois using the code below where t is the transmission object and n is my roi index:

ax.plot( t.df["_DFOF"].iloc[n]);
ax.plot( t.df["_RAW_CURVE"].iloc[n]);

Here is the plot for the three consecutive rois: https://imgur.com/a/nHUJ5ly Orange = "_RAW_CURVE" blue = "_DFOF"

It looks like the DFOF traces in blue matches with the orange trace of the previous roi

Kushal Kolar

@kushalkolar

@mjin1812 I'm away until next Wednesday so I'll get back to you then.

Kushal Kolar

@kushalkolar

The paper is out now! https://twitter.com/kushalkolar/status/1460152541015121920?t=wFgSpClBaIMfFeBtrzVBDg&s=19

Eric Thomson

@EricThomson

Congratulations on some great software and a great paper @kushalkolar !!!

Kushal Kolar

@kushalkolar

@mjin1812 Did manually remove ROIs in the Mesmerize ROI manager or from the cnmf.estimate object?

2 replies

Irene Ayuso

@AyusoAa

Hi Kushal, congratulations for the paper! I was wondering if it is possible that the MC output file get the name of the orinial file instead of a collection of letters and numbers. When trying a bunch of parameters for CNMF-E, It is very slow to have to check the match between the original filename and the mc filename everytime. Coud you give some tips on how to change that in the code?

Kushal Kolar

@kushalkolar

@mjin1812 I'm not sure what was causing the issue then in that one case but it seems like it only occurred once so I'm not too concerned.

@AyusoAa I'm not sure what you're trying to do. If you use the batch manager interface you don't need to know the uuids for each item to iterate over parameter variants.

Where communities thrive

People

Repo info

Activity