Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 04 2019 22:10
    muschellij2 commented #64
  • Nov 19 2019 17:58
    dlogan commented #103
  • Nov 08 2019 15:05
    epnev commented #104
  • Nov 01 2019 18:30
    saglag commented #104
  • Nov 01 2019 17:46
    epnev commented #104
  • Nov 01 2019 17:01
    saglag commented #104
  • Nov 01 2019 16:52
    epnev commented #104
  • Nov 01 2019 16:38
    saglag commented #104
  • Nov 01 2019 14:23
    epnev commented #104
  • Nov 01 2019 03:57
    saglag opened #104
  • Sep 23 2019 14:14
    epnev closed #103
  • Sep 23 2019 14:14
    epnev commented #103
  • Sep 18 2019 16:09
    tomatoccoo opened #103
  • Aug 28 2019 19:18
    wenxuanliang commented #102
  • Aug 28 2019 14:58
    epnev closed #102
  • Aug 28 2019 14:58
    epnev commented #102
  • Aug 26 2019 21:42
    wenxuanliang opened #102
  • Aug 20 2019 02:04
    wenxuanliang commented #101
  • Aug 20 2019 02:01
    epnev closed #101
  • Aug 20 2019 02:01
    epnev commented #101
oregonGCAMP
@oregonGCAMP
figure; plot (CNM.C_df(8,:))
hold on; plot (CNM.S(8,:))
eftychios pnevmatikakis
@epnev
@oregonGCAMP I cannot see why this would be happening. If you run again CNM.updateTemporal(); and then plot(CNM.S(8,:)) do you still get all zeros?
Han Chin Wang
@HCWangTW_gitlab
Hi @epnev , I'm wondering if the current version (matlab) has any interactive way to manually remove/add ROI? From previous discussion you mentioned the ROI_GUI is not functional now, but does the manually_refine_components works for patches? I experienced some difficulty on the border of my FOV; the program will recognize part of the border as a ROI, presumably some flickering signals due to registration, so I would like to remove them. Thank you!
Yuqi Zhu
@674067482
Hello!

I'm running run_pipeline.m, but get error list below. Anyone know how to solve it? Thanks for your help!Error using parallel.FevalFuture/fetchNext (line 217)
The function evaluation completed with an error.

Error in run_CNMF_patches (line 121)
[idx, value] = fetchNext(future_results);

Error in run_pipeline_zyq_hdf5 (line 136)
[A,b,C,f,S,P,RESULTS,YrA] = run_CNMF_patches(data.Y,K,patches,tau,0,options); % do not perform deconvolution here since

Caused by:
Error using images.internal.morphop>ParseInputs (line 154)
Expected input number 1, IM, to be non-NaN.

Yuqi Zhu
@674067482
Also, does the parameters 'max_size_thr' and 'min_size_thr' of options represent the maximum/minimum number of pixels for each component?
n11xia
@naomicxia

Hello, I'm using normcorre_batch/normcorre_batch_even to perform motion correction. For a 512x512x80,000 .mat file, I'm using bin_width of 30,000 and it is really slow even on a 40core computer with 500G memory. I started with parpool(20) but it appears to call 12 more workers midway (and errors because it is unable to do so) and eventually gives the following error. Is it possible that the larger memory requirement is somehow causing parallel workers to timeout? I don't have the same problem with smaller binwidths. Thanks so much for your help!

p = parpool(20)
Offset 4.0e-01 pixels due to bidirectional scanning detected.
Registering the first 10000 frames just to obtain a good template......done.
Template initialization complete. Now registering all the frames with new template.
The file was saved successfully. Elapsed time : 1140.249 s.
30000 out of 82521 frames registered, iteration 1 out of 1..Starting parallel pool (parpool) using the 'local' profile ...
connected to 12 workers.
Starting parallel pool (parpool) using the 'local' profile ...

Error using parfeval (line 58)
Parallel pool failed to start with the following error. For more detailed information, validate the profile 'local' in the Cluster Profile Manager.

Error in normcorre_batch_even (line 255)
future_results(i) = parfeval(@register_frame, 2, Yt,fftTempMat,fftTemp,patches,options);

Error in twostep_moco_applyshifts_clusterMAT (line 102)
[~,shifts,template] = normcorre_batch_even(fullname_R,options_mc,template);

Caused by:
Error using parallel.internal.pool.InteractiveClient>iThrowWithCause (line 676)
Failed to initialize the interactive session.
Error using parallel.internal.pool.InteractiveClient>iThrowIfBadParallelJobStatus (line 790)
The interactive communicating job failed with no message.

IdleTimeout has been reached.
Parallel pool using the 'local' profile is shutting down.
The client lost connection to worker 1. This might be due to network problems, or the interactive communicating job might have errored.

eftychios pnevmatikakis
@epnev
@naomicxia It would be better to deal with this if you open an issue. I can suggest the following:
  • Use a much lower bin_width parameter, e.g. 3000
  • Do not use memory mapped files for your output (if you’re using one)
  • HDF5 files have the best performance for input
eftychios pnevmatikakis
@epnev
@674067482 Yes, min_size_thr and max_size_thr do refer to the minimum/maximum number of pixels for each component.
For your error, please open an issue on github with the complete error log, and I will take a look.
Yuqi Zhu
@674067482
@epnev I have pasted my error log on github(flatironinstitute/CaImAn-MATLAB#95), many thanks for your help!
martinjendryka
@martinjendryka
Hello , Im running into the error below after running the run_pipeline.m script. apparently the dimensions of the variables buf1ft.*conj(buf2ft) are not identical. What causes that? How can I solve the problem? The h5 files were created for all my avi files. Im using the script for miniscope recordings at 20 Hz.
Error using parallel.FevalFuture/fetchNext (line 217) The function evaluation completed with an error.
Error in normcorre_batch_even (line 258)
[idx, shifts_temp, Mf_temp] = fetchNext(future_results);
Error in run_pipeline (line 36)
[M,shifts,template,options_mc,col_shift] = normcorre_batch_even(fullname,options_mc,template);
Caused by:
Error using dftregistration_min_max (line 95)
Matrix dimensions must agree.
eftychios pnevmatikakis
@epnev
@martinjendryka You should make sure that the folder where you’re reading your files from contains only the files you want to analyze that each file has more than one frames. Also bear in mind that this script is for 2p data. For 1p either use the CNMF-E package or the Python version of caiman.
martinjendryka
@martinjendryka
@epnev thanks for your quick suggestions. I will try that!
jingxie0425
@jingxie0425
Hello, I try to correct images with rotations with NoRMCorre. I ran the file demo.m demo_1p.m ... I found the rotations seems not corrected. All of the rotations are kept but the simple motion translation is recovered. Anyone knows why?
eftychios pnevmatikakis
@epnev
@jingxie0425 NoRMCorre is not designed to directly correct for rotations. It can approximately indirectly correct for those if you use piecewise rigid registration see lines 77-81 in the demo_1p file with a relatively small value for the grid_size parameter. Have you tried that?
jingxie0425
@jingxie0425
@epnev Thanks! My image size is 400400. I found the value of the grid_size parameter is [128, 128]2. I wonder how small it should be? Besides, my input has only one channel, so should the "*2" be eliminated?
eftychios pnevmatikakis
@epnev
@jingxie0425 Yes, and you can try even lower than [128, 128]
jingxie0425
@jingxie0425
@epnev I just tried with [32, 32], it still doesn't work. Can I try very small one like [3, 3] or [5, 5].
eftychios pnevmatikakis
@epnev
That was fast. How many frames is your movie?
jingxie0425
@jingxie0425
@epnev 1800 frames.
eftychios pnevmatikakis
@epnev
@jingxie0425 Values smaller than [32, 32] are unlikely to help although I’m surprised your code run so fast. It should take a few minutes or so
jingxie0425
@jingxie0425
@epnev Oh, I tried before using [32, 32] in another demo. It took me more than 10 min... hhh. I mean, since the [32, 32] still doesn't work, can I say that NoRMCorre doesn't work for the simulated rotations I made? Or is there other parameters related to this issue?
Another questions: I saw the rotations are simulated in the experiments in your paper title={NoRMCorre: An online algorithm for piecewise rigid motion correction of calcium imaging data}. Could you tell me more details about it? For example, how large is the rotation angle you applied?
eftychios pnevmatikakis
@epnev
@jingxie0425 The supplementary material in our paper has the information about the rotational example we used. As I said, it does not natively correct for rotations but in many cases it works just fine as it was shown in the paper and also seen in practice.
eftychios pnevmatikakis
@epnev
@jingxie0425 A couple of things I didn’t mention: 1) using options.shifts_method = ‘cubic’ is more appropriate for correcting rotations as it uses a higher resolution motion field. This will not be done automatically since the deafult value is FFT. 2) The demo_1p file is for one photon data so use that only if your data is one photon. 3) The values of overlap_pre should also be set to a smaller value as it is included in the patch size. You can try [16, 16] for example or even a bit smaller if you don’t have a lot of motion.
jingxie0425
@jingxie0425
@epnev Thanks!
jingxie0425
@jingxie0425
@epnev Is the demo.m for two-photon dataset?
eftychios pnevmatikakis
@epnev
yup
jingxie0425
@jingxie0425
I have a simulated dataset generated from neuron mask. But it's neither one nor two photon. Does it matter?
eftychios pnevmatikakis
@epnev
The only additional step for one photon is that it performs spatial high pass filtering prior to computing the shifts to remove some of the background. Depending on what your situation looks like, you can use either.
jingxie0425
@jingxie0425
@epnev Thanks!
n11xia
@naomicxia
How does the bidirectional offset correction work? It seems to detect it in the beginning of each dataset but what happens if the bidirectional offset changes throughout the video? is there a way to make the program aware of this possibility?
eftychios pnevmatikakis
@epnev
@naomicxia It detects it once and then applies to the whole movie. The offset is assumed to stay constant throughout the recording.
AdiDoron
@AdiDoron
Hi, I saw that you mentioned that the GUI is no longer compatible with the rest of the pipeline. Is there any way to manually add/remove ROIs (based on their centroids for example), without the GUI? Thanks!
saglag
@saglag

Hi @epnev , I'm trying to implement the MATLAB code and am trying to run through the demo script but i'm getting the following errors:
Error in montage/loadimages (line 1084)
M.im(nprev+i) = fn_structmerge(model,simi);

Error in montage (line 43)
loadimages(M,varargin{:})

Error in demo_script (line 90)
subplot(121); montage(extract_patch(A(:,keep),[d1,d2],[30,30]),'DisplayRange',[0,0.15]);

jingxie0425
@jingxie0425
Hi, we are trying to detect neurons in our own simulated data. The MATLAB code works well on no-noise dataset. Nearly all neurons are detected. But when we added temporally autocorrelated zero mean noise with standard deviation σ = 0.4 as well as shot noise with zero mean and σ = 1.0, no matter how we decrease the noise intensity, the recall and precision dropped to about 50%. Too many false positives occurred. It seems some neurons couldn't be detected even if a large K (number of neurons to be detected) applied.
jingxie0425
@jingxie0425
We've checked the matlab code step by step. The problem occurred in the component initialization. When the greedyROI is applied, there are too many false positive neuron centers. Also nearly half neurons in ground truth is not marked. We tried the alternative method (greedy_corr, sparse NMF, HALS). Neither of them works better than greedyROI which is the default method.
We set most of the parameters as default value. Could you please give us some suggestions of the parameter settings? Especially when the input frames has temporal noise.
eftychios pnevmatikakis
@epnev
@jingxie0425 It’s hard to say in a simulated scenario. A lot of the false positives are dropped during the component screening where a bunch of tests is performed. If you increase the noise level you may want to adjust the parameters for those tests. However, temporally autocorrelated noise is not something that occurs often in practice, especially at high levels. The algorithm assumes temporally uncorrelated noise
jingxie0425
@jingxie0425
@epnev Thanks for your reply! The colored noise we added in the simulated dataset is autocorrelated. Maybe that's the reason.
jingxie0425
@jingxie0425
@epnev We tried cases without autocorrelated noise (with only shot noise, very low level), but the same issue occurred. Half of the neurons are not detected and half of the detected ROIs are false positives. I wonder if there is some related params that were given wrong value?
eftychios pnevmatikakis
@epnev
@obarnstedt_twitter That looks better although probably some parameter tuning might be required for deconvolving F_dff. The estimates object has a deconvolve method that you can use. If you set the dff_flag=True then it would perform it on the DF/F traces.
Alisa Omelchenko
@AlisaOmel
Hello! I'm new to using NormCorre, and I'm having trouble figuring it out. I use a ZEISS LSM 880 Confocal for calcium imaging on live samples which move. This makes it difficult to analyze the fluorescence. I am trying to rum motion correction on the stacks of images, to make sure that the neuron/fluorescence change stays in the same relative place. The output comes out as a singular TIF for each time point(88) /z position (15 slices). I used matlabs imread/imwrite to create 15 larger TIF files with all the 88 time points. Is this a correct input for the motionCorrectSerial call? I get an output "motion_corrected.tif" but I cannot open it like I open regular tifs and I don't think it is a modified version of the input. I am very new to matlab, so please let me know if I am doing something wrong. This is the test script I am running:

name = imread('horizontalstack1.tif');

MC = MotionCorrection(name);
options_rigid = NoRMCorreSetParms('d1', MC.dims(1),'d2', MC.dims(2),'output_type', 'tif', 'bin_width',50,'max_shift',15,'us_fac',50,'init_batch',200, 'nFrames', 88);
MC.motionCorrectSerial(options_rigid);

And this is the output:
Offset -3.5e+00 pixels due to bidirectional scanning detected.
Cubic shifts will be applied.
Registering the first 3 frames just to obtain a good template......done.
Template initialization complete. Now registering all the frames with new template.
The file was saved successfully. Elapsed time : 0.170 s.

done.

eftychios pnevmatikakis
@epnev
@AlisaOmel I believe the function has worked since there are no errors. You can load the file in memory by using the read_file function or you can load in FIJI/ImageJ
@jingxie0425 You can try adjusting the parameters related to the screening tests, like min_SNR, space_thresh or cnn_thresh. Other than that it’s hard for me to say having looked at no images.
Alisa Omelchenko
@AlisaOmel
@epnev When I read the file there are some values, but when I open it with Imread/imshow it's just black. When I try to open it with ImageJ it says "File is not in a supported format, a reader plugin is not available, or it was not found". This is not the case for the original images. Also, I can open the original stack with preview and all the images are there, while this one is not compatible and gives an error. @jingxie0425 I don't see the explanations of those in the NoRMCorreSetParms.m file. What are the standard values so I know how to adjust them? Thanks for the help!
eftychios pnevmatikakis
@epnev
@AlisaOmel Like I said, use the 'read_file' function to load them in memory and inspect them
Rachele Catalano
@Rack95
Hello! I am using CaImAn on my dataset. The calcium analysis is relatively easy (thanks to your code!) since I am only tracking one single neuron (on C.elegans), but I need to account for an heavy motion artifact. I tried to perform the analysis with NormCorre both on Python (within the same pipeline) and Matlab (to try 1p data script) with similar results: all the traslational motion (animal moving back and forward) is corrected perfectly (max_shift parameter set to 100, FOV 512x512 and pixel size 6 micrometers), but additional deformations result in an apparent rotation of the neuron once the video is corrected with rigid motion correction. I use the rigid motion correction results to perform non-rigid motion correction afterwards (rather than the original file, uncoupling the two problems leads me to better results), but despite tuning the 'max_shift' and 'max_dev' parameters in the second step, I get no further improvement with respect to performing only the rigid motion correction. Is there any other parameter which might make sense changing to account for this "rotation" (<=20º) or any value I may try? Until now I only made guesses starting from values equal to [20,20] and [8,8] respectively. All the other parameters are set to the default value. Thank you in advance for any help you can provide.
eftychios pnevmatikakis
@epnev
@Rack95 I’ll have to see some figures to understand better what the problem is. NoRMCorre does not correct for rotations automatically but it can approximate them in a piecewise linear fashion. One way to make this approximation finer is by reducing the size of the patch used during piecewise motion correction.