Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Jeremy Freeman
    @freeman-lab
    @philippberens pretty sure it's totally fixed!
    Jeremy Freeman
    @freeman-lab
    (famous last words...)
    @all see above if you were having trouble
    seems to have been a error in file content not getting caught correctly
    didn't show up till now cause we hadn't seen these particular errors
    Philipp Berens
    @philippberens
    thanks all and especially @freeman-lab for taking care of this
    @/all new deadline may 4th! should give everybody who wanted to submit the chance to do so.
    Thomas McColgan
    @phreeza
    Wow looks like some powerful entries were holding out until the last moment
    thomasdeneux
    @thomasdeneux
    The landscape has changed a lot, but who is overfitting?, that is the question!!
    Philipp Berens
    @philippberens
    indeed, that is one of plots i am looking forward to making - delta performance on training /test set vs. submission time ;)
    Peter Rupprecht
    @PTRRupprecht
    My guess is that the convolutional networks are not really easily overfitting the training data. I tried fitting a single neuron with a CNN, and it could still nicely predict other neurons' spikes. However, the optimal temporal filters are different for the different datasets, and I guess that this point (choice of a filter set for a specific dataset) is where overfitting can happen. But I'm curious as well ...
    Thomas McColgan
    @phreeza
    Any decision on what the timezone for the deadline is?
    Philipp Berens
    @philippberens
    dear @/all - thanks a lot for your participation and your many submissions. I think this was a great success demonstrating nicely how powerful such competitions can be for advancing the state of the art and how important benchmark comparisons are.
    i will try to analyse the results quickly and announce the winners!
    Dario Ringach
    @darioringach
    @philippberens ...are you trying to build suspense? :-)
    Philipp Berens
    @philippberens
    sorry about the delay...
    Philipp Berens
    @philippberens
    /all here are the results:
    top10 submissions.png
    this is the mean of the correlation values for the five test datasets
    however, a weighted mean (by number of cells per datasets) gives slightly different ordering of the top 3
    top5 submissions weighted.png
    therefore:
    we have three winners: Patrick Mineault, Nikolay, Peter&Stephan! Congratulations. The absolute difference between these is less extremely small, so i think it is fair to award three first prizes
    Philipp Berens
    @philippberens
    paper.png
    all these submitters were better than STM, the state of the art, so these submissions qualify for being included in a review/overview paper of the challenge
    Philipp Berens
    @philippberens
    i sent you a separate email
    Thomas McColgan
    @phreeza
    Thanks, @philippberens . great to see the final results! I hope this kind of contest becomes a more regular thing.
    Philipp Berens
    @philippberens
    thanks and i agree
    thomasdeneux
    @thomasdeneux
    Hello, great to see the result. Lots of deep learning nowadays...
    thomasdeneux
    @thomasdeneux
    Yes such challenge is boosting the state of the art. I am also looking forward to the discussions that we will have about all this.
    Peter Rupprecht
    @PTRRupprecht
    @philippberens I've got a suggestion for something to analyze for the paper: One could select for each neuron in the test (or also training) dataset the submission that was best at predicting the spiking of this very neuron. Then order the neurons according to this highest correlation value (which tells roughly how well overall this neuron can be predicted) - neurons of which the spikes can be easily predicted would be plotted to the right hand side, and neurons of which the spikes cannot be well predicted to the left. Plotting the performance of the different models would then visualize that e.g. model 1 is exceptionally good at fitting neurons that are easy to fit, and model 2 is exceptionally good at fitting neurons that are difficult to fit, etc.
    Of course, if the result of the plot is just a mess, better discard it, but I'd be curious to see this, if it is not too much work. I hope I have explained this well enough (otherwise I could make a small dummy figure for visual explanation).
    Philipp Berens
    @philippberens
    yes, that sounds like a good idea!
    thanks
    once I have the traces I can do it
    Thomas McColgan
    @phreeza
    Since there seem to be a number of neural networks involved, should we agree on some common way of visualizing the architectures?
    Peter Rupprecht
    @PTRRupprecht
    Hi @phreeza : Agree. I have done it like this, and if you like it you can use it as well: https://www.dropbox.com/sh/oi1c1zy9dxichaj/AADuZe6z_sx8vHWAAdfs3Gffa?dl=0
    The numbers in the boxes indicate the output sizes. But I'm open to any other ideas ...
    Alex Klibisz
    @alexklibisz
    Hi, I'm starting to look at the spikefinder data and some of the models to hopefully apply them to our own datasets, but it's not obvious to me how a neuron from a series of calcium images (TIFFs) gets transformed into a single column vector? Maybe it's something as simple as taking the mean of all pixels in the neuron at each time step? Is the code for doing this transformation available somewhere? I appreciate any tips.
    Alex Klibisz
    @alexklibisz
    In other words, how do you go from TIFFs to calcium trace?
    Peter Rupprecht
    @PTRRupprecht
    Hi @alexklibisz , this is typically done as you have described it:
    1) Take the mean of all pixels in the neurons (ROI) at each time step. The ROI is drawn either manually (you could use e.g. ImageJ's ROImanager) or using a more elaborate semiautomatic/manual/automatic approach (there are many: http://neurofinder.codeneuro.org/). For starters, manual ROI-drawing should be fine.
    2) For the spikefinder competition, the timetraces (single column vectors) have been normalized. The normalization process follows Theiss et al. (2016) and can be found here: https://github.com/lucastheis/c2s/blob/master/c2s/c2s.py under the method "preprocess". In the end it is just a sort of normalization, and Philipp could give you more details on that ...
    One thing to add: When using TIFFs to extract calcium signals, you might want to subtract an offset (if there is any). After that, the signal is averaged over the ROI, and the dF/F0 is calculated. If you do not know what that is, check this out: https://www.quora.com/Why-is-the-equation-%CE%94F-F-used-to-measure-changes-in-fluorescence-intensity ;)
    Plus, many datasets suffer from background contamination. Therefore, many people perform a 'neuropil subtraction' of the neuron-timetraces. However, this was not done for this competition, if I'm not wrong.
    Alex Klibisz
    @alexklibisz
    @PTRRupprecht thanks for the detailed answer! That helps a lot.
    Alex Klibisz
    @alexklibisz

    @PTRRupprecht I'm still a little confused on the details, so to make sure I understand it correctly I'll describe the steps and indicate where I'm unsure. If you could correct me that would be very much appreciated.

    Given an ROI from a series of TIFFs (e.g. one of the neurofinder datasets), I would take the following steps to get a spikefinder format column vector:

    1. Subtract an offset -- what constitutes an offset? Do I want to subtract such that the min is 0, the data is centered at 0, etc?
    2. Average all pixels at each timestep.
    3. Transform each value F_t as F_t = (F_0 - F_t) / F_0 -- when I look at the datasets without preprocessing (https://goo.gl/GoEPfH), F_0 is much larger than the remaining values, but this isn't the case with the Neurofinder data, which makes me think I'm still misunderstanding this step.
    4. Call the preprocess function from c2s with fps/Hz = 100.

    Is there maybe a snippet of code somewhere for converting a Neurofinder ROI into Spikefinder format?

    Jérôme Tubiana
    @jertubiana
    Dear all,
    I’d like to download the raw, unprocessed data set. However, the link http://spikefinder.codeneuro.org/assets/nopreprocessing.zip does not work. Can someone provide me with a new link ?
    Best regards,
    Jerome Tubiana
    Peter Rupprecht
    @PTRRupprecht
    Hi Jerome, Philipp mentioned it somewhere above, it's on Dropbox (but only for datasets 1-5): https://www.dropbox.com/s/ugmet5eky8g0l5n/datasets%201-5%20nopreprocessing.zip?dl=0
    Best,
    Jérôme Tubiana
    @jertubiana
    Hi Peter,
    Thanks for your answer. - the first 5 dataset look good to me. For the other one, I suppose I can use the original links online ?
    Best,
    Jerome
    Peter Rupprecht
    @PTRRupprecht
    Yes, I think so : )
    Kyle
    @kr-hansen
    I had a question regarding these data. I know the competition is over, but I was playing around with the data for a class. The training dataset 07, with GCaMP6f data, has 37 cells in it. However, the source data provided on CRCNS says there were only 11 GCaMP6f cells in that dataset (cai-1). Is there a reason for the discrepancy?
    I could see a couple plausible things, but I thought I'd ask here to see why that might be. Are the cai-1 recordings really long, so they were split up for SpikeFinder (So in reality the 37 cells come from those same 11 cells), or were additional datasets beyond what the documentation of cai-1 says provided for SpikeFinder?