AllenInstitute/AllenSDK

updates to code and data related to the AllenSDK

Amy Christensen

@achristensen56

Thanks @saskiad ! And it's always centered on the center of that region? E.g. V1 as defined retinotopically is (probably) larger than the microscope FOV!

Saskia de Vries

@saskiad

@achristensen56 Well, so the FOV is targeted to that location, but might be adjusted if, for example, there is a large blood vessel running through the FOV or something like that. So each FOV is not precisely centered on the target. But yes, V1 is definitely larger than the FOV. Most HVAs are also larger than the FOV, but to a lesser extent.

Amy Christensen

@achristensen56

@saskiad Thanks!

Yueqi

@yueqiw

Hi, I recently started using the allensdk.ephys package to analyze my own patch-clamp electrophysiology data. I find the package very easy to use for extracting cellular features from raw data. It also allows me to compare my cells with those in the Allen Cell Type Database, and I think it may help other electrophysiologists too! Since I had to modify some code to work with my data, I cloned the package to my project repository and made some changes in the code. I was wondering if this is ok in terms of open-source license stuff? What is the proper way to cite/acknowledge the allensdk library?

David Feng

@dyf

Thanks @yueqiw! I forwarded your kind comment to the person who developed that package.

David Feng

@dyf

@yueqiw We are actually in the process of refactoring this module, plus some of our internal QC code, into a separate package so it can be managed independently from allensdk. This will probably take some time, though.

@yueqiw Regarding licensing: allensdk is licensed with a non-commercial 2-clause BSD, which I believe is incompatible with GPL3 (what you have now). I will ask my legal department for the official answer. In the mean time you may just want to preserve our license.

David Feng

@dyf

@yueqiw Regarding citing allensdk: we don't have a publication to cite/acknowledge, but you are free to cite the repository itself. You can use me and @tfliss as primary authors, if you need to. If you are acknowledging the ephys analysis code in particular, also include @gouwens. Looks like there are some options for creating repo DOIs right now, but I don't have one for you at the moment.

Yueqi

@yueqiw

@dyf Thanks for the detailed answer! I have switched to 2-clause BSD license in the main project directory and copied AllenSDK's license to the folder containing the modified ephys package. Thanks for the tips on how to cite/acknowledge. We will cite the repository for publications that includes data analyzed using the allensdk.ephys module. I really enjoy using the tools developed by the Allen Institute. Thanks again for the help!

David Feng

@dyf

@yueqiw happy to help!

Yueqi

@yueqiw

@dyf I have another question. If I use the data from Allen Brain Observatory for another project, should I also follow the same non-commercial 2-clause BSD or any type of open source license?

David Feng

@dyf

@yueqiw Our data and web site have non-commerical terms of use (see http://www.alleninstitute.org/legal/terms-use/). You are free to license your own software as you like.

Yueqi

@yueqiw

@dyf Thanks!

marmaduke woodman

@maedoc

hi all. we're using Allen SDK to grab mouse connectivity data, but have an issue resulting in a <urlopen error [Errno 11001] getaddrinfo failed> when the SDK is requesting the JSON. Have you put in place rate limiting or changed the API URLs recently?

David Feng

@dyf

@maedoc I don't think so, no. Can you tell me which queries/function calls you're using to see if I can reproduce the issue? Is it always or intermittent?

marmaduke woodman

@maedoc

We're calling get_projection_matrix (cf call site) which results in this traceback

I just learned that after retrying enough times, it succeeds. we're also using deprecated classes from allensdk=0.12.4so we will update our code first.

David Feng

@dyf

@maedoc Sounds like what happens when the database is under heavy load. Updating the SDK is a good idea, but I'd be surprised if it helped in this case. Let me run a few of these and see what happens.

David Feng

@dyf

@maedoc I'm having trouble reproducing the issue. We did have someone working on our servers on 11/23 in a way that might have caused some outages, so it's possible that's what you were experiencing.

Please report back if you continue to have trouble

FrancescaMelozzi

@FrancescaMelozzi

Hi, I think that the problem that marmaduke is talking depends on the function mcc.get_structure_mask(ID)

I update allensdk to the recent version (0.14.3) however when I run that function in a loop for several ID (almost all the ids that represent a primary injection structure), after a while I get the following error:

10112017-11-30 11:55:28,942 allensdk.api.api.retrieve_file_over_http INFO Downloading URL: http://download.alleninstitute.org/informatics-archive/current-release/mouse_ccf/annotation/ccf_2017/structure_masks/structure_masks_100/structure_934.nrrd
INFO:allensdk.api.api.retrieve_file_over_http:Downloading URL: http://download.alleninstitute.org/informatics-archive/current-release/mouse_ccf/annotation/ccf_2017/structure_masks/structure_masks_100/structure_934.nrrd
2017-11-30 11:55:29,606 allensdk.api.api.retrieve_file_over_http ERROR Couldn't retrieve file /dat1/francesca/PythonScript/Allen_data/annotation/ccf_2017/structure_masks/resolution_100/structure_934.nrrd from http://download.alleninstitute.org/informatics-archive/current-release/mouse_ccf/annotation/ccf_2017/structure_masks/structure_masks_100/structure_934.nrrd (request).
ERROR:allensdk.api.api.retrieve_file_over_http:Couldn't retrieve file /dat1/francesca/PythonScript/Allen_data/annotation/ccf_2017/structure_masks/resolution_100/structure_934.nrrd from http://download.alleninstitute.org/informatics-archive/current-release/mouse_ccf/annotation/ccf_2017/structure_masks/structure_masks_100/structure_934.nrrd (request).
2017-11-30 11:55:29,608 allensdk.api.api.retrieve_file_over_http ERROR We weren't able to download a structure mask for structure {0}.
You can instead build the mask locally using
ReferenceSpace.many_structure_masks
ERROR:allensdk.api.api.retrieve_file_over_http:We weren't able to download a structure mask for structure {0}.
You can instead build the mask locally using
ReferenceSpace.many_structure_masks

We are using that function since we need an efficient way to get the volume of each injection structure, how can I fix the problem? Or there is another way to get the volume information for each structure?

thanks

David Feng

@dyf

@FrancescaMelozzi How did you come up with your IDs? 0 is not a structure in the graph. You are correct that structure 934 (ENTmv) does not have a mask, however that's because ENTmv has not been drawn in the CCF.

David Feng

@dyf

ENTmv (934) is not the primary injection structure for any experiments as far as I can tell

Can you send me a link to the code you're using? @maedoc said he was calling get_projection_matrix, which does not require structure masks, so I expect that's a different issue.

David Feng

@dyf

Here's how I checked if ENTmv is an injection structure:

from allensdk.core.mouse_connectivity_cache import MouseConnectivityCache
mcc = MouseConnectivityCache(manifest_file='mcc/manifest.json')
exps = mcc.get_experiments(injection_structure_ids=[934])
len(exps) == 0  # this is true

# just to be safe, let's crawl through the experiments
exps = mcc.get_experiments()
for exp in exps:
    injection_ids = [ inj['id'] for inj in exp['injection-structures']]
    if 934 in injection_ids:
        print(exp) # this never happens

FrancescaMelozzi

@FrancescaMelozzi

yes, you are right, I am looking at the code to understand where I am wrong

I was using the old version of datset of IDs, I used a dataset of experiments that I downloaded previously (I think that you have change the IDs of the ontology at a certain point). I am sorry

David Feng

@dyf

No problem, glad it's working for you. Let me know if there's a gap in our documentation that would have helped here.

Amy Christensen

@achristensen56

I'm trying to get properities of cells that I previously downloaded and analyzed, I have the cell and dataset IDs. I'm using cells = boc.get_cell_specimens()
cells = pd.DataFrame.from_records(cells), and then looking up by cell_specimen_id. While I can find some of the cells I had previously analyzed, a large number of cells seem to be missing. These are cells I analyzed After the june data release and relabelling of cells, so it shouldn't be that. Have a number of cells been removed from the database since ~september?

NileGraddis

@NileGraddis

We've released version 0.14.4! This update mostly consists of improvements to Python 3 compatibility.

Saskia de Vries

@saskiad

@achristensen56 There were experiments that were failed in October. If you use cells = boc.get_cell_specimens(include_failed=True) I expect you will find the missing cells.

David Feng

@dyf

@achristensen56 You can see the list of containers that were failed and why here: https://github.com/AllenInstitute/AllenSDK/wiki/Release-Notes-(0.14.3)

Amy Christensen

@achristensen56

@dyf , @saskiad thanks so much for the information!!

Out of curiousity, should we be expecting anytime in say the next few months anymore updates to which experiments were failed, or further data released? (Planning on running some time consuming analysis soon, and would rather wait if there will be more changes soon)

David Feng

@dyf

@achristensen56 No, there shouldn't be any changes like this again until the October time frame. There should be some more data release at that point.

David Feng

@dyf

@/all We're live with 0.14.5 right now. Release notes are here:
https://github.com/AllenInstitute/AllenSDK/wiki/Release-Notes-(0.14.5)
Mostly a Cell Types Database update -- many more mouse morpho-electro cells from new Cre lines, many more transcriptomics-only cells (many now with cluster labels). We also added ~80 new data sets to the Connectivity Atlas.

David Feng

@dyf

@/all And now we're live with 0.16.0!
https://github.com/AllenInstitute/AllenSDK/wiki/Release-Notes-(0.16.0)

Brain Observatory users will want to know that we updated all of our NWB files because our timestamps were off by ~30ms.

David Feng

@dyf

There's also data from three inhibitory Cre lines (Vip, Sst, Pvalb), layer 6 via the Ntrs1 driver line, and a GCaMP6s Ai94 reporter from the pan-excitatory Slc17a7 driver line.

It's about 30,000 more cells, have fun!

Also, on the Cell Types Database front, there are ~50 more mouse neurons, ~100 more human neurons (~40 with morphological reconstructions). And while it's not accessible directly from AllenSDK, there are > 30K more scRNA-Seq cells to analyze.

http://portal.brain-map.org/latest-data-release