Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Aug 15 23:15
    mkitti commented #126
  • Aug 15 22:55
    jakirkham commented #1104
  • Aug 15 22:47

    jakirkham on pip

    (compare)

  • Aug 15 22:47

    jakirkham on main

    Bump numpy from 1.23.1 to 1.23.… (compare)

  • Aug 15 22:47
    jakirkham closed #1109
  • Aug 15 22:17
    MSanKeys963 commented #14
  • Aug 15 17:18
    MSanKeys963 commented #1109
  • Aug 15 16:37
    tasansal commented #720
  • Aug 15 16:36
    tasansal commented #720
  • Aug 15 16:28
    tasansal commented #720
  • Aug 15 16:27
    tasansal commented #720
  • Aug 15 16:24
    tasansal commented #1105
  • Aug 15 10:04
    jstriebel synchronize #152
  • Aug 15 09:16
    jstriebel synchronize #152
  • Aug 15 04:31
    codecov[bot] commented #1109
  • Aug 15 04:29
    codecov[bot] commented #1109
  • Aug 15 04:16
    dependabot[bot] labeled #1109
  • Aug 15 04:16
    dependabot[bot] opened #1109
  • Aug 15 04:16

    dependabot[bot] on pip

    Bump numpy from 1.23.1 to 1.23.… (compare)

  • Aug 12 16:22
    jstriebel edited #152
Josh Moore
@joshmoore
What happened the day before yesterday?!? https://pypistats.org/packages/zarr
jakirkham
@jakirkham
We made a release?
Josh Moore
@joshmoore
Don't think so. That's an almost 100x outlier. And it holds for other packages as well: https://pypistats.org/packages/anndata
Isaac Virshup
@ivirshup
image.png
image.png
Also dask, but not xarray?
Strong suspicion someone in single cell has a very high AWS bill
Josh Moore
@joshmoore
Egads
Ryan Abernathey
@rabernat
This is fascinating
Sanket Verma
@MSanKeys963

Reminder we have our bi-weekly community call tonight @ 23:30 IST/20:00 CEST/19:00 BST/14:00 EDT/11:00 PDT. 🔔

If you want to discuss any topic at the meeting or want the Zarr community to discuss it please add it to the agenda here: https://hackmd.io/tFcffc__QRuo1UdGcGUbsg

Link to join: https://zoom.us/j/300670033?pwd=OFhjV0FHQmhHK2FYbGFRVnBPMVNJdz09

Also, feel free to go through the meeting notes from the previous community call here: https://github.com/zarr-developers/community-calls
Ryan Abernathey
@rabernat
Who from our community is at SciPy right now?
8 replies
Josh Moore
@joshmoore
A heads up that both Sanket and I will miss the regular call tomorrow. I assume with SciPy going on, several others will as well. Shall we cancel?
jakirkham
@jakirkham
Yeah that sounds like a good idea
Ryan Abernathey
@rabernat
yes
Jonathan Striebel
@jstriebel
Unfortunatly I won't make it to todays community call. I think the progress on the v3 spec review is great! IMO one important thing to clarify is if the different extensions are part of ZEP001 or not. I think it's fine to have those separate if that makes review and agreement easier. If that's the case, I'd open another ZEP for the sharding extension soon.
jakirkham
@jakirkham
Running late
Davis Bennett
@d-v-b
the linux testing CI jobs have started taking 40+ minutes
22 replies
up from 13-17m last week
jakirkham
@jakirkham
On a different note @martindurant it would be interesting to learn more about the work you are doing with compressors
3 replies
Martin Durant
@martindurant
I am not doing too much with compressors just for the moment, except that kerchunk keeps coming on file formats including codecs that we don't have. Latest: "rice" compression in FITS images. But I don't mean to implement them unless there's a really compelling reason.
But I also have a dream of using kerchunk to do the equivalent of what is now in the blosc partial load: several compression formats can be loaded block-wise, and of course uncompressed has complete freedom; so I'd really like to be able to pass the chunk start/end down to the storage layer and let it only fetch the bytes needed.
(or maybe it's "frame" I want, not "block"...)
jakirkham
@jakirkham
Maybe PR ( zarr-developers/zarr-python#1096 ) has provides some inspiration?
Craig Russell
@ctr26
What’s the current state of using zarr in async with pyiode?
Martin Durant
@martindurant

@ctr26 : you cannot. Zarr calls the storage classes in sync code. Although the storage classes might have async internals (like fsspec), that is done using a separate thread. We could, in theory, rewrite zarr to bubble up async IO calls when doing arr[:], and this might be worthwhile, but some considerable work.
The only thing you could do right now, would be to download all of the zarr chunks you will need to the virtual filesystem presented by pyodide using JS's async HTTP calls, and then read that with zarr in sync-only mode.

Hm, that gives me an idea! Maybe we can wrap a zarr array with an async zarr array that does exactly those two stages and then allows you to do z await arr[:]?

6 replies
Davis Bennett
@d-v-b
i would love this
eager execution + TB-sized data isn't a great fit to be honest
but wrapping zarr arrays in dask can introduce fatal performance problems
28 replies
Martin Durant
@martindurant
Also note that you can already read zarr from http sources using the zarr JS library, you don't need python just to get and display the arrays.
Craig Russell
@ctr26
I’m happy to contribute to getting this to work
kshitijv09
@kshitijv09
@kshitijv09
Hello Everyone,
I am Kshitij Verma pursuing Computer Science and Engineering from Indian Institute of Information Technology Tiruchirappalli, India. I would really like to start contributing to the Zarr community but don't understand how and where to begin.
I have read the contributing guide and the Zarr tutorials and have familiarized myself a bit with the package and am wondering what to do next.
I was wondering you guys could help me on how to start contributing.
2 replies
Martin Durant
@martindurant
On pyscript/pyodide (or async zarr in general), I'll write up my hackish design maybe on Tuesday.
1 reply
Sanket Verma
@MSanKeys963
ICYMI, here's our recent blog post reflecting on CZI's EOSS4 grant: https://zarr.dev/blog/eoss4-roadmap/
Tweet: https://twitter.com/zarr_dev/status/1554193857335554048
Isaac Virshup
@ivirshup

Hey all, are there any meeting notes from the zarr–ASDF meeting at scipy?

The reason I ask is that there's some movement from graphblas on moving beyond matrix market for an on disk format. ASDF, HDF5, and zarr have come up as possible formats.

20 replies
Josh Moore
@joshmoore
Just a heads up that I'll be getting a 2.13.0a1 together with PRs from @d-v-b, @madbk, and @jstriebel.
1 reply
Davis Bennett
@d-v-b
cool, lmk if there's anything you need
4 replies
also, I would be curious to hear from anyone about how cut down on the code duplication in n5.py -- as has been suggested before, the clean way to implement an N5Store is by wrapping an existing store with an n5 compatibility layer. But I couldn't see a simple way to express the zarr -> n5 transformation, so instead we have two implementations (N5Store and N5FSStore) that just use inheritance. It's not very satisfying, and there's a lot of code duplication
7 replies
Jonathan Striebel
@jstriebel
To add sharding to the v3 spec I added another ZEP:
zarr-developers/zeps#13
This also comes with a PR for the spec:
zarr-developers/zarr-specs#152
I'll also provide an implementation PR for zarr-python next week. @joshmoore @MSanKeys963 et al, is this the right way forward for sharding? Please let me know if there is something missing or unclear in the ZEP. I assume the sharding spec PR will be under review until the ZIC decides upon the ZEP, right?
7 replies
Norman Rzepka
@normanrz
@joshmoore, John Bogovic and I are writing a paper about (OME-)zarr in the bioimaging (more specifically Volume EM) community. We are looking for citable papers where zarr has been used in other fields of research. Please send us your references and we'll try to incorporate them as use cases in our paper. Thanks!
10 replies
Jeremy Delahanty
@jmdelahanty

Hey everyone! I was curious if anyone knows about how to run optical flow algortihms on zarrs/H5 files with Dask. Our lab is using SLEAP to process some animal behavior and it uses opencv's flow calculation to maintain identities of subjects. You can see what I think is relevant here. It takes quite long time to run on their .slp files, which are based on H5 from what I can tell.

This seems like a great usecase for zarr and dask so read/write can be done in parallel to these files! Are there any examples of using dask for this kind of thing that any of you know about?

6 replies
Sanket Verma
@MSanKeys963
Hi everyone! 🙋🏻‍♂️
Just wanted to share a small update. All the info. related to Zarr bi-weekly community calls is available at the new URL: https://zarr.dev/community-calls/
The new webpage also offers a search bar at the top which can be used to search discussion topics from previous community meetings.
Also, feedback are most welcome for this. ^
Martin Durant
@martindurant
Someon please remind me, when using zarr.open_group with a URL, how do I specify that I expect the target to have "/" as the array key separator rather than "." ? (v2)
6 replies
Sanket Verma
@MSanKeys963
Reminder we have our bi-weekly community call tonight @ 23:30 IST/20:00 CEST/19:00 BST/14:00 EDT/11:00 PDT. 🔔
Agenda, joining instructions and previous meeting notes available here: https://zarr.dev/community-calls/
Martin Durant
@martindurant
Screen Shot 2022-08-10 at 13.34.03.png
^ async zarr inside pyscript
Martin Durant
@martindurant
Josh Moore
@joshmoore
@martindurant: :100: cc: @ctr26