Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 14 20:33
    ctrueden closed #367
  • Nov 14 20:33
    ctrueden commented #367
  • Nov 14 20:28

    ctrueden on master

    GatewayPreprocessor: use try/mu… Fix some instances of bogus par… (compare)

  • Nov 14 19:48
    ctrueden opened #367
  • Nov 13 21:06

    ctrueden on master

    POM: bump to next minor version… (compare)

  • Nov 13 21:05

    ctrueden on delegate-converters

    (compare)

  • Nov 13 21:05

    ctrueden on master

    Add abstract delegate converter… Add test for AbstractDelegateCo… Merge pull request #366 from sc… (compare)

  • Nov 13 21:05
    ctrueden closed #366
  • Nov 13 21:05
    ctrueden closed #182
  • Nov 13 21:04
    ctrueden commented #366
  • Nov 13 18:19

    ctrueden on master

    ServiceHelper: assign PluginInf… Gateway: default to plugin name… PluginInfo: remove unused import and 5 more (compare)

  • Nov 10 22:12
    SarenT opened #139
  • Nov 09 15:13

    ctrueden on master

    Bump to next development cycle … (compare)

  • Nov 09 15:13

    ctrueden on TrackMate_-5.0.1

    [maven-release-plugin] prepare … (compare)

  • Nov 08 20:00
    etarena commented #613
  • Nov 08 12:35

    tinevez on master

    Fix important TrackMate bug in … Add an error message in case so… Add the possibility to save a d… and 1 more (compare)

  • Nov 08 09:17
    imagejan review_requested #366
  • Nov 08 09:17
    imagejan ready_for_review #366
  • Nov 08 09:17
    imagejan edited #366
  • Nov 08 09:14
    imagejan synchronize #366
Philipp Hanslovsky
@hanslovsky
Using n5-aws-s3 should not be any different than any other N5Reader or N5Writer. Maybe n5-spark, n5-utils, n5-imglib2, or the paintera conversion helper have some inspiration for you.
John Bogovic
@bogovicj
yea, but the "other" stuff you do will be more involved (all the aws cli stuff) https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration
Philipp Hanslovsky
@hanslovsky
:plus1:
tpietzsch
@tpietzsch
bdv will soon have export to N5, just needs to be merged
is it 1 TB or many?
because for many TB, you will probably want to do it on a cluster. And for that @igorpisarev stuff might be perfect
Otherwise, if it is 1TB it might still be possible from Fiji/BDV locally
Christian "Tischi" Tischer
@tischi
Awesome that you have done the work for the Bdv! It is 8 TB if you naively load it into RAM, but since there are many black pixels, lossless compressed it is 2.5 TB. @tpietzsch would your code also support export to and reading from n5-aws-s3 or only to the standard file based n5?
Christian "Tischi" Tischer
@tischi
@bogovicj I guess I wait until Igor is back in case he already has some practical experience with aws-s3. And thanks for the pointers regarding the aws user guide. We would have the data stored locally, so I have to look at this with our IT guys.
NicoKiaru
@NicoKiaru
I have a hard time understanding the link between the SharedQueue beeing passed when using VolatileViews.wrapAsVolatile(final RandomAccessibleInterval< T > rai, final SharedQueue queue, final CacheHints hints ) and the CacheControl that I see a bit everywhere when Volatile type are involved. My main issue is to make a class that makes a VolatileSource from a non-volatile source (see here), but I don't know how to deal properly with the cache.
Christian "Tischi" Tischer
@tischi
Not sure that helps, but for me, when the RandomAccessibleInterval rai is backed by a CachedCellImg it was working fine automagically just using VolatileViews.wrapAsVolatile( rai ) without specifying any of the Queues and CacheHints..
Christian "Tischi" Tischer
@tischi
@hanslovsky @NicoKiaru @bogovicj Does any of you maybe have an idea about this issue? tischi/bdv-utils#2
Christian "Tischi" Tischer
@tischi
@NicoKiaru Thanks for posting in the issue. I think you found the "bug".
NicoKiaru
@NicoKiaru
Hi, http://fly.mpi-cbg.de:8081 do not seem to be alive. Is it temporary ? Is there another bigdataserver that can be used for testing ?
NicoKiaru
@NicoKiaru
I've just pull a PR in bigdataviewer-vistools which allows for some bdv.viewer.Sourceto be wrapped as Volatile. (bigdataviewer/bigdataviewer-vistools#38) Probably not optimal, but I think it exposes a current limitation of BdvFunctions.show(Souce src): it does wrap sources that could be wrapped as volatile. I put an example of such limitation being overcomed in this small video : https://www.youtube.com/watch?v=SdngXeEatqA. Maybe it's the wrong way to solve the limitation (exposing the SourceAndConverter is probably more general), but here it is if you want to have a look.
Curtis Rueden
@ctrueden
@NicoKiaru Bene knows. He says (via private mail) that server is permanently down, but he is working on restoring it somewhere.
Oh wait, I am talking about a different server I think. The 3D Viewer docs.
@turekg Do you have power/influence over the fly.mpi-cbg.de server?
Christian "Tischi" Tischer
@tischi
@NicoKiaru Here is a BigDataServer running, but many images, takes a while: http://cbb-bigdata01.embl.de/0.6.0
NicoKiaru
@NicoKiaru
thanks @tischi and @ctrueden !
Stephan Saalfeld
@axtimwalde
@tomancak @tpietzsch I think fly.mpi-cbg.de is not very alive, can you resurrect it?
(oh, and because there might still be some useful stuff, is it possible to get a copy of my home directory there?)
...probably asking too late...
Igor Pisarev
@igorpisarev
Hi @tischi! I'm back and can help with setting up n5-s3 if you have any questions. As @hanslovsky and @bogovicj suggested, you can use n5-spark to generate multiscale exports using an instance of N5AmazonS3Writer. The best examples would probably be the tests in n5-spark and n5-aws-s3 repositories
Igor Pisarev
@igorpisarev
n5-spark can run either locally or on a cluster, including Amazon's EMR platform (elastic map-reduce) that supports Spark. There is a wiki page in https://github.com/saalfeldlab/n5-spark/wiki that shows how to run a Spark application on AWS
turekg
@turekg
@ctrueden @axtimwalde I never heard of this machine, but I can look into it if you want...
Christian "Tischi" Tischer
@tischi
Hi @igorpisarev, thanks for offering help! Just to get started, do I need a cluster and n5-spark or could I also do it one a single machine for a small test data set?
Stephan Saalfeld
@axtimwalde
@turekg this is the old @tomancak lab application server. It used to run the original CATMAID, some databases maintained by Pavel and Radek, and the personal home-pages of @StephanPreibisch and @axtimwalde
Igor Pisarev
@igorpisarev
@tischi the n5 downsampling code is in n5-spark, but you can run it on your machine. Take a look at n5-spark repo, there are scripts for building the shaded jar and running it on a local machine. When you run locally, Spark emulates the cluster environment by using threads, and it comes as a self-contained maven dependency so you don't need to install anything extra. You can also run it in your IDE by supplying the following JVM variable: -Dspark.master=local[1] that sets the number of threads to use, or -Dspark.master=local[*] for using all cores
Christian "Tischi" Tischer
@tischi
@igorpisarev Check! I will try asap. Thanks again!!
Jan Eglinger
@imagejan
An imglib2-roi question: is it possible to sample (random) points on a sphere by taking some GeomMasks.closedSphere(...).xor(GeomMasks.openSphere(…))? And/or to get a full list of points representing the sphere surface by some Bresenham-like algorithm?
Or is using imglib2-roi for this a bad idea, and I should rather take a math function of a sphere and sample points “manually” ?
(I was thinking in the direction of ImgLib2 Example 4a, but taking a sphere surface instead of the full volume…)
turekg
@turekg
@axtimwalde must be tomancak-srv1. It is not available outside MPI-CBG. I can see CATMAID in /srv/www .
I am due to retire this machine, after moving some stuff over to bds.mpi-cbg.de and archiving the rest. Tell me where I can upload your stuff and I can get a tarball of your dir to you
Stephan Saalfeld
@axtimwalde
oh---that's great
Stephan Preibisch
@StephanPreibisch
@turekg would be great to have this of my directory as well … shouldn’t be too big, maybe just put it on google drive or so? Thank you so much!!
turekg
@turekg
@StephanPreibisch will put in on ownCloud and send you a link. It’s 39G so it will have to wait until @axtimwalde grabs his last directory off it. I’ll let you know when ready
Jean-Yves Tinevez
@tinevez
Hi all.
Does someone made Bessel functions for ImgLib2 already?
Stephan Preibisch
@StephanPreibisch
@turekg, thanks, I did not expect this amount of data … if you want you can check what the biggest things are and send me a directory listing, maybe we can delete most of it … or we have a look together via shared screen
Christian "Tischi" Tischer
@tischi
@igorpisarev May I ask couple of questions related to n5-s3?
  1. Are you using BigDataViewer to view the n5-s3 data?
  2. Do you have some experience what the optimal block size is for streaming n5-s3 data through the internet into some viewer such as Bdv?
  3. Is it possible to work with compressed n5-s3 blocks and decompress on the client side? Would this give you a better performance because you need to send less data through the web?
Igor Pisarev
@igorpisarev
Hi @tischi, for browsing I created N5 Viewer which is a BDV-based plugin that supports AWS and Google cloud: https://github.com/saalfeldlab/n5-viewer
2 - I've only used streaming through cloud platforms for smaller test projects, but I assume that the same guidelines for choosing the block size would apply there, and something like 64x64x64 would be good. I think both S3 and Google Cloud Storage charge not only for the data throughput, but also for the number of I/O requests, so you would probably want to avoid having too many tiny block files.
3 - if you use any N5 compression scheme such as GzipCompression, the blocks are always stored and sent in the compressed form
Christian "Tischi" Tischer
@tischi
hi @igorpisarev, thanks a lot for the informative answers!
tpietzsch
@tpietzsch
@NicoKiaru it should also be super easy to start a BigDataServer locally. See https://imagej.net/BigDataServer
Basically, just download the bigdataviewer.jar
and java -jar bigdataserver.jar myName myDataset.xml
@NicoKiaru regarding CacheControl and SharedQueue
CacheControl has only one method: prepareNextFrame()
This is called by BDV whenever the timepoint is changed, the viewer transformation is changed, etc
tpietzsch
@tpietzsch
It is a hint to the asynchronous loading infrastructure in the background: Data that might be enqueued for loading is not immediately needed anymore, and new requests should be prioritized. For example when moving to timepoint 2, then image tiles that were required to render timepoint 1 and are still enqueued for loading can be discarded (or bumped to a lower priority)
In general nothing bad happens if this hint is simply ignored.