by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Stephan Saalfeld
    @axtimwalde
    when I pop the highest element, the next pop should give me the second highest element, i.e. popping through the entire heap would deliver an ordered list
    Curtis Rueden
    @ctrueden
    I hate that Java's memory management is called "the heap". It makes it functionally impossible to web search for implementations of heap data structures. :disappointed:
    Stephan Saalfeld
    @axtimwalde
    yeah, very ugly
    I also hate that there is no proper struct handling in old java
    they did introduce it though recently, I think, didn't they?
    it'd be so much easier to use proper object logic instead of hacking some magic over primitive arrays or buffers
    Curtis Rueden
    @ctrueden
    https://pdfs.semanticscholar.org/2fc3/a70b3eaf526f7ec51d5be01c8164ca14e80c.pdf :arrow_left: Sounds promising, but no stated license or source code links at first glance. Grr.
    Curtis Rueden
    @ctrueden
    Oh yeah, Valhalla. :+1:
    Michael Doube
    @mdoube
    @axtimwalde take a look at Eclipse Collections, it's pretty full featured and actively developed (and previously well resourced by GS). I don't know without investigating your particular use case, but I understand maps of primitives is what it is meant to be good at.
    Michael Doube
    @mdoube

    I want to update existing ids with a new count (actually I want it to be a float)

    EC seems to handle most primitive pairs: long+float, long+int are in there:

    Michael Doube
    @mdoube
    @axtimwalde Sounds like a sorted tree map? Something like Trove's TDoubleLongMap would give you a memory-efficient hash from double to long... but sadly Trove doesn't include any tree-backed data structures from what I can tell
    EC has sorted tree maps, see the forum post for performance data.
    I will work on a PR
    Stephan Saalfeld
    @axtimwalde
    Thanks @mdoube !
    Hmmm... the tuple pairs are objects and the collections of those are collections of objects
    Stephan Saalfeld
    @axtimwalde
    I am after a solution that stores tuples of primitives in a set of primitive arrays juts like the primitive sets or lists in Trove, fastutils, EC do
    In the spirit of storing a list of 3D double precision points as 3 n-element arrays of doubles instead of one n-elements array of 3-element arrays of doubles, because it saves n - 3 * 52 bytes.
    Stephan Saalfeld
    @axtimwalde
    or, to avoid cache misses, chunks of reasonable size, but that's details
    Michael Doube
    @mdoube
    I made a PR for scijava/pom-scijava, is there anything else to add, @ctrueden ?
    (the pathway from declaring dependencies in pom.xml to having artefacts available on the updater is still a bit obscure to me)
    Michael Doube
    @mdoube

    In the spirit of storing a list of 3D double precision points as 3 n-element arrays of doubles instead of one n-elements array of 3-element arrays of doubles, because it saves n - 3 * 52 bytes.

    Good point. http://dx.doi.org/10.1145/3030207.3030221 this may help, certainly Object-bloat reduction & reduced memory footprint is one of the main goals of these projects.

    Stephan Saalfeld
    @axtimwalde
    Right, but there are no struct collections in any of those projects, or am I missing it?
    there are single primitive collections and primitive maps (sometimes in combination with objects for completeness)
    Imagine a set of a struct of {int, double, double, long} whose hash method uses the int
    Michael Doube
    @mdoube
    No idea, sorry, my use case was much simpler, just lots of .add() of int to a HashSet
    Stephan Saalfeld
    @axtimwalde
    or, for my use case, a sorted hash set of structs {long, float} that is sorted by the float and uses the long to do the hash and equals check
    Ok, thanks for this comparison though, very interesting!
    I always used Trove because simple and complete, but it doesn't look to good here.
    Michael Doube
    @mdoube
    ja, I think that Trove has been 'Eclipsed'
    Stephan Saalfeld
    @axtimwalde
    Right, and after all this time, nobody has collections of tuples of primitives?
    I am shocked
    Stephan Saalfeld
    @axtimwalde
    Well, nobody is not correct, there are mastodon-collections
    Curtis Rueden
    @ctrueden

    @mdoube wrote:

    I made a PR for scijava/pom-scijava

    Thanks! :sunny:

    the pathway from declaring dependencies in pom.xml to having artefacts available on the updater is still a bit obscure to me

    It's described on the Development Lifecycle page. About when it will be available from the Java-8 update site, the next steps would be:

    1. Update sc.fiji:fiji to depend on it. This could be directly, but more often happens because a shipped plugin uses it.
    2. Manually upload the latest component versions corresponding to the fiji/fiji master branch. We do this infrequently, because we want to test combinations of artifacts before uploading to avoid component version skew. But exceptions can be made if you need the changes urgently.

    The future goal is to replace (2) with Travis CI doing the uploads automatically based on either: A) the fiji/fiji master branch; or B) the latest release version of sc.fiji:fiji.

    The future goal beyond that is to get rid of the Java-8 update site in favor of a core ImageJ update site and core Fiji update site, like before, so that we can have a "plain" ImageJ2 without all the Fiji stuff.

    The future goal beyond that is to get rid of core update sites in favor of using Maven artifacts on demand, via a remote resolution mechanism in the Updater similar to how Maven works. We'd still support the old-style update sites for compatibility, but only for 3rd party extensions. The "Fiji" application is expressible as net.imagej:imagej+sc.fiji:fiji in Maven terms—we just need an updater smart enough to understand that.

    Steps we will do before (1) above include:

    • Merge the PR.
    • Release a new pom-scijava.
    • Update fiji/fiji to extend that version of pom-scijava.

    But those steps only make it so that fiji/fiji depending on EC artifacts don't have to declare versions explicitly. The fiji component still needs to actually depend on them for them to be part of Fiji.

    Igor Pisarev
    @igorpisarev
    @axtimwalde Do you only insert and remove elements, or also update the existing ones too? If you need to update them as well and want to do it quickly, a priority queue would not suffice since it doesn't offer fast lookups based on the key. On the other hand, TreeMap or similar uses a single comparator for both lookups and sorting, so this would not suffice too
    Maybe you can have both HashMap<Long, Int> and TreeMap<Int, Long> (or similar to account for multiple values with the same Int) and keep them in sync to do all operations efficiently, potentially at the cost of having to deal with objects instead of primitives. I did something similar in paintera to create a prioritized task queue: https://github.com/saalfeldlab/paintera/blob/master/src/main/java/org/janelia/saalfeldlab/util/HashPriorityQueue.java
    Stephan Saalfeld
    @axtimwalde
    @igorpisarev yes, with objects it is relatively straight forward, but it's not an option because of the memory overhead. BTW, as always, there may be a shortcut for the actual problem that will make me get away without any of that, but I feel that this is about the tenth time that I am looking for something along those lines and then write a special purpose solution ;).
    Michael Doube
    @mdoube
    @ctrueden future goals look great! Would using a Maven-like updater mean there may be a possibility of version skew? I guess the updater could check the scijava POM, and use the version there, unless the plugin developer specified some other version(which would I assume be firmly discouraged). Getting artefacts automagically from a single central location makes much more sense than manually copying them to another server.
    Michael Doube
    @mdoube
    It's described on the Development Lifecycle page.
    Ah yes, I shall RT(F)M...
    Jan Eglinger
    @imagejan
    imagej.net seems to be down?
    Curtis Rueden
    @ctrueden
    @imagejan Sorry to say it's going to be down for several hours, as we recover from an exploit. Will bring it back up as soon as we can.
    Same for the update sites.
    Jan Eglinger
    @imagejan
    ah, good to know that you’re on it! And sorry for you that you have to do that during vacation :-/
    Curtis Rueden
    @ctrueden
    You reap what you sow, yeah? I should've cleaned things up sooner. It's OK; the setup will be much better afterward.
    Curtis Rueden
    @ctrueden
    I brought the update sites back online, as well as downloads.imagej.net, but we're not yet ready to bring back imagej.net. I fear that may take a few days. For those interested, the forum thread about it is here.
    Curtis Rueden
    @ctrueden

    Ahh, I just found the craziest bug in imagej-updater. When updating an installation, the updater 1) checks for changes, 2) downloads new JARs and drops them into place (on macOS at least, since there are no file locking problems), and then 3) tries to write out a new db.xml.gz file to the local installation recording the current state.

    I am starting from a Fiji with pom-scijava-29.0.0-beta-3 artifact versions locally (populated by mvn scijava:populate-app), and then doing an update-force-pristine to pull down the latest from the update site—this downgrades a lot of JARs, which is what I want. But the updater fails on step (3) afterward, because when invoking the XMLFileWriter, it leans on Xalan, which invokes the ServiceLoader mechanism looking for service plugins, which then tries to iterate the classpath looking for META-INF/services/... entries in the JARs, but some of the JARs have now been deleted, so it fails with java.util.ServiceConfigurationError: java.nio.charset.spi.CharsetProvider: Error reading configuration file. 😱