Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Arseniy Kaploun
    @kapliars
    Hello, we are using MapDB for offheap cache. What we are missing are ability to extract snapshots/backups for hashmap (htreemap) and then import back, and on top of that do incremental backups. I guess our best option is to add support for that, but I have few questions maybe someone can help, as @jankotek does not seem to appear here anymore
    • I have actually saw few mentions of backups in documentation, but can't find anything in the code , did I miss it?
    • We are using 3.0.7 and I was exploring sources of that one, if we want to develop that functionality what version / branch we better start off with intention to be merged to upstream eventually? We still need it to be stable enough as we use it for our production (and with quite high uptime requirements). Should we develop in 3.0.7 and port it to master then? Or master has diverged too much, and it would have to be actually reimplemented differently?
    • As far as I understand to export htreemap, we can quite easily dump content of the volumes to files, but in addition to that will need to serialize expiration queues, right? What else we might be missing?
    Jan Kotek
    @jankotek
    @kapliars I worked on snapshots, incremental backups etc for a while. It will be in MapDB4, but I dont have best history with keeping my promises. Maybe we should have a chat, send me an email with your contact info. I am in Europe.
    MapDB 3.0.8 was released. Bugfixes and Java 11 compatibility. http://www.mapdb.org/changelog/
    Stuart Goldberg
    @javastu_twitter
    We recently upgraded from mapdb 1.0.7 to 3.0.6 and I find that queues have been removed. I tried replacing the queue with an IndexTreeList, but the performance was horrible as I had thread contention between producer thread and consumer thread. Any advice would be appreciated.
    micw
    @micw
    Hi. I'm using mapdb (3.0.8) as cache for batch jobs (~1 million jobs). each job has a small "jobState" object and a larger (few kb) binary payload. I often iterate over the jobState objects (without payload). Sometimes I need to iterate over the jobState objects plus their payload. What is the best pattern to store those in mapdb? One large state+payload object? Or better 2 maps, one for state, one for payload (in this case, iterating over state+payload would mean iterating over state, get payload for each by id).
    And which map type should I use?
    lteqgt
    @sebastianharko

    Hi, Expiration does not seem to kick in for me:

    val db = DBMaker.memoryDirectDB().make()
      val map: HTreeMap[java.lang.Long, String] = db.hashMap("map")
        .expireMaxSize(3L)
        .expireAfterCreate
        .expireAfterUpdate
        .keySerializer(Serializer.LONG)
        .valueSerializer(Serializer.STRING)
        .create
    
    
      map.put(1L, "A")
      map.put(2L, "B")
      map.put(3L, "C")
      map.put(4L, "D")
      map.put(5L, "E")
    
    
      println(map.get(1L))
      println(map.get(2L))
      println(map.get(3L))
      println(map.get(4L))
      println(map.get(5L))

    I'm expecting to see nulls for 1 and 2 (null, null, C, D, E) ....... but instead I get A,B,C,D,E

    Mitesh Patel
    @MasterDDT
    Hi, I'm using mapdbutils (1.1.1) with mapdb (2.0-beta). Has anyone tried creating a map using Spark's UnsafeRow? I'd like a hashMap[UnsafeRow, Seq[UnsafeRow]] but I'm having trouble figuring out the serialization. UnsafeRow is already Externalizable and KryoSerializable, so I thought I wouldn't have to write a serializer myself....but I guess I do?
    Vishrawas Gopalakrishnan
    @vishrawas
    I am working of mapdb 3.0.8 ... I am trying to follow the example of composite keys. I am not able to find Tuple2 or any Tuple related classes. I can see the corresponding classes in https://github.com/Paxata/MapDB/tree/master/src/main/java/org/mapdb however not in the jar files that get downloaded via Maven. Manually copy pasting the relevant files like Tuple, Tuple2, Tuple2Serializer throws an error DB.DBAware not found ... Fixing each of these by copy pasting relevant portions from the git results in recreation of the entire code base. Am I missing something here? Is there an easier way or is the Tuple no longer supported. My use-case involves composite key and querying based on that. If Tuple is no longer the right way, can someone please advise on how to proceed with composite keys (of strings) in version 3.0.8
    Jan Kotek
    @jankotek
    @MasterDDT I would recommend converting UnsafeRow into a byte[].
    @vishrawas That code is from 1.0 branch, that causes compilation errors. Tuples were replaced with array tuples in MapDB3, source here: https://github.com/jankotek/mapdb/blob/release-3.0/src/main/java/org/mapdb/serializer/SerializerArrayTuple.java
    Mitesh Patel
    @MasterDDT
    @jankotek Thanks, yes I converted the row into hashcode Int for key, and wrote a custom Serializer<UnsafeRow> for value. Works now
    Mitesh Patel
    @MasterDDT
    Does mapdb 3.x support Scala 2.11.8? I'm trying to compile it into Spark 2.3.2 (https://github.com/apache/spark/blob/v2.3.2/pom.xml) and seeing this error: https://gist.github.com/MasterDDT/6ea59fd88258f946459d9dbeda98b5f1
    Jason Schoeman
    @TitanKing
    Does anyone perhaps use this library on Android, if so, how did you overcome this bit upon compile? Method name '%%%verify$mapdb' in class 'org.mapdb.DB$Maker' cannot be represented in dex format.
    MarianJones
    @MarianJones

    Similarly, @Jason Shoeman above, I would like to use mapdb on my android app.
    When using 3.0.0 I got similar error.
    However, I don't have a compilation issue when using 2.xx (2.0-beta13) but that's an old and beta...

    So my querry would be if it possible to rename this method so it goes well with android, as per my understnading name of the method is your choice in kotlin (I didn't do anything in kotlin...), unless some some obligatory part in kotlin. What do you reckon @Jan Kotek

    MarianJones
    @MarianJones

    From 3.0.6 it reports usage of static method (java 8).

    Could it be avoided for for making app a bit more backword compatibilie?
    "Static interface methods are only supported starting with Android N (--min-api 24): java.lang.Object org.eclipse.collections.api.InternalIterable.$deserializeLambda$(java.lang.invoke.SerializedLambda)
    "
    "Default interface methods are only supported starting with Android N (--min-api 24): org.eclipse.collections.impl.lazy.parallel.Batch org.eclipse.collections.impl.lazy.parallel.OrderedBatch.collect(org.eclipse.collections.api.block.function.Function)
    "

    Jan Kotek
    @jankotek
    @MarianJones This method has some strange characters, to prevent it from being imported from Java code. Weird case between Kotlin compiler, JVM bytecode and Java code.
    I will have a look and make Android friendly release.
    @TitanKing see above
    MarianJones
    @MarianJones
    Many Thanks @jankotek !
    ravinallappan
    @ravinallappan
    Hi, I have cluster system with multiple nodes. Each nodes persist state into mapdb (per node) so it able to resume from the last point across restarts. Due to adding new nodes in to the cluster - the load balancing may change. This basically means some processing records may available in other node's mapdb - for this, I have implemented - each nodes to open its main db in write/create mode while also open all other available mapdb in readonly. This seems to be working fine despite warning messages from mapdb - I able to suppress it. Question - will this may lead to data corruption?
    runmain
    @runmain
    HI
    I am using mapdb v3.0.8
    how to reclaim the file size ?
    list is empty,but file is too large
            localDB = DBMaker.fileDB(DB_FILE_PATH)
                    .fileMmapEnable()
                    .checksumHeaderBypass()
                    .make();
    
            IndexTreeList<T> list = localDB.indexTreeList(DB_FILE_EVENT, new ObjectSerializer<T>()).createOrOpen();
    abadash
    @abadash
    Hi, I recently started using Ant Media Server which is using MapDB. I found online few ways to see the MapDB content with Java. Are there any examples of how to read MapDB from shell, javascript or php?
    Jan Kotek
    @jankotek
    @abadash MapDB is java library, it does not work with other languages.
    abadash
    @abadash
    Thank you @jankotek . does that mean that there is no shell/python wrapper yet to access its data? Or do you know any interpreted java environment which I can use to quickly see the data from a MapDB?
    Or is there any tool like TablePlus for example which can get data from a MapDB file?
    Jan Kotek
    @jankotek
    write program in java and use and call it to access data
    Nilesh Injulkar
    @injulkarnilesh
    Does it make sense to use MapDb just to reduce the memory footprint of the application by using DBMaker.tempFileDB? The application I am working on parses remote files, sometimes the files are larger in size so I was thinking of using MapDb to store intermediate results of the parsing?
    Ayman Madkour
    @aymanmadkour
    Hello. How do you delete a Map? I know db.delete() was removed in 3.x, and I can't figure out how to use db.getStore().delete(). Any help would be really appreciated.
    Jan Kotek
    @jankotek
    @injulkarnilesh It is going to be slower than just using memory. You could partition your data model, so part of it is stored on disk.
    @aymanmadkour it is not possible to delete collection in MapDB 3
    vimil
    @vimil
    I had been using mapdb 1.0 and was looking at upgrading to 3.0.8. Looking at maven central It seems mapdb 3.0.8 has many more compile time dependencies. 1.0 had none. I was wondering if there are any plans to bring down compiletime dependencies.
    Jan Kotek
    @jankotek
    @vimil Yes, MapDB 4 will have very little deps (no Kotlin, EC).
    swampcoder
    @swampcoder
    I am going to start using this library for a display application - happy to see there is an active gitter room - might have some questions in the future
    swampcoder
    @swampcoder

    HI
    I am using mapdb v3.0.8
    how to reclaim the file size ?
    list is empty,but file is too large

            localDB = DBMaker.fileDB(DB_FILE_PATH)
                    .fileMmapEnable()
                    .checksumHeaderBypass()
                    .make();
    
            IndexTreeList<T> list = localDB.indexTreeList(DB_FILE_EVENT, new ObjectSerializer<T>()).createOrOpen();

    I believe you can call list.getStore().compact()

    swampcoder
    @swampcoder
    is there a point to calling close with transactions enabled? as long as you call commit before shutting down?
    runmain
    @runmain
    @swampcoder tks
    Jan Kotek
    @jankotek
    @swampcoder I think in 3.0 close will finalize transaction and flushes log. Without it there would be some cleanup on next reopen.
    Dario Romano
    @DarioRomano
    Hi, I'm currently working on making a legacy application available through open source and Github. To make it more appealing for open source user to contribute we're creating a Tycho-Maven build system. We're using mapDBv2beta6 which is available through maven central, but unfortunately it requires sun.misc and sun.nio.ch in its manifest. I've found a workaround here https://search.maven.org/artifact/net.sdruskat/net.sdruskat.fragment.sun.misc/1.0.0/jar to expose sun.misc to OSGi. But unfortunately I haven't found something similar for the sun.nio.ch package. I've tried creating this kind of fragment myself, but I haven't been that successful so far. I would also like to avoid to upgrade to mapDBv3 since I'm not sure if there are any compatability issues with the source code, and I'd also have to get kotlin running in an eclipse developing environment and within the tycho maven build.
    Any advice on getting sun.nio.ch exposed to OSGi / the tycho build?
    Dario Romano
    @DarioRomano
    I've also checked if I could just use version 1.0.8 instead, but we seem to be using some methods exclusive to version 2 of MapDB
    Dario Romano
    @DarioRomano
    I've used grep on the source of v2 and noticed that sun.nio.ch seems kind of optional. So I've modified the jar to not require the import. Unfortunately that can't really be a permanent solution since it reduces the portability of the build
    swampcoder
    @swampcoder
    @jankotek have had alot of success with early integration into a data analysis application related to radar processing. Curious, what is your estimate for mapdb 4 being completed? I see it has a lot of interesting and features so just curious.
    Jan Kotek
    @jankotek
    @swampcoder Beginning of 2021
    @DarioRomano I no longer support that version. But feel free to just remove all references to sun.nio.ch. It should be optional for some type of storage.
    Bradley L Schatz
    @blschatz
    db.close() doesnt seem to work correctly. I'm getting the following exception on close, regardless of using mmap, filechannel, or the default.
    java.nio.file.FileSystemException: I:\xxxx: The process cannot access the file because it is being used by another process.
    This is using 3.0.7 on windows x64
    Any suggestions? Thanks!
    Jan Kotek
    @jankotek
    @blschatz most likely file locking. Storage was opened twice. Also on Windows file might remain locked, even after JVM process exits.
    Bradley L Schatz
    @blschatz
    Thanks Jan! It turned out to be I was missing a close in an earlier unit test.