jankotek/mapdb

MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.

Jan Kotek

@jankotek

I will have a look and make Android friendly release.

@TitanKing see above

MarianJones

@MarianJones

Many Thanks @jankotek !

ravinallappan

@ravinallappan

Hi, I have cluster system with multiple nodes. Each nodes persist state into mapdb (per node) so it able to resume from the last point across restarts. Due to adding new nodes in to the cluster - the load balancing may change. This basically means some processing records may available in other node's mapdb - for this, I have implemented - each nodes to open its main db in write/create mode while also open all other available mapdb in readonly. This seems to be working fine despite warning messages from mapdb - I able to suppress it. Question - will this may lead to data corruption?

runmain

@runmain

HI
I am using mapdb v3.0.8
how to reclaim the file size ?
list is empty，but file is too large

        localDB = DBMaker.fileDB(DB_FILE_PATH)
                .fileMmapEnable()
                .checksumHeaderBypass()
                .make();

        IndexTreeList<T> list = localDB.indexTreeList(DB_FILE_EVENT, new ObjectSerializer<T>()).createOrOpen();

abadash

@abadash

Hi, I recently started using Ant Media Server which is using MapDB. I found online few ways to see the MapDB content with Java. Are there any examples of how to read MapDB from shell, javascript or php?

Jan Kotek

@jankotek

@abadash MapDB is java library, it does not work with other languages.

abadash

@abadash

Thank you @jankotek . does that mean that there is no shell/python wrapper yet to access its data? Or do you know any interpreted java environment which I can use to quickly see the data from a MapDB?

Or is there any tool like TablePlus for example which can get data from a MapDB file?

Jan Kotek

@jankotek

write program in java and use and call it to access data

Nilesh Injulkar

@injulkarnilesh

Does it make sense to use MapDb just to reduce the memory footprint of the application by using DBMaker.tempFileDB? The application I am working on parses remote files, sometimes the files are larger in size so I was thinking of using MapDb to store intermediate results of the parsing?

Ayman Madkour

@aymanmadkour

Hello. How do you delete a Map? I know db.delete() was removed in 3.x, and I can't figure out how to use db.getStore().delete(). Any help would be really appreciated.

Jan Kotek

@jankotek

@injulkarnilesh It is going to be slower than just using memory. You could partition your data model, so part of it is stored on disk.

@aymanmadkour it is not possible to delete collection in MapDB 3

vimil

@vimil

I had been using mapdb 1.0 and was looking at upgrading to 3.0.8. Looking at maven central It seems mapdb 3.0.8 has many more compile time dependencies. 1.0 had none. I was wondering if there are any plans to bring down compiletime dependencies.

Jan Kotek

@jankotek

@vimil Yes, MapDB 4 will have very little deps (no Kotlin, EC).

swampcoder

@swampcoder

I am going to start using this library for a display application - happy to see there is an active gitter room - might have some questions in the future

swampcoder

@swampcoder

HI
I am using mapdb v3.0.8
how to reclaim the file size ?
list is empty，but file is too large

        localDB = DBMaker.fileDB(DB_FILE_PATH)
                .fileMmapEnable()
                .checksumHeaderBypass()
                .make();

        IndexTreeList<T> list = localDB.indexTreeList(DB_FILE_EVENT, new ObjectSerializer<T>()).createOrOpen();

I believe you can call list.getStore().compact()

swampcoder

@swampcoder

is there a point to calling close with transactions enabled? as long as you call commit before shutting down?

runmain

@runmain

@swampcoder tks

Jan Kotek

@jankotek

@swampcoder I think in 3.0 close will finalize transaction and flushes log. Without it there would be some cleanup on next reopen.

Dario Romano

@DarioRomano

Hi, I'm currently working on making a legacy application available through open source and Github. To make it more appealing for open source user to contribute we're creating a Tycho-Maven build system. We're using mapDBv2beta6 which is available through maven central, but unfortunately it requires sun.misc and sun.nio.ch in its manifest. I've found a workaround here https://search.maven.org/artifact/net.sdruskat/net.sdruskat.fragment.sun.misc/1.0.0/jar to expose sun.misc to OSGi. But unfortunately I haven't found something similar for the sun.nio.ch package. I've tried creating this kind of fragment myself, but I haven't been that successful so far. I would also like to avoid to upgrade to mapDBv3 since I'm not sure if there are any compatability issues with the source code, and I'd also have to get kotlin running in an eclipse developing environment and within the tycho maven build.
Any advice on getting sun.nio.ch exposed to OSGi / the tycho build?

Dario Romano

@DarioRomano

I've also checked if I could just use version 1.0.8 instead, but we seem to be using some methods exclusive to version 2 of MapDB

Dario Romano

@DarioRomano

I've used grep on the source of v2 and noticed that sun.nio.ch seems kind of optional. So I've modified the jar to not require the import. Unfortunately that can't really be a permanent solution since it reduces the portability of the build

swampcoder

@swampcoder

@jankotek have had alot of success with early integration into a data analysis application related to radar processing. Curious, what is your estimate for mapdb 4 being completed? I see it has a lot of interesting and features so just curious.

Jan Kotek

@jankotek

@swampcoder Beginning of 2021

@DarioRomano I no longer support that version. But feel free to just remove all references to sun.nio.ch. It should be optional for some type of storage.

Bradley L Schatz

@blschatz

db.close() doesnt seem to work correctly. I'm getting the following exception on close, regardless of using mmap, filechannel, or the default.

java.nio.file.FileSystemException: I:\xxxx: The process cannot access the file because it is being used by another process.

This is using 3.0.7 on windows x64

Any suggestions? Thanks!

Jan Kotek

@jankotek

@blschatz most likely file locking. Storage was opened twice. Also on Windows file might remain locked, even after JVM process exits.

Bradley L Schatz

@blschatz

Thanks Jan! It turned out to be I was missing a close in an earlier unit test.

FYI mmap is working for me on Win10/Java8.

Organ bank from sector 7G

@testchamber1_twitter

Hi there! Has anyone tried to use the expireOverflow feature successfully? The problem I'm facing is that the only function that retrieves the data from the disk memory is the memoryMap.get(some_key). If I call a function like underlyingMap.keys, underlyingMap.values, etcetc they come back empty UNLESS the data was refreshed recently. Is this expected? thanks in advance.

liuyibox

@liuyibox

Hey I recently work on an android project have to use org.mapdb:mapdb:jar:3.0.7, but I got Method name '%%%verify$mapdb' in class 'org.mapdb.DB$Maker' cannot be represented in dex format. build error message. I saw this happen before, I'm wondering if there are any progresses on this problem?

liuyibox

@liuyibox

If I guess correctly, in order to use mapdb.jar on android, the java source files and kotlin source files of MapDB need to be specified separately when compiling them into class/dex files.

Jan Kotek

@jankotek

@liuyibox one method in kotlin has strange name in order to obfuscate it. It is not problem in JVM, but on Android. I will have to release new minor version for that

nate8628

@nate8628

db.treeMap can't do batch import anymore? I'm trying to use the example from the documentation but createFrom() is an invalid method.

nate8628

@nate8628

@jankotek

SJS830

@SJS830

How do you serialize an entire database to a byte array?

I want to save my databases to amazon s3 without storing them in a file

Or is there another database that is better suited for that

Nuwan Sanjeewa Abeysiriwardana

@nuwansa

Hi , NavigableSet.headSet with inclusive 'true' is not working, also tailSet with inclusive 'false' not working too. has anyone else experience this. I'm using latest version of mapdb 3.0.8

to explain further , NavigableSet.headSet with inclusive 'true' not including the equal value in the result set.
NavigableSet. tailSet with inclusive 'false' including the equal value in the result set

Jan Kotek

@jankotek

@nuwansa it should work, there are unit tests for this case. Maybe you are using some exotic data type where 'equal' is not correctly handled? what is your serializer and config?

Nuwan Sanjeewa Abeysiriwardana

@nuwansa

@jankotek this is the initialization code

this.db
            .treeSet("map")
            .serializer(new SerializerArrayTuple(Serializer.BIG_DECIMAL, Serializer.LONG))
            .counterEnable()
            .createOrOpen();

Jan Kotek

@jankotek

That should work

I will look into it jankotek/mapdb#980

Nuwan Sanjeewa Abeysiriwardana

@nuwansa

@jankotek thank you very much

Where communities thrive

People

Repo info

Activity