Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 25 11:53
    UlyssesWu opened #2252
  • Nov 25 11:53
    UlyssesWu labeled #2252
  • Nov 24 14:17
    hmi95 commented #1504
  • Nov 22 23:42
    MollieofMSVR edited #2249
  • Nov 20 16:26
    sangeethnandakumar commented #1546
  • Nov 20 16:07
    sangeethnandakumar commented #1546
  • Nov 18 23:45
    BenWoodford commented #1984
  • Nov 18 14:05
    MollieofMSVR edited #2249
  • Nov 18 14:04
    MollieofMSVR edited #2249
  • Nov 18 14:03
    MollieofMSVR edited #2249
  • Nov 18 14:03
    MollieofMSVR edited #2249
  • Nov 17 15:27
    BenWoodford commented #1984
  • Nov 16 13:25
    sgf commented #2075
  • Nov 16 13:25
    sgf commented #2075
  • Nov 16 13:13
    sgf opened #2251
  • Nov 16 13:13
    sgf labeled #2251
  • Nov 16 13:08
    sgf commented #2227
  • Nov 16 00:24
    ultravelocity commented #2017
  • Nov 14 09:05
    TrevorDArcyEvans closed #2247
  • Nov 14 09:05
    TrevorDArcyEvans commented #2247
heatherjoanne44
@heatherjoanne44
@lbnascimento thanks so much. I'll give that a try. Do you also have an example of using litedb with multithreading by any chance?
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 You can simply share an instance of LiteDatabase<T> or LiteCollection<T> between threads
Tim Moore
@tjmoore

Hi, I'm currently working on a project with LiteDB 5 (5.0.8) and POCO data model classes that are isolated from the data access layer. I want to ignore certain properties but can't use BsonIgnore attribute as I want to avoid a reference to LiteDB in the data model assemblies. Instead I'm trying to use Ignore on the mapper, but while it compiles fine, it's throwing an exception saying it can't find the property to ignore it.

e.g. class property of

        public bool MyProperty { get; set; }

Mapper...

            var mapper = BsonMapper.Global;

            mapper.Entity<MyClass>()
                .Ignore(x => x.MyProperty);

This exception occurs on the Ignore:

  Message: 
    System.ArgumentNullException : Value cannot be null. (Parameter 'Member 'MyProperty' not found in type 'MyClass' (use IncludeFields in BsonMapper)')
  Stack Trace: 
    EntityBuilder`1.GetMember[TK,K](Expression`1 member, Action`1 action)
    EntityBuilder`1.Ignore[K](Expression`1 member)
Tim Moore
@tjmoore
Ah, follow up. I think the issue is calling the mapper twice. As with this issue I'm hitting this in unit tests. If I make the mappings calls occur once only across all tests the problem goes away. #1159
Tim Moore
@tjmoore

In LiteDB 4 if multiple instances of LiteDatabase in the same process or separate processes access the same file, I believe it uses file locks and waits on the file? i.e. defaults to 'shared' mode.

In LiteDB 5, as the default is 'direct' mode, instead these situations would create an IOException for file in use, even in the same process or even thread?

At least I've observed with unit tests where I've discovered a bug in my code that was creating multiple instances of LiteDatabase in the same thread for the same database, and this was throwing IOException with file in use. The same tests in LiteDB 4 did not.

Obviously if it's intended 'shared' mode can be used, but I've noticed this is significantly slower seemingly than in LiteDB 4 default mode. Is this due to the use of mutexes now (instead of file locks?) and/or the closure of the database per operation ? In my tests it adds 1s or so to each DB operation and if that was scaled up it would be far too slow, so in LiteDB 5 it looks like concurrent access would be too slow. Therefore I need to design for a single process that talks to the database files and other processes communicate with that one. Which is okay though am concerned about bottleneck potential, but then the same could be true on waiting for file locks.

Leonardo Nascimento
@lbnascimento
@tjmoore LiteDB v5.0.x shared mode uses a global mutex to control file access. Basically, it opens the datafile before every operation and closes it after every operation. This has a significant overhead.
Shared mode should only be used if you really need to access the same datafile from different processes. If you can avoid it, you should use direct mode, which is much faster and works even for multi-threaded usage.
Leonardo Nascimento
@lbnascimento
In the next major release (v5.1, no release date yet) the shared mode will be entirely reworked and will use lock files instead of mutex for access control
heatherjoanne44
@heatherjoanne44
@lbnascimento So I have redesigned it where it shares a single db instance. My connection string looks like this: var conectionString = $"Filename={_databaseFileName};Connection=Direct;"; I have tried Task.Run and Parallel.Invoke(() on the database method that call Insert on the database that inserts a list of records. This insert method is the method that is causing a performance hit. The contents of the database method looks like this:
var collection = _db.GetCollection<IdentityKeyPair>(_tableName);
collection.Insert(identityKeyPairs);
Although this way has significantly improved performance, I still am having a performance hit. If you have any tips on improving the performance for this insert, that would be greatly appreciated
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 If you're inserting a lot of documents in a single collection, you could use the LiteCollection<T>.Insert(IEnumerable<T>) overload to improve performance
Or you could run all the inserts in a single transaction (that's what LiteCollection<T>.Insert(IEnumerable<T>) does)
This will be a lot faster than, for example, using a foreach loop that calls LiteCollection<T>.Insert(T) for every object
heatherjoanne44
@heatherjoanne44
@lbnascimento It is currently inserting a list and not through a loop at the very end. However, this whole process gets called frequently per row in a datatable. In the code I shared, identityKeyPairs is
a list: collection.Insert(identityKeyPairs); Unfortunately, it has to be like this, as we cant wait until the entire file processes to do the insert
Leonardo Nascimento
@lbnascimento
The smallest lock granularity LiteDB uses are collection-wide locks, so having multiple threads running inserts to the same collection is pointless. You can have multiple threads reading from a single collection or writing to different collections and these operations will run in parallel.
You could try using a foreach loop and calling LiteCollection<T>.Insert(T) for every object in a single transaction (between calls to db.BeginTrans() and db.Commit()). It is possible that the overhead is smaller this way, but I'm not sure.
heatherjoanne44
@heatherjoanne44
Do you mean the insert method would look something like this db.BeginTrans(); LiteCollection<T>.Insert(T); db.Commit(); and I would make this call inside the loop instead of sending the list to the insert?
Leonardo Nascimento
@lbnascimento
Something like this:
db.BeginTrans();

foreach(var item in identityKeyPairs)
    collection.Insert(item);

db.Commit();
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 I ran some tests and indeed using the foreach loop in a single transaction (like the example above) should be faster. The downside compared to using Insert(IEnumerable<T>) is that in case the loop doesn't finish (process crash, power loss, exception...), all inserted items will be discarded (because they are part of an unfinished transaction).
weirdyang
@weirdyang
image.png

Hi guys,

I'm trying to use litedb as a log event storage,

I have certain properties in the properties field, which are unstructured. is it possible to query for field that might not be there?
For example querying out all logevents where there is a typeTag key present
this is my model, is it ok to use Dictionary<string, string> or should I use a list instead?
    public class LogMessage
    {
        public string Level { get; set; }

        public string Message { get; set; }

        public Dictionary<string, string> Properties { get; set; } = new Dictionary<string, string>();

    }
Leonardo Nascimento
@lbnascimento
@captmomo Dictionaries are serialized as sub-documents, so it's pretty simple to use them in queries.
@captmomo For example, if you wanted all the events which have the property Name, you could simply run the query select $ from logevents where $.Properties.Name != null
weirdyang
@weirdyang
@lbnascimento I'm trying to query for entries where there is certain key in the properties. I ended up with:
SELECT $ FROM logEvents
WHERE $.Properties.Customer != null
@lbnascimento thanks!
WTTSoftwareSolutions
@WTTSoftwareSolutions
Hey Guys,
Is there a way in v5 to load a collection from json? I know that there is the ability to load from json file using the sql syntax:
db.Execute("select $ into mycollection from $file('myfile.json')")

however my json is not in a file on disc, it is in memory. This used to work:

db.Engine.Insert("mycol", JsonSerializer.Deserialize(json).AsArray.ToArray());

There does not seem to be accessibility to the Engine from LiteDatabase anymore.

Leonardo Nascimento
@lbnascimento
@WTTSoftwareSolutions If you open your collection as var col = db.GetCollection("mycollection"), you could do something like col.Insert(JsonSerializer.DeserializeArray(json).Cast<BsonDocument>())
weirdyang
@weirdyang

is there anyway to save message

@WTTSoftwareSolutions If you open your collection as var col = db.GetCollection("mycollection"), you could do something like col.Insert(JsonSerializer.DeserializeArray(json).Cast<BsonDocument>())

thanks! been trying to figure this out too

WTTSoftwareSolutions
@WTTSoftwareSolutions
@lbnascimento Thanks for the help! For some reason it didn't work. It kept throwing exceptions about the json formatting. The json was from me using Newtonsoft.Json to serialize a table from an OleDb DataSet. I changed the way I'm doing the conversion and am now using NPoco Query to get an IEnumerable of my Poco's and passing that into the Collection Insert() method. Success!! We have now officially dumped using Access and will be completely LiteDB in the January release of our product!
weirdyang
@weirdyang
Can I store a class with a bsondocument property? or should i just store it as a string and do a query e.g "Select JSON($.MyProperty) From Collection"
iSunOfficial
@iSunOfficial
Did I make a mistake, or is this not possible to use regex?
ILiteQueryable<Class> h = class.Query().Where(x => x.Query.Count(s => Regex.IsMatch(text, @$"^(.*)({s})(net)$")) > 0);
Shady3cho
@Shady3cho
I joined just to say, yo I love this db thanks to who ever made it
weirdyang
@weirdyang
Recommendation
Single instance (second option) is much faster than multi instances. In multi instances environment, each instance must do expensive data file operations: open, lock, unlock, read, close. Also, each instance has its own cache control and, if used only for a single operation, will discard all cached pages on close of datafile. In single instance, all pages in cache are shared between all read threads.
Does this mean for a web app I should be using the litedbrepo as a singleton?
weirdyang
@weirdyang
What is the best way to implement async methods?
Can I return a Task.FromResult or do I need to use lock?
Leonardo Nascimento
@lbnascimento
@iSunOfficial Regex is currently not supported, I'll see if it's possible to add support for it in a future version
@captmomo Yes, you can have a BsonProperty if you want
@captmomo Having a single instance of LiteDatabase is the recommended way to use LiteDB.
@captmomo There is currently no support for async methods in LiteDB, but someone made an async wrapper for LiteDB, you should check it out: https://github.com/mlockett42/litedb-async
weirdyang
@weirdyang
Thanks! sorry for all the questions!

btw,

            BsonMapper.Global.RegisterType<LogEventLevel>(
                serialize: (lvl) => (int)lvl,
                deserialize: (bson) => (LogEventLevel)bson.AsInt32);

I am trying to store an enum as an int and getting back the enum when deserializing.

this is my mapping method, however, the string representation is stored and I get a null object reference when I try to fetch it:
var results = context.LiteRepository.Fetch<LogMessage>(Query.EQ("Level", "Error"));

image.png
Leonardo Nascimento
@lbnascimento
@captmomo If you want to store enums as int, you can just configure the mapper with: BsonMapper.Global.EnumAsInteger = true;
weirdyang
@weirdyang
ahh, thanks.
weirdyang
@weirdyang

thanks that works! but now I get a null object reference when I try to fetch the items, it seems my mapping for the DbRef is causing the error:
+ TargetSite {System.Object Deserialize(System.Type, LiteDB.BsonValue)} System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}

            BsonMapper.Global.Entity<LogMessage>()
                .Id(x => x.Id)
                .DbRef(x => x.User, nameof(User));
var results = context.LiteRepository.Fetch<LogMessage>(Query.EQ("Level", (int)LogEventLevel.Error));

Do I need to do an .include?

Leonardo Nascimento
@lbnascimento
@captmomo You shouldn't use DbRef with LiteRepository. LiteRepository is meant to be used as a single "container" for all your data types. If you want distinct collections for different types (that's what DbRef does), you should use LiteCollection<T>
weirdyang
@weirdyang
Oh, so LiteRepository is only to be used for a single collection