Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 01 01:21
    rube200 edited #638
  • Oct 01 01:21
    rube200 commented #638
  • Oct 01 01:21
    rube200 commented #638
  • Sep 30 17:41
    jurasans commented #2232
  • Sep 29 19:05
    mrpip137 commented #931
  • Sep 29 14:15
    mrpip137 commented #931
  • Sep 28 14:40
    BanjoBenStanford edited #2242
  • Sep 28 14:39
    BanjoBenStanford edited #2242
  • Sep 28 14:36
    BanjoBenStanford labeled #2242
  • Sep 28 14:36
    BanjoBenStanford opened #2242
  • Sep 28 12:38
    Iron2213 commented #2232
  • Sep 26 10:26
    jurasans commented #2232
  • Sep 22 13:22
    LazarQt commented #2221
  • Sep 22 06:55
    devmaestro1 edited #2241
  • Sep 22 05:23
    devmaestro1 edited #2241
  • Sep 22 05:22
    devmaestro1 edited #2241
  • Sep 22 04:57
    devmaestro1 edited #2241
  • Sep 22 04:23
    devmaestro1 edited #2241
  • Sep 22 04:23
    devmaestro1 labeled #2241
  • Sep 22 04:23
    devmaestro1 opened #2241
heatherjoanne44
@heatherjoanne44
@lbnascimento So I have redesigned it where it shares a single db instance. My connection string looks like this: var conectionString = $"Filename={_databaseFileName};Connection=Direct;"; I have tried Task.Run and Parallel.Invoke(() on the database method that call Insert on the database that inserts a list of records. This insert method is the method that is causing a performance hit. The contents of the database method looks like this:
var collection = _db.GetCollection<IdentityKeyPair>(_tableName);
collection.Insert(identityKeyPairs);
Although this way has significantly improved performance, I still am having a performance hit. If you have any tips on improving the performance for this insert, that would be greatly appreciated
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 If you're inserting a lot of documents in a single collection, you could use the LiteCollection<T>.Insert(IEnumerable<T>) overload to improve performance
Or you could run all the inserts in a single transaction (that's what LiteCollection<T>.Insert(IEnumerable<T>) does)
This will be a lot faster than, for example, using a foreach loop that calls LiteCollection<T>.Insert(T) for every object
heatherjoanne44
@heatherjoanne44
@lbnascimento It is currently inserting a list and not through a loop at the very end. However, this whole process gets called frequently per row in a datatable. In the code I shared, identityKeyPairs is
a list: collection.Insert(identityKeyPairs); Unfortunately, it has to be like this, as we cant wait until the entire file processes to do the insert
Leonardo Nascimento
@lbnascimento
The smallest lock granularity LiteDB uses are collection-wide locks, so having multiple threads running inserts to the same collection is pointless. You can have multiple threads reading from a single collection or writing to different collections and these operations will run in parallel.
You could try using a foreach loop and calling LiteCollection<T>.Insert(T) for every object in a single transaction (between calls to db.BeginTrans() and db.Commit()). It is possible that the overhead is smaller this way, but I'm not sure.
heatherjoanne44
@heatherjoanne44
Do you mean the insert method would look something like this db.BeginTrans(); LiteCollection<T>.Insert(T); db.Commit(); and I would make this call inside the loop instead of sending the list to the insert?
Leonardo Nascimento
@lbnascimento
Something like this:
db.BeginTrans();

foreach(var item in identityKeyPairs)
    collection.Insert(item);

db.Commit();
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 I ran some tests and indeed using the foreach loop in a single transaction (like the example above) should be faster. The downside compared to using Insert(IEnumerable<T>) is that in case the loop doesn't finish (process crash, power loss, exception...), all inserted items will be discarded (because they are part of an unfinished transaction).
weirdyang
@weirdyang
image.png

Hi guys,

I'm trying to use litedb as a log event storage,

I have certain properties in the properties field, which are unstructured. is it possible to query for field that might not be there?
For example querying out all logevents where there is a typeTag key present
this is my model, is it ok to use Dictionary<string, string> or should I use a list instead?
    public class LogMessage
    {
        public string Level { get; set; }

        public string Message { get; set; }

        public Dictionary<string, string> Properties { get; set; } = new Dictionary<string, string>();

    }
Leonardo Nascimento
@lbnascimento
@captmomo Dictionaries are serialized as sub-documents, so it's pretty simple to use them in queries.
@captmomo For example, if you wanted all the events which have the property Name, you could simply run the query select $ from logevents where $.Properties.Name != null
weirdyang
@weirdyang
@lbnascimento I'm trying to query for entries where there is certain key in the properties. I ended up with:
SELECT $ FROM logEvents
WHERE $.Properties.Customer != null
@lbnascimento thanks!
WTTSoftwareSolutions
@WTTSoftwareSolutions
Hey Guys,
Is there a way in v5 to load a collection from json? I know that there is the ability to load from json file using the sql syntax:
db.Execute("select $ into mycollection from $file('myfile.json')")

however my json is not in a file on disc, it is in memory. This used to work:

db.Engine.Insert("mycol", JsonSerializer.Deserialize(json).AsArray.ToArray());

There does not seem to be accessibility to the Engine from LiteDatabase anymore.

Leonardo Nascimento
@lbnascimento
@WTTSoftwareSolutions If you open your collection as var col = db.GetCollection("mycollection"), you could do something like col.Insert(JsonSerializer.DeserializeArray(json).Cast<BsonDocument>())
weirdyang
@weirdyang

is there anyway to save message

@WTTSoftwareSolutions If you open your collection as var col = db.GetCollection("mycollection"), you could do something like col.Insert(JsonSerializer.DeserializeArray(json).Cast<BsonDocument>())

thanks! been trying to figure this out too

WTTSoftwareSolutions
@WTTSoftwareSolutions
@lbnascimento Thanks for the help! For some reason it didn't work. It kept throwing exceptions about the json formatting. The json was from me using Newtonsoft.Json to serialize a table from an OleDb DataSet. I changed the way I'm doing the conversion and am now using NPoco Query to get an IEnumerable of my Poco's and passing that into the Collection Insert() method. Success!! We have now officially dumped using Access and will be completely LiteDB in the January release of our product!
weirdyang
@weirdyang
Can I store a class with a bsondocument property? or should i just store it as a string and do a query e.g "Select JSON($.MyProperty) From Collection"
iSunOfficial
@iSunOfficial
Did I make a mistake, or is this not possible to use regex?
ILiteQueryable<Class> h = class.Query().Where(x => x.Query.Count(s => Regex.IsMatch(text, @$"^(.*)({s})(net)$")) > 0);
Shady3cho
@Shady3cho
I joined just to say, yo I love this db thanks to who ever made it
weirdyang
@weirdyang
Recommendation
Single instance (second option) is much faster than multi instances. In multi instances environment, each instance must do expensive data file operations: open, lock, unlock, read, close. Also, each instance has its own cache control and, if used only for a single operation, will discard all cached pages on close of datafile. In single instance, all pages in cache are shared between all read threads.
Does this mean for a web app I should be using the litedbrepo as a singleton?
weirdyang
@weirdyang
What is the best way to implement async methods?
Can I return a Task.FromResult or do I need to use lock?
Leonardo Nascimento
@lbnascimento
@iSunOfficial Regex is currently not supported, I'll see if it's possible to add support for it in a future version
@captmomo Yes, you can have a BsonProperty if you want
@captmomo Having a single instance of LiteDatabase is the recommended way to use LiteDB.
@captmomo There is currently no support for async methods in LiteDB, but someone made an async wrapper for LiteDB, you should check it out: https://github.com/mlockett42/litedb-async
weirdyang
@weirdyang
Thanks! sorry for all the questions!

btw,

            BsonMapper.Global.RegisterType<LogEventLevel>(
                serialize: (lvl) => (int)lvl,
                deserialize: (bson) => (LogEventLevel)bson.AsInt32);

I am trying to store an enum as an int and getting back the enum when deserializing.

this is my mapping method, however, the string representation is stored and I get a null object reference when I try to fetch it:
var results = context.LiteRepository.Fetch<LogMessage>(Query.EQ("Level", "Error"));

image.png
Leonardo Nascimento
@lbnascimento
@captmomo If you want to store enums as int, you can just configure the mapper with: BsonMapper.Global.EnumAsInteger = true;
weirdyang
@weirdyang
ahh, thanks.
weirdyang
@weirdyang

thanks that works! but now I get a null object reference when I try to fetch the items, it seems my mapping for the DbRef is causing the error:
+ TargetSite {System.Object Deserialize(System.Type, LiteDB.BsonValue)} System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}

            BsonMapper.Global.Entity<LogMessage>()
                .Id(x => x.Id)
                .DbRef(x => x.User, nameof(User));
var results = context.LiteRepository.Fetch<LogMessage>(Query.EQ("Level", (int)LogEventLevel.Error));

Do I need to do an .include?

Leonardo Nascimento
@lbnascimento
@captmomo You shouldn't use DbRef with LiteRepository. LiteRepository is meant to be used as a single "container" for all your data types. If you want distinct collections for different types (that's what DbRef does), you should use LiteCollection<T>
weirdyang
@weirdyang
Oh, so LiteRepository is only to be used for a single collection
Leonardo Nascimento
@lbnascimento
@captmomo Internally it still uses separate collections in the data file, but it's meant to be accessed as a single container.
weirdyang
@weirdyang
ok, sorry I'm a bit confused as to what's the difference. do you mind explaining what you mean by a single container?
isn't dbref for referencing collections within the same db?
Leonardo Nascimento
@lbnascimento
@captmomo The most common way to manipulate data in LiteDB is using LiteCollection<T>. For example, if you have two classes Customer and Product, you would open two distinct collections: var customerCol = db.GetCollection<Customer>(); and var productCol = db.GetCollection<Product>();
@captmomo If you want a single "entry point" for all your data, that's when you use LiteRepository. Internally, it still uses distinct collections, but you can simply throw all your objects at it and it will handle the collections itself.
Leonardo Nascimento
@lbnascimento
@captmomo Now, imagine you have a property of type Product inside Customer. LiteDB will store Product as a sub-document inside Customer, so they'll both be stored in the Customer collection. By using DbRef, you can store instances of Product in its own collection and store only a reference to it in every customer. This, however, should not be used with LiteRepository.
weirdyang
@weirdyang
hmm ok, I'm still not sure why LiteRepository can't handle the the dbref. does it have issues loading the referenced entities?
weirdyang
@weirdyang
image.png