Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Dec 06 17:13
    lbnascimento added as member
  • Dec 06 17:12
    lbnascimento removed as member
  • Dec 06 15:45
    dangergrrl commented #2256
  • Dec 06 15:45
    dangergrrl closed #2256
  • Dec 06 14:20
    kcsombrio commented #2250
  • Dec 06 14:19
    kcsombrio commented #2250
  • Dec 06 10:25
    arcanelab opened #2259
  • Dec 06 10:09
    arcanelab closed #2258
  • Dec 06 10:09
    arcanelab synchronize #2258
  • Dec 06 10:08
    arcanelab synchronize #2258
  • Dec 06 09:41
    arcanelab synchronize #2258
  • Dec 06 09:23
    arcanelab synchronize #2258
  • Dec 05 23:43
    7702244 commented #2250
  • Dec 05 17:11

    lbnascimento on master

    Fix object deserialization when… (compare)

  • Dec 05 10:02
    jackmcbrezel commented #2255
  • Dec 05 10:02
    jackmcbrezel commented #2255
  • Dec 05 10:00
    jackmcbrezel commented #2255
  • Dec 04 17:34
    7702244 commented #2082
  • Dec 02 16:10
    ProKn1fe commented #2255
  • Dec 01 23:12
    arcanelab edited #2258
Sathanu
@sathanu
Is litedb support transaction ?
Megha Narasimhamurthy
@megha-cn
Hi. I have been trying to access a LiteDB file that my application uses via PowerShell (https://github.com/nightroman/Ldbc) . I inturn need to execute the powershell script from protractor via shelljs for some pre-conditioning on my collections data. However, I am unable to open my database as it throws "Use-LiteDatabase : Ticks must be between DateTime.MinValue.Ticks and DateTime.MaxValue.Ticks." Any pointers?
Leonardo Nascimento
@lbnascimento
@sathanu Yes, LiteDB supports transactions.
@megha-cn Could you send me your data file?
Megha Narasimhamurthy
@megha-cn
Thank you @lbnascimento for your time and resolving the access issue that I had!
Leonardo Nascimento
@lbnascimento
@megha-cn No problem
5d2
@5d2
@mbdavid HI master of litedb,I got an exception “Maximum number of transactions reached” when used “ col.FindById(new ObjectId(Id)) 、col.Find(bse).ToList();” for search,do you knew the reason?
Helmut Wahrmann
@hwahrmann
how can i cast back to my class that was used to insert a document?
i am running a "select $ from .....".
This returns a List of BsonDocument. How can i cast this BsonDocument to my original class?
Helmut Wahrmann
@hwahrmann
found it: BsonMapper.Global.Deserialize<MyClass>
heatherjoanne44
@heatherjoanne44
Hello, How can I combine this to one query in LiteDB to mprove performane. Value is unique in this particular case and RecordId is not.
using (var db = new LiteDatabase(_databaseFileName))
{
// Get a collection (or create, if doesn't exist)
var collection = db.GetCollection<IdentityKeyPair>(_tableName);
            collection.EnsureIndex(x => x.Value);
            collection.EnsureIndex(x => x.RecordId);

            var entityKeyRecordId = collection.Query().Where(p => p.Value == uniqueValue).Select(x => x.RecordId).FirstOrDefault();

            if (entityKeyRecordId != Guid.Empty)
                results = collection.Query().Where(p => p.RecordId == entityKeyRecordId).ToList();

        }
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 If Value is unique (different for every document in the collection), you could try to use the unique parameter in the EnsureIndex: collection.EnsureIndex(x => x.Value, true)
See if that gives you better performance
heatherjoanne44
@heatherjoanne44
@lbnascimento thanks so much. I'll give that a try. Do you also have an example of using litedb with multithreading by any chance?
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 You can simply share an instance of LiteDatabase<T> or LiteCollection<T> between threads
Tim Moore
@tjmoore

Hi, I'm currently working on a project with LiteDB 5 (5.0.8) and POCO data model classes that are isolated from the data access layer. I want to ignore certain properties but can't use BsonIgnore attribute as I want to avoid a reference to LiteDB in the data model assemblies. Instead I'm trying to use Ignore on the mapper, but while it compiles fine, it's throwing an exception saying it can't find the property to ignore it.

e.g. class property of

        public bool MyProperty { get; set; }

Mapper...

            var mapper = BsonMapper.Global;

            mapper.Entity<MyClass>()
                .Ignore(x => x.MyProperty);

This exception occurs on the Ignore:

  Message: 
    System.ArgumentNullException : Value cannot be null. (Parameter 'Member 'MyProperty' not found in type 'MyClass' (use IncludeFields in BsonMapper)')
  Stack Trace: 
    EntityBuilder`1.GetMember[TK,K](Expression`1 member, Action`1 action)
    EntityBuilder`1.Ignore[K](Expression`1 member)
Tim Moore
@tjmoore
Ah, follow up. I think the issue is calling the mapper twice. As with this issue I'm hitting this in unit tests. If I make the mappings calls occur once only across all tests the problem goes away. #1159
Tim Moore
@tjmoore

In LiteDB 4 if multiple instances of LiteDatabase in the same process or separate processes access the same file, I believe it uses file locks and waits on the file? i.e. defaults to 'shared' mode.

In LiteDB 5, as the default is 'direct' mode, instead these situations would create an IOException for file in use, even in the same process or even thread?

At least I've observed with unit tests where I've discovered a bug in my code that was creating multiple instances of LiteDatabase in the same thread for the same database, and this was throwing IOException with file in use. The same tests in LiteDB 4 did not.

Obviously if it's intended 'shared' mode can be used, but I've noticed this is significantly slower seemingly than in LiteDB 4 default mode. Is this due to the use of mutexes now (instead of file locks?) and/or the closure of the database per operation ? In my tests it adds 1s or so to each DB operation and if that was scaled up it would be far too slow, so in LiteDB 5 it looks like concurrent access would be too slow. Therefore I need to design for a single process that talks to the database files and other processes communicate with that one. Which is okay though am concerned about bottleneck potential, but then the same could be true on waiting for file locks.

Leonardo Nascimento
@lbnascimento
@tjmoore LiteDB v5.0.x shared mode uses a global mutex to control file access. Basically, it opens the datafile before every operation and closes it after every operation. This has a significant overhead.
Shared mode should only be used if you really need to access the same datafile from different processes. If you can avoid it, you should use direct mode, which is much faster and works even for multi-threaded usage.
Leonardo Nascimento
@lbnascimento
In the next major release (v5.1, no release date yet) the shared mode will be entirely reworked and will use lock files instead of mutex for access control
heatherjoanne44
@heatherjoanne44
@lbnascimento So I have redesigned it where it shares a single db instance. My connection string looks like this: var conectionString = $"Filename={_databaseFileName};Connection=Direct;"; I have tried Task.Run and Parallel.Invoke(() on the database method that call Insert on the database that inserts a list of records. This insert method is the method that is causing a performance hit. The contents of the database method looks like this:
var collection = _db.GetCollection<IdentityKeyPair>(_tableName);
collection.Insert(identityKeyPairs);
Although this way has significantly improved performance, I still am having a performance hit. If you have any tips on improving the performance for this insert, that would be greatly appreciated
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 If you're inserting a lot of documents in a single collection, you could use the LiteCollection<T>.Insert(IEnumerable<T>) overload to improve performance
Or you could run all the inserts in a single transaction (that's what LiteCollection<T>.Insert(IEnumerable<T>) does)
This will be a lot faster than, for example, using a foreach loop that calls LiteCollection<T>.Insert(T) for every object
heatherjoanne44
@heatherjoanne44
@lbnascimento It is currently inserting a list and not through a loop at the very end. However, this whole process gets called frequently per row in a datatable. In the code I shared, identityKeyPairs is
a list: collection.Insert(identityKeyPairs); Unfortunately, it has to be like this, as we cant wait until the entire file processes to do the insert
Leonardo Nascimento
@lbnascimento
The smallest lock granularity LiteDB uses are collection-wide locks, so having multiple threads running inserts to the same collection is pointless. You can have multiple threads reading from a single collection or writing to different collections and these operations will run in parallel.
You could try using a foreach loop and calling LiteCollection<T>.Insert(T) for every object in a single transaction (between calls to db.BeginTrans() and db.Commit()). It is possible that the overhead is smaller this way, but I'm not sure.
heatherjoanne44
@heatherjoanne44
Do you mean the insert method would look something like this db.BeginTrans(); LiteCollection<T>.Insert(T); db.Commit(); and I would make this call inside the loop instead of sending the list to the insert?
Leonardo Nascimento
@lbnascimento
Something like this:
db.BeginTrans();

foreach(var item in identityKeyPairs)
    collection.Insert(item);

db.Commit();
Leonardo Nascimento
@lbnascimento
@heatherjoanne44 I ran some tests and indeed using the foreach loop in a single transaction (like the example above) should be faster. The downside compared to using Insert(IEnumerable<T>) is that in case the loop doesn't finish (process crash, power loss, exception...), all inserted items will be discarded (because they are part of an unfinished transaction).
weirdyang
@weirdyang
image.png

Hi guys,

I'm trying to use litedb as a log event storage,

I have certain properties in the properties field, which are unstructured. is it possible to query for field that might not be there?
For example querying out all logevents where there is a typeTag key present
this is my model, is it ok to use Dictionary<string, string> or should I use a list instead?
    public class LogMessage
    {
        public string Level { get; set; }

        public string Message { get; set; }

        public Dictionary<string, string> Properties { get; set; } = new Dictionary<string, string>();

    }
Leonardo Nascimento
@lbnascimento
@captmomo Dictionaries are serialized as sub-documents, so it's pretty simple to use them in queries.
@captmomo For example, if you wanted all the events which have the property Name, you could simply run the query select $ from logevents where $.Properties.Name != null
weirdyang
@weirdyang
@lbnascimento I'm trying to query for entries where there is certain key in the properties. I ended up with:
SELECT $ FROM logEvents
WHERE $.Properties.Customer != null
@lbnascimento thanks!
WTTSoftwareSolutions
@WTTSoftwareSolutions
Hey Guys,
Is there a way in v5 to load a collection from json? I know that there is the ability to load from json file using the sql syntax:
db.Execute("select $ into mycollection from $file('myfile.json')")

however my json is not in a file on disc, it is in memory. This used to work:

db.Engine.Insert("mycol", JsonSerializer.Deserialize(json).AsArray.ToArray());

There does not seem to be accessibility to the Engine from LiteDatabase anymore.

Leonardo Nascimento
@lbnascimento
@WTTSoftwareSolutions If you open your collection as var col = db.GetCollection("mycollection"), you could do something like col.Insert(JsonSerializer.DeserializeArray(json).Cast<BsonDocument>())
weirdyang
@weirdyang

is there anyway to save message

@WTTSoftwareSolutions If you open your collection as var col = db.GetCollection("mycollection"), you could do something like col.Insert(JsonSerializer.DeserializeArray(json).Cast<BsonDocument>())

thanks! been trying to figure this out too

WTTSoftwareSolutions
@WTTSoftwareSolutions
@lbnascimento Thanks for the help! For some reason it didn't work. It kept throwing exceptions about the json formatting. The json was from me using Newtonsoft.Json to serialize a table from an OleDb DataSet. I changed the way I'm doing the conversion and am now using NPoco Query to get an IEnumerable of my Poco's and passing that into the Collection Insert() method. Success!! We have now officially dumped using Access and will be completely LiteDB in the January release of our product!
weirdyang
@weirdyang
Can I store a class with a bsondocument property? or should i just store it as a string and do a query e.g "Select JSON($.MyProperty) From Collection"
iSunOfficial
@iSunOfficial
Did I make a mistake, or is this not possible to use regex?
ILiteQueryable<Class> h = class.Query().Where(x => x.Query.Count(s => Regex.IsMatch(text, @$"^(.*)({s})(net)$")) > 0);
Shady3cho
@Shady3cho
I joined just to say, yo I love this db thanks to who ever made it
weirdyang
@weirdyang
Recommendation
Single instance (second option) is much faster than multi instances. In multi instances environment, each instance must do expensive data file operations: open, lock, unlock, read, close. Also, each instance has its own cache control and, if used only for a single operation, will discard all cached pages on close of datafile. In single instance, all pages in cache are shared between all read threads.
Does this mean for a web app I should be using the litedbrepo as a singleton?
weirdyang
@weirdyang
What is the best way to implement async methods?
Can I return a Task.FromResult or do I need to use lock?