Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • May 26 11:00
    potnoddle edited #2192
  • May 26 05:53
    devmaestro1 edited #2193
  • May 26 05:26
    devmaestro1 edited #2193
  • May 26 04:48
    devmaestro1 labeled #2193
  • May 26 04:48
    devmaestro1 opened #2193
  • May 25 01:14
    ronnieoverby commented #298
  • May 25 01:13
    ronnieoverby commented #298
  • May 24 22:47
    visschersm commented #2088
  • May 23 12:26
    chucksullivancvs commented #1767
  • May 22 16:07
    potnoddle labeled #2192
  • May 22 16:07
    potnoddle opened #2192
  • May 22 09:05
    roubachof commented #2143
  • May 21 15:52
    Lonli-Lokli commented #2143
  • May 21 15:47
    Lonli-Lokli commented #2143
  • May 21 07:24
    gbthakkar commented #1767
  • May 21 05:50
    superlucker edited #2190
  • May 21 03:02
    superlucker labeled #2190
  • May 21 03:02
    superlucker opened #2190
  • May 20 23:16
    devmaestro1 closed #2189
  • May 20 23:15
    devmaestro1 edited #2189
weirdyang
@weirdyang
mbdavid/LiteDB#169 <<- DateTimeOffSet has builtin support, so I'm not sure what is the issue
Leonardo Nascimento
@lbnascimento
@captmomo Are you using any custom deserializer?
weirdyang
@weirdyang
No.
The JsonConvert is from NewtonSoft, could that be an issue?
Leonardo Nascimento
@lbnascimento
I'm not sure, could you show me how you're using objects of this class? How you're inserting, how they're being serialized/deserialized etc. Where is this error happening?
weirdyang
@weirdyang
   internal async Task<IEnumerable<LogMessage>> ProcessLogEvents(IEnumerable<LogEvent> logEvents, int clientId)
        {
            List<LogMessage> messages = new List<LogMessage>();
            foreach (var evt in logEvents)
            {
                var logMessage = new LogMessage()
                {
                    Level = evt.Level,
                    Message = evt.RenderMessage(),
                    TimeStamp = evt.Timestamp,
                    UserId = clientId
                };
                if (evt.Exception != null)
                {
                    logMessage.Exception = evt.Exception.ToString();
                }

                if (evt.Properties.Any())
                {

                    string complete = await Task.Run(() => Newtonsoft.Json.JsonConvert.SerializeObject(evt.Properties));
                    logMessage.Properties = LiteDB.JsonSerializer.Deserialize(complete).AsDocument;

                }
                messages.Add(logMessage);
            }
            return messages;
        }
Converting the log events into the model for storage
//retrieval

                return _repository.Query<T>()
                .OrderBy(x => x.Id, order: Query.Descending)
                .Where(predicate)
                .Skip(skip)
                .Limit(take)
                .ToList();
            return Ok(results);
the error happens when I return the ActionResult
the last step
Leonardo Nascimento
@lbnascimento
I see you're deserializing LogEvent.Properties with NewtonSoft.Json and serializing it right after with LiteDB.JsonSerializer, there's probably some form of incompatibility (I believe Newtonsoft serializes dates as strings, while LiteDB serializes them to a custom document)
weirdyang
@weirdyang
but there's no datetime in the properteis.
properties*
Leonardo Nascimento
@lbnascimento
What type is it?
weirdyang
@weirdyang
strings
{{"custom":{"TypeTag":null,"Properties":[{"Name":"message","Value":{"Value":"Yesenia-orange"}}]},"SourceContext":{"Value":"Sample.Program"},"Application":{"Value":"Sample"}}}

{{"SourceContext":{"Value":"Sample.Program"},"Application":{"Value":"Sample"}}}
Leonardo Nascimento
@lbnascimento
I believe Newtonsoft is getting lost trying to serialize a BsonValue
weirdyang
@weirdyang
ahah oh no
Leonardo Nascimento
@lbnascimento
Could you try something like: logMessage.Properties = BsonMapper.Global.Serialize(evt.Properties).AsDocument?
weirdyang
@weirdyang
Ok
Leonardo Nascimento
@lbnascimento
Also, if it makes your class easier to use, you can make LogMessage.Properties to be a Dictionary<string,string> and the BsonMapper will serialize it to a document when inserting
weirdyang
@weirdyang

I tried that but it ends up becoming

{{"custom": "{"TypeTag":null,"Properties":[{"Name":"message","Value":{"Value":"Yesenia-orange"}}]},""}

i.e the value of custom is a string, and I cant access the properties.
using a bsondocument, i can do Properties.Custom.TypeTag
if I use dictionary<string, string> i need to do a json(properties.custom) then access the properties
weirdyang
@weirdyang
Newtonsoft.Json.JsonSerializationException: Error getting value from 'AsBoolean' on 'LiteDB.BsonValue'. ---> System.InvalidCastException: Unable to cast object of type 'System.String' to type 'System.Boolean'. at lambda_method(Closure , Object )
Leonardo Nascimento
@lbnascimento
Are you still calling the NewtonSoft serializer?
weirdyang
@weirdyang
                if (evt.Properties.Any())
                {

                    logMessage.Properties = BsonMapper.Global.Serialize(evt.Properties).AsDocument;

                }
                messages.Add(logMessage);
I'm using a new database too
Leonardo Nascimento
@lbnascimento
In which line is this exception being thrown?
weirdyang
@weirdyang

When the controller action returns the result in

Ok(result)

//retrieval

                return _repository.Query<T>()
                .OrderBy(x => x.Id, order: Query.Descending)
                .Where(predicate)
                .Skip(skip)
                .Limit(take)
                .ToList();
           var results = getLogMessages();
            return Ok(results);
Leonardo Nascimento
@lbnascimento
Well, the Newtonsoft serializer is being called somewhere (maybe inside this Ok method), and it is failing to serialize BsonValues (it's trying to cast string to DateTime, string to Boolean etc.)
weirdyang
@weirdyang
yea, the mvc builder uses it. the weird thing is none of the propeties have boolean or datetime
Leonardo Nascimento
@lbnascimento
The class BsonValue has properties for all valid bson types: AsBoolean, AsInt32, AsDateTime...
These properties obviously only work if the internal value of that BsonValue matches the property, it will throw otherwise
Newtonsoft is probably trying to serialize every property in the BsonValue (BsonDocument inherits from BsonValue) and that's why it's throwing
weirdyang
@weirdyang
Thansk for the hint @lbnascimento :)
    public class BsonValueConverter : Newtonsoft.Json.JsonConverter<BsonValue>
    {
        public override BsonValue ReadJson(Newtonsoft.Json.JsonReader reader, Type objectType, [AllowNull] BsonValue existingValue, bool hasExistingValue, Newtonsoft.Json.JsonSerializer serializer)
        {
            throw new NotImplementedException();
        }

        public override void WriteJson(Newtonsoft.Json.JsonWriter writer, [AllowNull] BsonValue value, Newtonsoft.Json.JsonSerializer serializer)
        {
            writer.WriteValue(value.ToString());
        }
    }
Tim Moore
@tjmoore

Hi, I'm working on a big project refactor and part of that involves abstracting the data model POCO classes from the data access layer. This allows for multiple database options, possibly MongoDB at least, and removing dependencies on the database assemblies in the rest of the project when using the data model classes.

I've been changing ObjectId properties to string and using Guids for ID instead, although these may not be as efficient as an ObjectId. I have a problem though with this approach on deserialising existing databases as ObjectId doesn't cast to string.

Is there a recommended way around this? I suppose I could register a mapping of ObjectId to string, but would that prevent me serialising an ObjectId if I really wanted to?

Alternatively a generic solution that still uses ObjectId, good for LiteDB and MongoDB (as they have virtually the same thing I believe), but I need that abstraction with no dependency on LiteDB in the model and dependent non-DB code needs access to an ID as a string.

If any of that makes sense.

Leonardo Nascimento
@lbnascimento
@tjmoore While I believe it may be possible to create a context-aware mapping for ObjectId (using the UserVersion pragma or something like that), I think it would be janky.
I believe your best bet is to simply convert the ObjectId properties into strings both in your classes and in your datafiles.
Leonardo Nascimento
@lbnascimento
@tjmoore I came up with this piece of code:
var collectionNames = db.GetCollectionNames().ToList();
db.CheckpointSize = 0;

foreach (var collectionName in collectionNames)
{
    db.BeginTrans();

    var tmpColName = collectionName + "_tmp";
    var col = db.GetCollection(collectionName);
    var tmpCol = db.GetCollection(tmpColName);

    var indexes = db.Execute($"select $ from $indexes where collection = '{collectionName}'").ToList();

    foreach(var index in indexes)
    {
        tmpCol.EnsureIndex(index["name"].AsString, index["expression"].AsString, index["unique"].AsBoolean);
    }

    foreach(var doc in col.FindAll())
    {
        var newDoc = new BsonDocument();
        foreach(var kvp in doc)
        {
            if(kvp.Value.IsObjectId)
            {
                newDoc[kvp.Key] = kvp.Value.AsObjectId.ToString();
            }
            else
            {
                newDoc[kvp.Key] = kvp.Value;
            }
        }

        tmpCol.Insert(newDoc);
    }

    db.Commit();

    db.DropCollection(collectionName);
    db.RenameCollection(tmpColName, collectionName);

    db.Checkpoint();
}

db.CheckpointSize = 1000;
db.Rebuild();
It processes all collections in a datafile, transforming ObjectIds into strings. I ran some tests and it seems to run fine - of course, you should do your own testing, to ensure that it works properly and that it fits your needs.
If you have a distributed aplication (like a mobile app), you can use db.UserVersion to determine whether the file has to be processed or not
Tim Moore
@tjmoore
@lbnascimento Cool, thanks for that. I'll give it a go. It does seem like the simplest way to do it.
Nate Komodo
@NateKomodo
How does one use UpdateMany? the example is to use an anonymous type, but i get a type conversion error when i do this
db.GetCollection<Setting>("settings").UpdateMany(s => new { setting.Name, setting.Value }, s => s.Name == setting.Name);
The example is Extend expression must return an anonymous class to be merge with entities. Eg: col.UpdateMany(x => new { Name = x.Name.ToUpper() }, x => x.Age > 10)
weirdyang
@weirdyang
col.UpdateMany(x => new Customer { Name = x.Name.ToUpper(), Salary: 100 }, x => x.Name == "John")
image.png