These are chat archives for Automattic/mongoose

19th
Apr 2018
Arun Gadag
@arun-awnics
Apr 19 2018 08:39
Hey guys, I have 2 tables
One is message table and the other one is group_message_table
group_message_map table
I am storing a messageId which is the PK of message table in messageId of group_message_map table
Now, I want to filter the messages depending the messageId present in group_message_map
Arun Gadag
@arun-awnics
Apr 19 2018 08:48
I tried this:
getLimitedMessages(receiverId, senderId, page, size, callback) {
var messages = [];
        GroupMessageMap.find({ receiverId: receiverId, senderId: senderId })
            .skip(parseInt(((size * page) - size)))
            .limit(parseInt(size))
            .sort({ $natural: -1 })
            .exec(function(err, maps){
                      maps.map(map => {
                           Message.find({_id: map.messageId})
                      }, function(err, message){
                                  messages.push(message);
                                  console.log(message); //prints messages here
                          });
            });
console.log(messages); // prints empty array
    }
@lineus
You there buddy?
Kev
@lineus
Apr 19 2018 08:52
Hey @arun-awnics ! I slept in til 4:30 today, just drinking coffee and waking up :)
Arun Gadag
@arun-awnics
Apr 19 2018 08:53
Didn't mean to interrupt you. Take your time :)
Kev
@lineus
Apr 19 2018 08:57
@arun-awnics Due to the async nature of the find calls, that last console.log will execute before the GroupMessageMap.find(..).exec() callback returns.
Arun Gadag
@arun-awnics
Apr 19 2018 08:58
Hmm... I assumed so
Is there a way to fix it?
Sivabalan
@sivabalan02
Apr 19 2018 09:04
@arun-awnics you can use callback for that
Arun Gadag
@arun-awnics
Apr 19 2018 09:05
I have to wait until the loop is completed and then use callback(messages)
@sivabalan02 But I am not able to get it
getLimitedMessages(receiverId, senderId, page, size, callback) {
var messages = [];
        GroupMessageMap.find({ receiverId: receiverId, senderId: senderId })
            .skip(parseInt(((size * page) - size)))
            .limit(parseInt(size))
            .sort({ $natural: -1 })
            .exec(function(err, maps){
                      maps.map(map => {
                           Message.find({_id: map.messageId})
                      }, function(err, message){
                                  messages.push(message);
                                  callback(messages); // this throws an error
                                  console.log(message); //prints messages here
                          });
            });
console.log(messages); // prints empty array
    }
Sivabalan
@sivabalan02
Apr 19 2018 09:06
where is the loop here?
Kev
@lineus
Apr 19 2018 09:06
maps.map()
Arun Gadag
@arun-awnics
Apr 19 2018 09:06
maps.map
Sivabalan
@sivabalan02
Apr 19 2018 09:07
sorry my eyes were bad. ok then it will be like this
let totalLength = maps.length
maps.map(map, index => {
    Message.find({_id: map.messageId})
}, function(err, message){
            messages.push(message);
            callback(messages); // this throws an error
            console.log(message); //prints messages here
            if(totalLength - 1 === index)
            // your callback
    });
Arun Gadag
@arun-awnics
Apr 19 2018 09:13
Let me try that
Kev
@lineus
Apr 19 2018 09:54
@nsuchy I don't see why that wouldn't work, though arrays of arrays can add a layer of complexity to future queries (ie. adding a search feature later) and depending on the size of each message, the size of the conversation documents could become unwieldy, eventually even hitting the 16Mb limit.
panigrah
@panigrah
Apr 19 2018 10:31
@arun-awnics why not use await to block until the call returns at each step?
Arun Gadag
@arun-awnics
Apr 19 2018 10:32
@arun-awnics why not use await to block until the call returns at each step?
I am not sure how that is done
Arun Gadag
@arun-awnics
Apr 19 2018 10:41
@sivabalan02 Your solution didn't work
panigrah
@panigrah
Apr 19 2018 10:41
const results = await GroupMesaageMap.find(...) should wait for the find to finish and results will contain the array of results that you can loop over.
Arun Gadag
@arun-awnics
Apr 19 2018 10:41
let totalLength = maps.length
maps.map(map, index => {
    Message.find({_id: map.messageId})
}, function(err, message){
            messages.push(message);
            callback(messages); // this throws an error
            console.log(message); //prints messages here
            if(totalLength - 1 === index)
            // your callback
    });
I am guessing index is out of scope in the if condition @sivabalan02
Sivabalan
@sivabalan02
Apr 19 2018 10:43
it can't be a out of scope
panigrah
@panigrah
Apr 19 2018 10:43
You can do the same thing with the const message = await Message.find(). Then you will have message before@proceeding to next step
Sivabalan
@sivabalan02
Apr 19 2018 10:44
as @panigrah said await is also an option
inside index scope only Message.find come s
Arun Gadag
@arun-awnics
Apr 19 2018 10:46
getLimitedMessages(receiverId, senderId, page, size, callback) {
var messages = [];
        const message = await GroupMessageMap.find({ receiverId: receiverId, senderId: senderId })
            .skip(parseInt(((size * page) - size)))
            .limit(parseInt(size))
            .sort({ $natural: -1 })
            .exec(function(err, maps){
                      maps.map(map => {
                           Message.find({_id: map.messageId})
                      }, function(err, message){
                                  messages.push(message);
                                  callback(messages); // this throws an error
                                  console.log(message); //prints messages here
                          });
            });
console.log(message); 
    }
Is this what you are suggesting? @panigrah
panigrah
@panigrah
Apr 19 2018 10:48
You don’t need exec and callbacks if you use await
Read up online on await
On my phone and can’t type long code. perhaps @sivabalan02 can show you
Arun Gadag
@arun-awnics
Apr 19 2018 10:54
I have to send the updated values, i.e., messages back to the calling function
How else would the calling function gets the messages if I don't send them back?
Kev
@lineus
Apr 19 2018 10:59

maybe something like this @arun-awnics

async function getLimitedMessages(receiverId, senderId, page, size, callback) {
  let promises = [];
  let found = await GroupMessageMap.find({ receiverId: receiverId, senderId: senderId })
            .skip(parseInt(((size * page) - size)))
            .limit(parseInt(size))
            .sort({ $natural: -1 })
            .exec()

  found.map(doc => {
    promises.push(Message.find({_id: doc.messageId}));
  });

  let messages = await Promise.all(promises)

  callback(messages)
}

I didn't test this out, but who knows, maybe I got lucky :)

Arun Gadag
@arun-awnics
Apr 19 2018 11:06
Alright... Let me try :D
Kev
@lineus
Apr 19 2018 11:07
also, you could almost certainly instead of querying the db for each individual message, query the db once with the array of message ids thus only ever making 2 queries, instead of 1 + found.length queries. but let's see if we can get this working first :)
niluroy
@niluroy
Apr 19 2018 11:23
image.png
Arun Gadag
@arun-awnics
Apr 19 2018 11:23
Hey @lineus
That didn't work either
There are some compilation issues
@niluroy has shared the same in the screenshot above
She will take over the discussion from here
Nathaniel Suchy
@nsuchy
Apr 19 2018 11:37
@lineus I’m open to new ideas. The 16MB limit sounds bad. Any alternate approaches?
Kev
@lineus
Apr 19 2018 11:52
@niluroy are you using babel-polyfill? I think that's required to use async/await with babel.
Nathaniel Suchy
@nsuchy
Apr 19 2018 11:53
@lineus Is there a limit to the number of documents in a collection assuming I have the disk space? Could I store every message as a separate document and use a query to grab them when needed. What performance challenges do I face?
niluroy
@niluroy
Apr 19 2018 11:53
@lineus yes
Kev
@lineus
Apr 19 2018 11:54
@niluroy do you have other async functions that are working?
@nsuchy not that I'm aware of, but I'd have to look at the mongodb docs to say for sure.
Nathaniel Suchy
@nsuchy
Apr 19 2018 11:56
@lineus The idea would be I’d have a collection of conversations and another collection of messages. I’d pull with queries as needed. Right now performance isn’t a huge issue but in the rare event usage of my tiny app explodes well you know 😑😑😑
niluroy
@niluroy
Apr 19 2018 11:58
@lineus not used it yet
@niluroy, it might be worth it to create a small test script that you can compile with babel and get that working with async first, then take what you learn over to your app. I'd offer more specific help if I had any meaningful experience with babel.
I'll see what I can figure out though too :)
niluroy
@niluroy
Apr 19 2018 12:02
okay
Nathaniel Suchy
@nsuchy
Apr 19 2018 12:13
@lineus The biggest concern is a collection with millions of object will slow stuff down significantly
The more users I have, the more monetization options I have, so buying 10 super powered MongoDB Nodes and clustering them is an option although long term I'll have to find a better solution than buying more servers.
I'll try to get it going with polyfill next instead of runtime. I'm shooting in the dark here ;)
panigrah
@panigrah
Apr 19 2018 13:13
@niluroy, @lineus is correct you need polyfill. See here how to get async and await setup. https://stackoverflow.com/questions/33527653/babel-6-regeneratorruntime-is-not-defined
Kev
@lineus
Apr 19 2018 13:18
@panigrah maybe polyfill is a dependency of babel-runtime, I got it working without explicitly using polyfill in my gist above.
niluroy
@niluroy
Apr 19 2018 13:34
async and await is working fine in other cases..but still after trying this code getting the empty messages array
async getLimitedMessages(receiverId, senderId, page, size, callback) {
        var promises = [];
        var found = await GroupMessageMap.find({ receiverId: receiverId, senderId: senderId })
            .skip(parseInt(((size * page) - size)))
            .limit(parseInt(size))
            .sort({ $natural: -1 })
            .exec()
        found.map(doc => {
            promises.push(Message.find({ _id: doc.messageId }));
        });
        var messages = await Promise.all(promises);
        console.log('messages are: ' + messages); //still it is empty
        callback(messages);
    }
Kev
@lineus
Apr 19 2018 13:36
is doc.messageId a string or a BSON ObjectId?
niluroy
@niluroy
Apr 19 2018 13:37
string
Kev
@lineus
Apr 19 2018 13:39

so if you do:

        found.map(doc => {
            console.dir(doc.messageId)
            promises.push(Message.find({ _id: doc.messageId }));
        });

the console logs are strings instead of object ids?

even still, I would expect mongoose to cast that string to an objectId for you
in the Messages.find() call
as a side note, that should probably be Messages.findOne() since you're explicitly looking for one message id.
Kev
@lineus
Apr 19 2018 13:47
@niluroy give me a sec, I think I know what's happening, I just want to mock it up before I give anymore bad advice ;)
Kev
@lineus
Apr 19 2018 13:56
@niluroy here's a gist that shows an implementation of that function, though I didn't take the time to create a second collection ( that really shouldn't matter though ).
even if I change the findOne to search for .id instead of ._id, it still works, as mongoose casts the string to an object id before sending the query.
Kev
@lineus
Apr 19 2018 14:03
@niluroy maybe console.log the doc.messageId in the map function before pushing the promise and share a sample of one here
Nate Mallison
@NJM8
Apr 19 2018 14:06
Hi all, are the results from queries immutable? I was trying to add/ delete properties from the array of objects returned and nothing would change
  return db.Image.find().sort({'updatedAt': -1}).limit(10)
    .select({ searchQuery: 1, createdAt: 1, _id:0 })
    .exec().then(data => {
      data.forEach(item => {
        const date = new Date(item.createdAt);
        item.searchDate = `${date.toDateString()} ${date.toTimeString()}`;
      })
      return res.json(data);
item remains unchanged, I had to map over data and replace it
Kev
@lineus
Apr 19 2018 14:10
@NJM8 do you just intend on returning the searchDate property to the http client without saving it to the db?
Nate Mallison
@NJM8
Apr 19 2018 14:13
yes, it's just a simple image search api, this route just serves the last 10 searches, title and date only, but I got my answer that the query result is a mongoose document obj not a plain js obj. so I added .lean() to the query,
I was just wanting to make the search date more readable
Kev
@lineus
Apr 19 2018 14:14
you can add a virtual to the Image Schema that will allow you to include it
imageSchema.virtual('searchDate').get(function() { 
  return this._searchDate
}).set(function(val) {
  this._searchDate = val
})
const Image = mongoose.model(...)
with the virtual set on the schema, you should be able to set and get the property like your function above and see the results in your http client
Nate Mallison
@NJM8
Apr 19 2018 14:16
Ah very cool, thanks
Kev
@lineus
Apr 19 2018 14:16
anytime :)
Nate Mallison
@NJM8
Apr 19 2018 14:16
will that allow you to remove props as well?
oh, or change the name of a prop?
basically this is what I want:
searchQuery    "coffee"
searchDate    "Thu Apr 19 2018 10:15:46 GMT-0400 (EDT)"
this is what I'm getting:
searchQuery    "coffee"
createdAt    "2018-04-19T14:15:46.431Z"
so changing createdAt to searchDate and making the date string more readable. Mapping over the data to make a new object worked and so did .lean so I can modify the returned object.
I guess with the virtual I could not return the createdAt property and then call the virtual method to add the formatted version and property back in
Kev
@lineus
Apr 19 2018 14:22
this inside the virtual is the document, so you can get at any of the properties.
imageSchema.virtual('searchDate').get(function() { 
  return this.createdAt // add formatting etc
})
should work just fine.
Nate Mallison
@NJM8
Apr 19 2018 14:28
cool, thanks
Kev
@lineus
Apr 19 2018 14:33
just a note, you mentioned potentially using .lean(), I'm quite certain this will mean the docs that get returned from the query won't have the virtuals on them.
panigrah
@panigrah
Apr 19 2018 23:14

Have a question about best practice when there are multiple related documents that need updating or checking. Will use @niluroy ‘s use case as example:

Groups have Members who have Posts. A post needs a text message, a member and a group.

When I add a post, I need three different checks - is group valid ID, is member valid ID, does member belong to group?

In SQL I could do a join to figure out all 3 conditions in one call. But my only option in mongoose seems to be to check each condition sequentially.

Yes I could just check to make sure that “member and group” exist in the relationship document that stores both. But this is a hypothetical scenario!

Is there a pattern that works better than 3 separate calls and then a fourth to update the document? I am concerned about performance and “atomicity” of the check.