These are chat archives for Automattic/mongoose

4th
Jun 2018
panigrah
@panigrah
Jun 04 2018 11:50

hi All, I need help with a schema. I have Projects and Users. Its a many to many relationship. I am using a Membership table to hold the relationship between the Project and User.

I have to search for Project members and display them on a page. I need to search for user names, departments and titles for the members and display them and this is to be sortable.

If I do a Membership.find() will not let me search User.name, User.department. So I need to read the members and search outside of mongoose. I am looking at using Member.aggregate now. Is this the best way?

Or can I redo my schema; I considered storing the user data inside the Membership table - but then I need to update the embedded document when the user data changes.

How do you normally handle such a situation?

Kev
@lineus
Jun 04 2018 12:18
I would just store an array of user _ids in each project, eliminate the membership collection and the project reference in user. Then I would query against the project collection when I wanted to know which projects a user is connected to or which users are in a specific project etc.
Daniel Netzer
@DanielNetzer
Jun 04 2018 12:36
afternoon all, how do you cast an array of objects to become an array of those objects sub documents id's?
Kev
@lineus
Jun 04 2018 12:37
can you show the schema @DanielNetzer
Daniel Netzer
@DanielNetzer
Jun 04 2018 12:37
i'm trying to save a new BOT
const botSchema = new mongoose.Schema({
    domain: { type: String, unique: true },
    name: String,
    accessToken: String,
    active: { type: Boolean, default: false },
    settings: {
        openingMessage: {
            messageType: { type: String, enum: ['Opening', 'Text', 'Result', 'Action'], default: 'Opening' },
            message: String,
            options: [String]
        },
        apiKey: String,
        languageCode: String,
        currency: String,
        platform: String
    },
    appearance: {
        colors: {
            fabBG: { type: String, default: '#2196f3' },
            chatTop: { type: String, default: '#2196f3' },
            chatControls: { type: String, default: '#2196f3' },
            botMessageBG: { type: String, default: '#2196f3' },
            chatControlsBG: { type: String, default: '#f2f5f7' },
            userMessageBG: { type: String, default: '#f2f5f7' },
        },
        icons: {
            openIcon: String,
            closeIcon: String,
            avatar: String
        },
        position: {
            offsetY: { type: Number, default: 0 },
            offsetX: { type: Number, default: 0 },
            positionX: { type: String, default: 'right' },
            positionY: { type: String, default: 'bottom' }
        }
    },
    categories: [{ type: mongoose.Schema.Types.ObjectId, ref: 'BotCategory' }]
}, { timestamps: true });
but on the req.body the categories array from the frontend is populated with the real objects
like so
categories:
[Node]    [ { ai: [Object],
[Node]        products: [Array],
[Node]        conversation: [],
[Node]        domain: 'cupoteststore.myshopify.com',
[Node]        platformId: '35968679993',
[Node]        createdAt: '2018-06-02T15:11:39.008Z',
[Node]        name: 'Accessories fdadfs',
[Node]        description: 'Some test description',
[Node]        imageUrl: 'https://cdn.shopify.com/s/files/1/0022/8242/2329/collections/Capture.PNG?v=1524299567',
[Node]        viewUrl: 'https://cupoteststore.myshopify.com/collections/t-shirt',
[Node]        updatedAt: '2018-06-04T12:32:44.537Z',
[Node]        id: '5b12b3aadbe69f60eab59fcb' },
how do I update the main Bot document and all the referenced category documents?
Kev
@lineus
Jun 04 2018 12:42
you can just use map: const catIDs = categories.map(c => c.id)
Daniel Netzer
@DanielNetzer
Jun 04 2018 12:42
to update the bot document
but how can i update the referenced sub documents before i'm the saving the bot doc?
more accurate is "what is the best way to do that?"
Kev
@lineus
Jun 04 2018 12:44
oh, I was answering your first question. hadn't seen the last one yet
in addition to the categories, you have the id of the bot right?
Daniel Netzer
@DanielNetzer
Jun 04 2018 12:48
yea, the entire new Bot object
Kev
@lineus
Jun 04 2018 12:49

how can i update the referenced sub documents before i'm the saving the bot doc

what do you mean by update before save? what exactly are you doing here?

Daniel Netzer
@DanielNetzer
Jun 04 2018 12:50
i'm sending the entire populated BOT doc
in it there are all sorts of sub docs, one of them is the categories
I want to update and save the categories sub docs, then save the updated bot with the ObjectID's of the populated categories
I can iterate easily over each of them and do it, but I was wondering if there is a moongose way to update them
in a simpler query
Kev
@lineus
Jun 04 2018 12:53
categories aren't subdocs, they're references to another collection. you'll need to save each category in the BotCategory collection and save the references in the Bot Doc.
so you can do
const catIDs = req.body.categories.map(c => c.id)
await BotCategory.create(req.body.categories)
const doc = Object.assign({}, req.body)
doc.categories = catIDs
Bot.create(doc).then(etc)
Daniel Netzer
@DanielNetzer
Jun 04 2018 12:58
BotCategory.create(req.body.categories)
this will just iterate over the id's and save/update them right? no mongoose ready made query to do this?
Kev
@lineus
Jun 04 2018 12:58
oh, the bot is new, but the categories are not?
Daniel Netzer
@DanielNetzer
Jun 04 2018 12:58
none of them is new
Kev
@lineus
Jun 04 2018 12:58
if that's the case, then you don't need to save them, just eliminate that step entirely
Daniel Netzer
@DanielNetzer
Jun 04 2018 12:59
its a findOneAndUpdate query
on them all
Kev
@lineus
Jun 04 2018 13:04
const catIDs = req.body.categories.map(c => c.id)
const doc = Object.assign({}, req.body)
doc.categories = catIDs
findOneAndUpdate({ _id: doc._id },doc, { upsert: true}).then(etc)
mongoose should cast the array of category id strings to objectIds for you
Daniel Netzer
@DanielNetzer
Jun 04 2018 13:14
thank you very much for your help @lineus
Kev
@lineus
Jun 04 2018 13:19
anytime :)
Daniel Netzer
@DanielNetzer
Jun 04 2018 13:48
another question :D, how do you handle findOneAndUpdate with complexed Objects when you want to only add new data without overriding the old data in the document?
Daniel Netzer
@DanielNetzer
Jun 04 2018 13:49
damn, that was fast
oh that's for the question earlier
Kev
@lineus
Jun 04 2018 13:49
yeah, hehe
I'm not that fast ;)
you have to remove the properties in the update doc (2nd arg) that you don't want to change. Otherwise it will update the doc(s) that match the condition to the update doc's values.
Daniel Netzer
@DanielNetzer
Jun 04 2018 13:53
i'm trying to make a dot notation method to solve that
or even using mongo-dot-notation, but it aint working for some reason
Kev
@lineus
Jun 04 2018 13:57

when you want to only add new data

do you mean a path that is in your schema that just isn't set on the doc in the db yet?

or do you mean a new path that isn't in your schema?
Daniel Netzer
@DanielNetzer
Jun 04 2018 14:01
i have something like this, botCategory.ai.utterances which is an Array of objects, a part of that object should be updated while the other part shouldn't be touched
Daniel Netzer
@DanielNetzer
Jun 04 2018 14:14
@lineus any thoughts?
panigrah
@panigrah
Jun 04 2018 14:15
@lineus you are being overworked! The member list can run into 1000s and I have to store additional data such as date joined, the type of membership, active or not. I worked out a way using aggregate pipeline - but worry about its efficiency.
const query = { role: 'basic', active: true };
match = { 'user.name': new RegExp( text, 'i' ) }
Membership.aggregate([
      { $match: query },
      { $lookup: { from: 'users', localField: 'user', foreignField: '_id', as: 'user' } },
      { $unwind: '$user' },
      { $match: match },
      { $sort: { 'user.name': 1, 'user._id': 1 }},
      { $limit: limit },
      { $addFields: { id: '$_id', 'user.id': '$user._id' } },
@DanielNetzer if utterances is an array - look at the push call?
Daniel Netzer
@DanielNetzer
Jun 04 2018 14:17
push wont help here as it will create duplicates
maybe my data structure is not right
panigrah
@panigrah
Jun 04 2018 14:19
you can do a $addToSet instead of push to avoid duplicates
Daniel Netzer
@DanielNetzer
Jun 04 2018 14:21

imagine an array with plenty of objects like so

{
    text: string;
    intentName: string;
    entityLabels: EntityLabel[];
}

when a user reinstall the application I get from a 3rd party an array of those objects without the entityLabels field, I want to update when text and intentName already exists (without removing the entityLabels) and create when not found

i'm afraid it might require a much more complex querying then what mongo can offer
panigrah
@panigrah
Jun 04 2018 14:23
ah, then addToSet will not work because it expects the object to be identical.
Daniel Netzer
@DanielNetzer
Jun 04 2018 14:24
might have to break this down to find, if found do the complex preservation, if not create
sounds like step one to me :D
panigrah
@panigrah
Jun 04 2018 14:25
what about this
update({ utternaces: {$elemMatch: {intentName: input.intentName}}, {$set: {'utterances.$.text', input.text}}})
there may be a syntax error in it but hoping it gives you an idea
panigrah
@panigrah
Jun 04 2018 14:31
aah wait, you want to insert if not available. sorry wasn't reading as fast as you were typing.
what if you move utterances to its own collection and only keep a reference in your parent document? So you have a related list. Then the create and update is easy
Daniel Netzer
@DanielNetzer
Jun 04 2018 14:34

can this be written as

update({ utternaces: {$elemMatch: {intentName: input.intentName, text:input.text}}, {$set: {'utterances.$.text': input.text, 'utterances.$.intentName': input.intentName}}})

?

panigrah
@panigrah
Jun 04 2018 14:35
@DanielNetzer but that would only update if the entry exists, if it doesnt then there is no insert
I think I found another way. But it will require you to change the schema slightly. See here https://stackoverflow.com/questions/10277174/upsert-in-an-embedded-document
Daniel Netzer
@DanielNetzer
Jun 04 2018 14:38
cant remember why its not seperated now, but I do remember there was a reason

assuming the data structure is as follows

platformId: String,
    name: String,
    imageUrl: String,
    description: String,
    domain: String,
    viewUrl: String,
    ai: {
        utterances: [{
            text: String,
            intentName: String,
            entityLabels: [{
                entityValue: String,
                entityName: String,
                startCharIndex: Number,
                endCharIndex: Number
            }]
        }],
        entities: [String]
    },
...

how can I change/simplify it to make this update/create easier?

panigrah
@panigrah
Jun 04 2018 14:49
reading through the stack exchange link I posted earlier - I am not sure how to define the document schema in that case. But it looks like it may work in your case.
I think it will be a two step process for you - simpler to follow and implement. Find and update and if it doesn't exist then push
Daniel Netzer
@DanielNetzer
Jun 04 2018 14:56
BotCategory.findOne({ domain: category.domain, platformId: category.platformId })
                .exec((err, existingCategory) => {

                    if (existingCategory) {
                        const updatedCategory = <BotCategoryModel>existingCategory.toJSON();

                        updatedCategory.ai.utterances.forEach(utterance => console.log(utterance));

                        // existingCategory.update()
                    } else {
                        const botCategory = new BotCategory(category);
                        botCategory.save((err, newBotCategory) => {
                            if (err) { return cb(err); }
                            res.locals.savedBotCatNames.push(category.name);
                            res.locals.savedBotCatIds.push(newBotCategory.id);
                            cb();
                        });
                    }

                });
got here so far, should be much easier from here. just need to debug a bit
panigrah
@panigrah
Jun 04 2018 15:00
good luck @DanielNetzer
Daniel Netzer
@DanielNetzer
Jun 04 2018 15:00
ty :F
Daniel Netzer
@DanielNetzer
Jun 04 2018 15:41
BotCategory.findOne({ domain: category.domain, platformId: category.platformId })
                .exec((err, existingCategory) => {

                    if (existingCategory) {
                        const updatedCategory = <BotCategoryModel>existingCategory.toJSON();

                        updatedCategory.ai.utterances.forEach(utterance => {
                            if (utterance.entityLabels.length > 0) {

                                const updatedUtterance = category.ai.utterances.find((val) => {
                                    return (val.intentName === utterance.intentName && val.text === utterance.text);
                                });

                                utterance.text = updatedUtterance.text;
                                utterance.intentName = updatedUtterance.intentName;

                            }
                        });

                        existingCategory.update(updatedCategory, (err, raw) => {
                            if (err) { return cb(err); }
                            res.locals.savedBotCatNames.push(updatedCategory.name);
                            res.locals.savedBotCatIds.push(updatedCategory.id);
                            cb();
                        });
                    } else {
                        const botCategory = new BotCategory(category);
                        botCategory.save((err, newBotCategory) => {
                            if (err) { return cb(err); }
                            res.locals.savedBotCatNames.push(category.name);
                            res.locals.savedBotCatIds.push(newBotCategory.id);
                            cb();
                        });
                    }

                });
@panigrah worst solution possible, but quick and dirty is my methodology at the moment
warlockD
@warlockdn
Jun 04 2018 18:43
is there a way to find and replace a document ?
warlockD
@warlockdn
Jun 04 2018 19:22
@lineus always to the rescue ;) thanks it worked