These are chat archives for Automattic/mongoose

27th
Sep 2017
Paul "Joey" Clark
@joeytwiddle
Sep 27 2017 03:32

@karthikeyana__twitter Why do you want to?

You can stop it from loading with query.select(‘-_id’)

Or you can get plain objects using query.lean() or doc.toObject() that will allow delete but not .save()

KARTHIKEYAN A
@karthikeyana__twitter
Sep 27 2017 03:47
ok @joeytwiddle Thanks
johnnyXcrane
@johnnyXcrane
Sep 27 2017 08:41

Can someone help me with Mongoose/Mongodb?
How can I delete one part of my document?

var keywordSchema = new mongoose.Schema({
    url: String,
    name: String,
    rankings: [
        { "date": Date, "rank": Number }    
    ],
    top100: [
        { "url": String, "rank": Number }
    ]
});

This is my schema. I want to clear top100 that it's empty again

Dennis Belany
@azarus
Sep 27 2017 09:42
hi
Mårten Pettersson
@mtnptrsn
Sep 27 2017 09:43
hi
Dennis Belany
@azarus
Sep 27 2017 09:43
@azarus
Hi i have an aggregation grouping query that takes 20 seconds to execute. This query is used for realtime anytics to visualize data on a dashboard.
Can someone help me debug my schema indexes and query to speed it up? I have one write operation every second that just writes the unix timestamp and the current stats of a sensor.
And in my query i want to search in any historical range. And ths data can be grouped by 5 minute, 10 minute, 30 minute, hourly, 2 hourly and so on levels.
i didn't do pre aggregation since i may need additional ranges, like 5 minues, 10 minutes, we may need 15 minutes data later if i change my mind
Also i wanted to analyise the past 5 minutes, past 1 hours to check if the sensor shows outlying values, and detect common patterns.
The query is super slow. Can someone help me out?
Patryk
@spartanPAGE
Sep 27 2017 10:09
Hi, I'd like to update an array element and I'm having hard time with figuring this one out using findByIdAndUpdate.
Dennis Belany
@azarus
Sep 27 2017 10:18
you can just use it like this:
myDocument.myArray.push(newItem);
myDocument.save()
after find
for update operation the following works:
https://docs.mongodb.com/manual/reference/operator/update-array/
{
 $addToSet: {
    letters: [ "c", "d" ] 
  } 
}
this adds the array [C,D ] to the array
so you'll have omsething like
letters: [ ["c", "d"] ]
Patryk
@spartanPAGE
Sep 27 2017 10:20
Ok, I've resolved my problem with findOneAndUpdate with _id and array._id in query and array.$.data in $set operation. thanks anyway
Dennis Belany
@azarus
Sep 27 2017 10:20
{
 $addToSet: {
    letters: "a", 
  } 
}
does
letters: ["a"]
ok cool
Patryk
@spartanPAGE
Sep 27 2017 10:22
Ok, one more
How can I filter out some array elements?
i.e. disabled ones
{
  someStuff...
  array: [
    {
       disabled: true,
       data: 'foo"
    },
    {
       data: 'bar"
    },
    {
       disabled: true,
       data: 'biz"
    }
  ]
}
How can I exclude the disabled element so I can get this result?
{
  someStuff...
  array: [
    {
       data: 'bar"
    },
    {
       disabled: true,
       data: 'biz"
    }
  ]
}
Patryk
@spartanPAGE
Sep 27 2017 10:35
or, with more real data...

I've got this:

{
  "success": true,
  "data": {
    "_id": "59afb815d672842f248d6b02",
    "author": "john",
    "content": "test",
    "image": null,
    "book": "Nowa Era MATeMATyka klasa 3 podstawa",
    "type": "Zadanie",
    "identificator": "6/124",
    "__v": 0,
    "comments": [],
    "ratings": [],
    "solutions": [
      {
        "content": "test",
        "author": "johnCena",
        "_id": "59cb767d4a32921a240d69d4",
        "disabledBy": "johnCena",
        "disabled": true,
        "comments": [],
        "ratings": [],
        "date": "2017-09-27T09:59:25.613Z"
      },
      {
        "content": "test 2",
        "author": "johnCena",
        "_id": "59cb7e766949250ccc3ad5aa",
        "disabled": false,
        "comments": [],
        "ratings": [],
        "date": "2017-09-27T10:33:26.467Z"
      }
    ],
    "disabled": false,
    "created": "2017-09-06T08:55:49.506Z"
  }
}

as a result of this:

Exercise
    .findOne({ disabled: {$ne: true}, '_id': id })
    .select('-solutions.image')
    .exec()
    .then((data) => res.json({ success: true, data }));

How can I modify the code from above to get this:

{
  "success": true,
  "data": {
    "_id": "59afb815d672842f248d6b02",
    "author": "john",
    "content": "test",
    "image": null,
    "book": "Nowa Era MATeMATyka klasa 3 podstawa",
    "type": "Zadanie",
    "identificator": "6/124",
    "__v": 0,
    "comments": [],
    "ratings": [],
    "solutions": [
      {
        "content": "test 2",
        "author": "johnCena",
        "_id": "59cb7e766949250ccc3ad5aa",
        "disabled": false,
        "comments": [],
        "ratings": [],
        "date": "2017-09-27T10:33:26.467Z"
      }
    ],
    "disabled": false,
    "created": "2017-09-06T08:55:49.506Z"
  }
}
Ok, I'm stupid, I can just filter the data returned by query
nevermind
johnnyXcrane
@johnnyXcrane
Sep 27 2017 11:53
var keywordSchema = new mongoose.Schema({
    url: String,
    name: String,
    rankings: [
        { "date": Date, "rank": Number }    
    ],
    top100: [
        { "url": String, "rank": Number }
    ]
});
How can I replace all whats in top100 with new data? findOneAndReplace is not supported by mongoose
Malik Hassan
@muhammad786
Sep 27 2017 12:09
hey All, I wanted to know the mongo db driver exact version that is using by MONGOOSE? kindly help
Dennis Belany
@azarus
Sep 27 2017 12:32
Nobody helps here
:D
Malik Hassan
@muhammad786
Sep 27 2017 12:33
@azarus whats the issue ... may be i could help
Dennis Belany
@azarus
Sep 27 2017 12:33
Johnny i just posted an answer for someone else the same works for you. Just update the array field as you would do with regular arrays or use one of those methods listed on the link
Thanks Malik
I am having performance issues with my sensor data
When i am qeurying timeseries data from mongo
Malik Hassan
@muhammad786
Sep 27 2017 12:35
are you using indexes?
Dennis Belany
@azarus
Sep 27 2017 12:35
Thought mongo can do this super fast but still fetching the documents to grouo costs a lot of disk IO.
Yes i have indexes on two fields date and sensorType wich is a string
Malik Hassan
@muhammad786
Sep 27 2017 12:36
are you showing this data on some client? if yes then why not paging is used
Dennis Belany
@azarus
Sep 27 2017 12:36
I want to get average, min, max data out of my sensor
And visualize them in real time
Malik Hassan
@muhammad786
Sep 27 2017 12:37
ah ok, so you are using aggregation framework ?
Dennis Belany
@azarus
Sep 27 2017 12:37
Yea
I posted the query above
Scott Gilroy
@sgilroy
Sep 27 2017 12:38
@johnnyXcrane To remove an attribute from a document, you can set an attribute on a mongoose document to undefined and then call .save(), as in keyword.top100 = undefined; keyword.save();. Or use findOneAndUpdate with $unset. Replacing data can be done similarly by setting attribute keyword.top100 = [{ url: 'http://example.com', rank: 10 }]; keyword.save(); or use findOneAndUpdate.
Dennis Belany
@azarus
Sep 27 2017 12:38
Maybe i should pre-aggregate my data but then its just gets more complicated.
Malik Hassan
@muhammad786
Sep 27 2017 12:40
actually mongodb aggregation framework is not super fast as per my understanding ... so pre aggregation may solve your issues ... and definitely its bit complex
Dennis Belany
@azarus
Sep 27 2017 12:40
Right now i have like 5 million records and increasing dayli
Y
So I have to pre aggregate all of that :D
Malik Hassan
@muhammad786
Sep 27 2017 12:41
this is what i have seen on some blogs, did not do performance test myself
Dennis Belany
@azarus
Sep 27 2017 12:42
I've made my reseach and this is the only thing i have missing and sharding. How ever im not sure how i t solves my issue on the long term
Malik Hassan
@muhammad786
Sep 27 2017 12:43
your data size is very big ... and i never faced such big data set yet -)
Dennis Belany
@azarus
Sep 27 2017 12:43
After a year im gona have 500 million records and if i would like to visualize the whole year im gona end up with the same performance even if i pre aggregate.
Malik Hassan
@muhammad786
Sep 27 2017 12:43
but will try my level best to post any comments here if i get related your case ...
is it some production app or some prototype only ?
Dennis Belany
@azarus
Sep 27 2017 12:44
Such issues are solved on exchanges thought :)
Its a prototype now
Malik Hassan
@muhammad786
Sep 27 2017 12:45
Ah ok,
Dennis Belany
@azarus
Sep 27 2017 12:45
I have sensor data coming from
Malik Hassan
@muhammad786
Sep 27 2017 12:45
do you have any experience for MONGO DB atlas
?
Dennis Belany
@azarus
Sep 27 2017 12:46
A third party api and it has like 1 million sensors. Now i only tested with a single sensor :x:
Malik Hassan
@muhammad786
Sep 27 2017 12:47
OMG -)
Dennis Belany
@azarus
Sep 27 2017 12:48
Its quite huge but just one single number each
Malik Hassan
@muhammad786
Sep 27 2017 12:48
I will try my level best to get something for you my friend.
Dennis Belany
@azarus
Sep 27 2017 12:50
And no i havent tried mongo atlas. But i have an expensive dedicated server for testing with lots of resources.
Malik Hassan
@muhammad786
Sep 27 2017 12:50
hmm .. ok. thx
I m facing this error with mongoose & mongodb ATLAS ............ https://prnt.sc/gqb9sg
Dennis Belany
@azarus
Sep 27 2017 12:53
Mb Too many connections?
Malik Hassan
@muhammad786
Sep 27 2017 12:53
hmm .. I will see it.
Dennis Belany
@azarus
Sep 27 2017 12:54
Ot maybe the server is too busy to answer
stefano capra
@stecapra
Sep 27 2017 13:40

Hi, i'm trying to use populate function.
I have those models:
User.js

'use strict'

const mongoose= use('Mongoose')
const Schema = mongoose.Schema
let userSchema = new Schema({
    username: String
})
module.exports = mongoose.model('User', userSchema)

Session.js

'use strict'

const mongoose= use('Mongoose')
const Schema = mongoose.Schema

let sessionSchema = new Schema({
    uid: { type: String, require: true },
    user: { type: Schema.Types.ObjectId, ref: 'User', required: true},
})
module.exports = mongoose.model('Session', sessionSchema)

If i execute this code:

...
let session = await Session.findOne({ 'uid' : request.cookie('uid') }).populate('User')
console.log(session.user)  ===> show the user id, not the model

Instead, if use "user" instead of "User"(notice the upper char), it returns this error:

let session = await Session.findOne({ 'uid' : request.cookie('uid') }).populate('user')

MissingSchemaError: Schema hasn't been registered for model "User".
Use mongoose.model(name, schema)

Does anyone know why? What am i doing wrong?
Thanks in advance

thetaris-slack
@thetaris-slack
Sep 27 2017 13:42
H ho
any idea?
Malik Hassan
@muhammad786
Sep 27 2017 13:44
@thetaris-slack are you using mlab or local mongo db ?
thetaris-slack
@thetaris-slack
Sep 27 2017 13:44
mlab
oh wait
no
we used mlab, now I have EBS (AWS)
Mongo StateFul Set
apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo
        environment: test
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "slow"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 4Gi
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: slow
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1
  zone: eu-west-1a
  iopsPerGB: "10"
Mongo Pod Log:
017-09-27T13:40:12.572+0000 I -        [conn157437] end connection 10.2.6.93:37368 (1 connection now open)
2017-09-27T13:40:12.582+0000 I NETWORK  [thread1] connection accepted from 10.2.6.93:37376 #157438 (1 connection now open)
2017-09-27T13:40:12.583+0000 I NETWORK  [conn157438] received client metadata from 10.2.6.93:37376 conn157438: { driver: { name: "nodejs", version: "2.2.24" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.12.14-coreos" }, platform: "Node.js v6.11.1, LE, mongodb-core: 2.1.8" }
2017-09-27T13:40:12.586+0000 I -        [conn157438] end connection 10.2.6.93:37376 (1 connection now open)
Mongoose Error:
[36mDB connection error: MongoError: no primary found in replicaset[39m
0|product- | Unhandled rejection MongoError: no primary found in replicaset
0|product- |     at /opt/source/node_modules/mongodb-core/lib/topologies/replset.js:524:28
0|product- | From previous event:
Deployment of NodeJS App:
        - name: MONGODB_URI
          value: mongodb://mongo-0.mongo,mongo-1.mongo,mongo-2.mongo:27017/saveup-product-api?replicaSet=globaldb
We had 4.8.6 mongoose I now upgraded to lastest... but I fear it wont work either
Malik Hassan
@muhammad786
Sep 27 2017 13:48
I have faced such issue with mongodb in recent past & issue was directly
related to MONGODB connection string
so make sure it
var URI_Atlas = 'mongodb://username:password@' +
'foodate0-shard-00-00-qukqy.mongodb.net:27017,'+
'foodate0-shard-00-01-qukqy.mongodb.net:27017,'+
'foodate0-shard-00-02-qukqy.mongodb.net:27017/Live_db?'+
'ssl=true&replicaSet=FoodAte0-shard-0&authSource=admin';
thetaris-slack
@thetaris-slack
Sep 27 2017 13:56
TY so much. Will try and report.
thetaris-slack
@thetaris-slack
Sep 27 2017 14:01
apiVersion: apps/v1beta1
kind: StatefulSet
will replicaset param work given that it is a statefulset
replicasets for mongo on k8s are depreciated
Scott Gilroy
@sgilroy
Sep 27 2017 14:06
@stecapra You should certainly be specifying the path user not the model User with .populate. Perhaps the problem is that you are not executing the code in User.js in the context where you invoked .populate('user'). Make sure you have a require for User.js so that the User model is actually registered with mongoose.
stefano capra
@stecapra
Sep 27 2017 14:55

@sgilroy thanks for the reply.

Actually i have this code in my route file

const User = use('App/Models/User')
const Session = use('App/Models/Session')

Route.get('/get', async ({ request, response }) => {
    try {
        await Session.findOne({ 'uid' : request.cookie('uid') }).populate('user')

        return 'ok'
    } catch(e) {
        return e.stack
    }
})

Since i am i newbie, what would you do?

thetaris-slack
@thetaris-slack
Sep 27 2017 15:06
Mhh I still get Unhandled rejection MongoError: no primary found in replicaset
thetaris-slack
@thetaris-slack
Sep 27 2017 15:27
Automattic/mongoose#4596
I tried internal IPs now without success
Any help is apprieciated as we are in production already and my boss is ready to pull my head off
I am willling to hand out ethereum or bitcoin for info that lead to a solution
thetaris-slack
@thetaris-slack
Sep 27 2017 16:09
IP address resilution is not the issue
Quincy Koots
@qkoots
Sep 27 2017 16:23
Hello how can I stop mongoose/mongo from assigning a second “_id”/ObjectId in the nested object? If you check the “meterInfo” property, you will notice that it has an ObjectId assign to it.
{ "_id" : ObjectId("59cbcbd5150b2a095a385bbc"), "firstName" : "Bjordan", "lastName" : "Koots", "company" : "BKontructions", "address" : "Blue Bay 1", "phone" : "242352424", "email" : "info@qreativeweb.solutions", "password" : "$2a$10$T3WEWlyl4m2pE2KuhwE/z.oqapoiyCbbXf1vrwGR9J92kKo0qVdcW", "meterInfo" : [ { "owner" : "Quincy", "meterId" : "202212", "_id" : ObjectId("59cbcbd5150b2a095a385bbd") } ], "purchaseHistory" : [ ], "__v" : 0 }
Adam L
@MrJadaml
Sep 27 2017 16:47
@qkoots what does your schema look like?
I am also having a mongoose issue. Having trouble formulating a query that only returns instances who's sub-doc matches a provided ID. This sub-doc is in an array of nested objects.
Quincy Koots
@qkoots
Sep 27 2017 17:12
@MrJadaml
const newUserProfileSchema = new Schema(
    {
        firstName: {type: String, required: true , trim: true},
        lastName:  {type: String, required: true , trim: true},
        email:     {type: String, required: true , trim: true, unique : true},
        password:  {type: String, required: true , trim: true},
        company:   String,
        address:   {type: String, required: true , trim: true},
        phone:     String,
        purchaseHistory :[String],
        meterInfo: [{
                    owner:{type:String, required: true, trim: true},
                    meterId: {type:String, required: true, trim: true}
                }]
    }
);
Adam L
@MrJadaml
Sep 27 2017 17:55
what was the initial query you made to get that output
Quincy Koots
@qkoots
Sep 27 2017 18:44
@MrJadaml Here you go
/ Using bcrypt to hash users password.
                bcrypt.genSalt(10, function(err, salt) {
                    if(err) throw err;

                    bcrypt.hash(req.body.password, salt, (err, hash) => {
                        if(err) throw err;

                        //New user Schema that will be used for saving users data into the database.
                        let newUserProfile = new newUser({
                            firstName : req.body.firstName,
                            lastName  : req.body.lastName,
                            company   : req.body.company,
                            address   : req.body.address,
                            phone     : req.body.phone,
                            email     : req.body.email,
                            password  : hash,
                            meterInfo   : {owner :'Quincy', meterId: "202212"},
                        });

                        // Query the database and check if the provided email address has already been taken.
                        newUser.findOne({email : req.body.email}, (err, data) =>{
                            if(err) throw err;

                            // If email address is already in use, renders signup page and display error messages.
                            if(data !== null) {
                                res.render("signup", {
                                    metaTitle   : "App - Sign up",
                                    title       : "Sign Up",
                                    errorsAlert : null
                                });
                            } else if(data === null) {

                                //if email address is not in use, a new user profile will created and saved into the database
                                // with the data provided in the form fields.

                                newUserProfile.save((err, user) => {
                                    if(err) throw err;

                                    // User will be "redirected" to the login page after.
                                    res.redirect('/login');
                                });
                            }
                        });
                    });
                });