These are chat archives for Automattic/mongoose

18th
Mar 2018
Sedric Heidarizarei
@saeedhei
Mar 18 2018 13:21
How Can I insert 1.000.000 of This Document to DB? I need Dublicate insert, Just for query speed test.
const MessageMsg = new Message({
        text: iMsg,
        userId: id,
      }); 

MessageMsg.save(); // This Save 1 Document
Kev
@lineus
Mar 18 2018 13:35
hi @saeedhei. Do you want to measure the speeds of 1 million individual insert queries, or the speed of 1 query inserting 1 million docs? Also, why 1 million specifically? depending on your doc size, you're going to run out of heap very quickly with node ( though this is tunable ).
Sedric Heidarizarei
@saeedhei
Mar 18 2018 13:38
I want to insert 1.000.000 doc in db and test find() speed
I mean for example, If I had many users, my Project will work well?!
Kev
@lineus
Mar 18 2018 13:42
there's a function Model.bulkWrite() docs here that allows you to send an array of inserts. If you want to use Model instances instead there is Model.create() that takes an array of instances like MessageMsg in your above example and inserts them into the collection.
Sedric Heidarizarei
@saeedhei
Mar 18 2018 13:44
my Question is: How Should Make 1.000.000 elem of array?
Kev
@lineus
Mar 18 2018 13:48
you'll need to create an array with 1M objects ( though I would suggest adding them in smaller batches)
let docs = []
for (let i = 0; i < 1000000;i++){
  docs.push(new Message({..})
}
then call Messages.create(docs)
Sedric Heidarizarei
@saeedhei
Mar 18 2018 13:49
Thanks, I will do That.
Kev
@lineus
Mar 18 2018 13:53
@saeedhei you're welcome
here is an example I used to do something similar using insertMany
#!/usr/bin/env node
'use strict'

const uri = 'mongodb://localhost:27017/test'
const mongoose = require('mongoose')

const Schema = mongoose.Schema

const conn = mongoose.createConnection(uri)
const faker = require('faker')

const testSchema = new Schema({
  name: String,
  age: Number
})

const Test = conn.model('tests', testSchema)

const tests = []

for (let i = 0; i < 9; i++) {
  tests.push(new Test({
    name: faker.name.firstName(),
    age: Math.floor(Math.random() * 80)
  }))
}

conn.then(() => {
  Test.insertMany(tests, (err, res) => {
    if (err) { return console.error(err) }
    return console.log(res)
  })
}, (err) => {
  mongoose.connection.close()
  return console.error(err)
})
though I only added 9 documents to the collection :)
Sedric Heidarizarei
@saeedhei
Mar 18 2018 13:57
appreciate that
Kev
@lineus
Mar 18 2018 14:01
also, if this is stub data, it might be worth it to consider skipping mongoose altogether ( just for the insert of test data) and use the native driver to insert the documents. You don't really need the validation or other features that mongoose provides in this case. That will reduce the overhead as well.
just thinking out loud :)
Kev
@lineus
Mar 18 2018 15:22
@saeedhei it took my machine less than 30 seconds to insert a million docs to localhost with this code:
#!/usr/bin/env node
'use strict'

const MC = require('mongodb').MongoClient

const uri = 'mongodb://localhost:27017'
const dbName = 'native'
const colName = 'users'
const num = 1000000

const docs = []

for (let i = 0; i < num; i++) {
  docs.push({
    name: 'billy',
    nick: 'the kid'
  })
}

async function run () {
  const client = await MC.connect(uri)
  const db = client.db(dbName)
  await db.createCollection(colName)
  await db.collection(colName).insertMany(docs)
  return client.close()
}

run()
mongo shell:
bin: mongo --quiet native 
> db.users.count()
1000000
> db.users.findOne()
{
    "_id" : ObjectId("5aae83d0aa4bb67eecfced85"),
    "name" : "billy",
    "nick" : "the kid"
}
>
Syffs
@Syffs
Mar 18 2018 19:03
How is the summary collection presented here (https://www.mongodb.com/blog/post/retail-reference-architecture-part-1-building-flexible-searchable-low-latency-product) supposed to be generated/maintained with mongoose?
Sedric Heidarizarei
@saeedhei
Mar 18 2018 23:51
@lineus Awsome 👍