dancannon/gorethink

Please use #gorethink on the RethinkDB slack

Daniel Cannon

@dancannon

@andreis The system stats tables might be able to tell you the size of your tables http://rethinkdb.com/docs/system-stats/

Andrei Simionescu

@andreis

@dancannon maybe convert each document to a binary type and count bytes ? This is mainly for smaller documents, for larger ones i.e. files I already record their size.

William McGann

@tehbilly

The stats table in the rethinkdb database shows byte usage on a per database level. You could track each document size as a property on the document and do a sum query.

Andrei Simionescu

@andreis

Indeed, that's what I should do. Thank you :)

William McGann

@tehbilly

No problem!

Sumit. M. Asok

@sumitasok

Please help me form a Between Query to find documents between a date time

.Filter(func(row r.Term) r.Term {
        return row.Between(r.Time(2014, 8, 12, 'Z'), r.Time(2014, 8, 12, 'Z'), r.BetweenOpts{Index: "updated_at"})
    }).Run(session)

Daniel Cannon

@dancannon

@sumitasok For times you should use During instead http://www.rethinkdb.com/api/javascript/during/

This message was deleted

For example

Filter(func(row r.Term) r.Term {
    return row.Field("updated_at").During(r.Time(2014, 8, 12, 'Z'), r.Time(2014, 8, 12, 'Z'))
}).Run(session)

Aran Wilkinson

@aranw

Hey @dancannon is it more efficient to have multiple sessions for reading/writing data or is it just as efficient to have one Session open and use it for multiple queries?

Daniel Cannon

@dancannon

@aranw It should be more efficient to use a single session

Internally the session uses a connection pool which can be configured with the MaxIdle and MaxOpen ConnectOpts

Aran Wilkinson

@aranw

@dancannon ah ok :) thanks for the info I'll stick with the single session then I guess only way I can get better performance would be clustering rethink

Daniel Cannon

@dancannon

@aranw might help. What issues are you seeing at the moment?

Aran Wilkinson

@aranw

No issues, would just like better performance but not sure what my constraints are at moment

What's holding me back...

Daniel Cannon

@dancannon

Its hard to tell, it might be worth profiling your application http://blog.golang.org/profiling-go-programs.

Aran Wilkinson

@aranw

Oh this looks interesting

I think my latest version is a lot better in terms of performance

I've switched to using worker pools and having a single RethinkDB connection session

dalan

@dalanmiller

I'm currently querying RethinkDB and then pushing them through https://github.com/martini-contrib/render.

How can I make this faster? Do I need to fill up the slice using cursor.All? Or can I stream the cursor directly into the JSON creator in some way?

r.DB("pi_baker").Table("observations").OrderBy(r.Desc("datetime")).Limit(5760).Pluck("datetime", "temp", "humidity").Run(session)

if err != nil {
    log.Fatalln(err.Error())
}

var docRows []interface{}

cursor.All(&docRows)

renderer.JSON(200, map[string]interface{}{"obs": docRows})

Right now it's taking about 30 seconds for Martini / RethinkDB to process the request

William McGann

@tehbilly

Martini is pretty terrible in terms of performance (and not very idiomatic), but I doubt that's the source if your lengthy rendering time. How many results is that query returning, and about how large are the documents?

dalan

@dalanmiller

Well, just by looking at the RethinkDB admin UI I can see that it's doing a table scan

cursor, err := r.DB("pi_baker").Table("observations").OrderBy(
            r.OrderByOpts{
                Index: r.Desc("datetime"),
            }).Limit(5760).Pluck("datetime", "temp", "humidity").ToJSON().Run(session)

This is giving me an error however "2015/07/24 16:52:57 gorethink: Expected type DATUM but found SEQUENCE: VALUE SEQUENCE in:"

dalan

@dalanmiller

Ah ToJSON() is causing the problem

I'm getting sub 2s timings now

dalan

@dalanmiller

Doesn't look like there's an example in the gorethink docs for ToJSON()

I'm not quite sure where to put it

William McGann

@tehbilly

Instead of using .All() I'd probably loop na cursor of results, append each one to a slice of pointers to a struct/object that matches the output you want to display, then render that.

I'm on my phone so can't jot down a good example, but I will be later on and I'd be happy to write something up really quickly.

dalan

@dalanmiller

That would be super helpful. How does one append to a slice?

William McGann

@tehbilly

Looking for the best example isn't going well. I'm getting my daughter clean and to sleep, then I'll hop online. Apologies for the delay

Daniel Cannon

@dancannon

Hey dalanmiller just got in, are you still around?

@dalanmiller ^^

dalan

@dalanmiller

Still around :D, howdy Mr. @dancannon

I'm pretty new to Go, so I'm a bit perplexed on a few levels.

Daniel Cannon

@dancannon

Good to hear :) ill try to help out where I can

So reading back it seems like you have fixed your performance issues

Are you still looking to append to a slice?

dalan

@dalanmiller

Nah, I ended up just using cursor.All(&rows) but I was trying to use .ToJSON() to see if that could improve performance by just getting a JSON string straight from RethinkDB but was coming across issues.

For instance, in my last code snippet up there, why is that an inappropriate place for .ToJSON()?

Daniel Cannon

@dancannon

Were there performance issues when you were using All? I am a little unsure why that is returning an error. Perhaps it has to be coerced to an array first?

That error comes from the database directly so unless the query is being built incorrectly it is not a gorethink specific issue (I think)

dalan

@dalanmiller

No the .All always seemed to work, was just working with 1000s of documents

I think the query is just built incorrectly

Daniel Cannon

@dancannon

Hmm ok, regarding All I think @tehbilly was correct about using Next instead as this means you dont have to load everything into memory and then marshal to JSON.

William McGann

@tehbilly

My thought was to use a cursor, and a json encoder to encode each result into the response writer and flush. I still have a toddler in my lap, though. Heh

Daniel Cannon

@dancannon

Ok I think adding . CoerceTo("array") before the ToJSON() might help

Where communities thrive

People

Repo info

Activity