Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 04 2022 05:16
    klokare edited #503
  • Jun 29 2022 17:40
    klokare commented #503
  • Jun 29 2022 16:46
    klokare commented #503
  • Jun 29 2022 16:13
    klokare opened #503
  • Jun 12 2022 03:56
    GaikwadPratik edited #502
  • Jun 12 2022 03:55
    GaikwadPratik opened #502
  • Jun 02 2022 15:05

    CMogilko on v6.2.2

    (compare)

  • Jun 02 2022 15:03

    CMogilko on master

    upgrade version golang.org/x/sy… (compare)

  • Jun 02 2022 15:03
    CMogilko closed #501
  • Jun 01 2022 06:29
    MehrunesSky opened #501
  • Apr 07 2022 15:52
    GaikwadPratik closed #493
  • Apr 07 2022 15:14
    Gerrit91 edited #500
  • Apr 07 2022 15:09
    Gerrit91 edited #500
  • Apr 07 2022 15:05
    Gerrit91 edited #500
  • Apr 07 2022 15:02
    Gerrit91 edited #500
  • Apr 07 2022 14:58
    Gerrit91 edited #500
  • Apr 07 2022 14:52
    Gerrit91 edited #500
  • Apr 07 2022 14:42
    Gerrit91 commented #493
  • Apr 07 2022 13:45
    Gerrit91 edited #500
  • Apr 07 2022 13:44
    Gerrit91 edited #500
Daniel Cannon
@dancannon
@andreis sorry for taking so long to reply, I cant think of any good way about doing this with GoRethink. GoRethink doesnt know anything about the way RethinkDB stores data
@andreis The system stats tables might be able to tell you the size of your tables http://rethinkdb.com/docs/system-stats/
Andrei Simionescu
@andreis
@dancannon maybe convert each document to a binary type and count bytes ? This is mainly for smaller documents, for larger ones i.e. files I already record their size.
William McGann
@tehbilly
The stats table in the rethinkdb database shows byte usage on a per database level. You could track each document size as a property on the document and do a sum query.
Andrei Simionescu
@andreis
Indeed, that's what I should do. Thank you :)
William McGann
@tehbilly
No problem!
Sumit. M. Asok
@sumitasok
Please help me form a Between Query to find documents between a date time
.Filter(func(row r.Term) r.Term {
        return row.Between(r.Time(2014, 8, 12, 'Z'), r.Time(2014, 8, 12, 'Z'), r.BetweenOpts{Index: "updated_at"})
    }).Run(session)
Daniel Cannon
@dancannon
@sumitasok For times you should use During instead http://www.rethinkdb.com/api/javascript/during/
This message was deleted
For example
Filter(func(row r.Term) r.Term {
    return row.Field("updated_at").During(r.Time(2014, 8, 12, 'Z'), r.Time(2014, 8, 12, 'Z'))
}).Run(session)
Aran Wilkinson
@aranw
Hey @dancannon is it more efficient to have multiple sessions for reading/writing data or is it just as efficient to have one Session open and use it for multiple queries?
Daniel Cannon
@dancannon
@aranw It should be more efficient to use a single session
Internally the session uses a connection pool which can be configured with the MaxIdle and MaxOpen ConnectOpts
Aran Wilkinson
@aranw
@dancannon ah ok :) thanks for the info I'll stick with the single session then I guess only way I can get better performance would be clustering rethink
Daniel Cannon
@dancannon
@aranw might help. What issues are you seeing at the moment?
Aran Wilkinson
@aranw
No issues, would just like better performance but not sure what my constraints are at moment
What's holding me back...
Daniel Cannon
@dancannon
Its hard to tell, it might be worth profiling your application http://blog.golang.org/profiling-go-programs.
Aran Wilkinson
@aranw
Oh this looks interesting
I think my latest version is a lot better in terms of performance
I've switched to using worker pools and having a single RethinkDB connection session
dalan
@dalanmiller

I'm currently querying RethinkDB and then pushing them through https://github.com/martini-contrib/render.

How can I make this faster? Do I need to fill up the slice using cursor.All? Or can I stream the cursor directly into the JSON creator in some way?

r.DB("pi_baker").Table("observations").OrderBy(r.Desc("datetime")).Limit(5760).Pluck("datetime", "temp", "humidity").Run(session)

if err != nil {
    log.Fatalln(err.Error())
}

var docRows []interface{}

cursor.All(&docRows)

renderer.JSON(200, map[string]interface{}{"obs": docRows})
Right now it's taking about 30 seconds for Martini / RethinkDB to process the request
William McGann
@tehbilly
Martini is pretty terrible in terms of performance (and not very idiomatic), but I doubt that's the source if your lengthy rendering time. How many results is that query returning, and about how large are the documents?
dalan
@dalanmiller
Well, just by looking at the RethinkDB admin UI I can see that it's doing a table scan
cursor, err := r.DB("pi_baker").Table("observations").OrderBy(
            r.OrderByOpts{
                Index: r.Desc("datetime"),
            }).Limit(5760).Pluck("datetime", "temp", "humidity").ToJSON().Run(session)
This is giving me an error however "2015/07/24 16:52:57 gorethink: Expected type DATUM but found SEQUENCE: VALUE SEQUENCE in:"
dalan
@dalanmiller
Ah ToJSON() is causing the problem
I'm getting sub 2s timings now
dalan
@dalanmiller
Doesn't look like there's an example in the gorethink docs for ToJSON()
I'm not quite sure where to put it
William McGann
@tehbilly
Instead of using .All() I'd probably loop na cursor of results, append each one to a slice of pointers to a struct/object that matches the output you want to display, then render that.
I'm on my phone so can't jot down a good example, but I will be later on and I'd be happy to write something up really quickly.
dalan
@dalanmiller
That would be super helpful. How does one append to a slice?
William McGann
@tehbilly
Looking for the best example isn't going well. I'm getting my daughter clean and to sleep, then I'll hop online. Apologies for the delay
Daniel Cannon
@dancannon
Hey dalanmiller just got in, are you still around?
@dalanmiller ^^
dalan
@dalanmiller
Still around :D, howdy Mr. @dancannon
I'm pretty new to Go, so I'm a bit perplexed on a few levels.
Daniel Cannon
@dancannon
Good to hear :) ill try to help out where I can
So reading back it seems like you have fixed your performance issues
Are you still looking to append to a slice?
dalan
@dalanmiller
Nah, I ended up just using cursor.All(&rows) but I was trying to use .ToJSON() to see if that could improve performance by just getting a JSON string straight from RethinkDB but was coming across issues.
For instance, in my last code snippet up there, why is that an inappropriate place for .ToJSON()?
Daniel Cannon
@dancannon
Were there performance issues when you were using All? I am a little unsure why that is returning an error. Perhaps it has to be coerced to an array first?
That error comes from the database directly so unless the query is being built incorrectly it is not a gorethink specific issue (I think)
dalan
@dalanmiller
No the .All always seemed to work, was just working with 1000s of documents
I think the query is just built incorrectly
Daniel Cannon
@dancannon
Hmm ok, regarding All I think @tehbilly was correct about using Next instead as this means you dont have to load everything into memory and then marshal to JSON.
William McGann
@tehbilly
My thought was to use a cursor, and a json encoder to encode each result into the response writer and flush. I still have a toddler in my lap, though. Heh