Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jul 04 05:16
    klokare edited #503
  • Jun 29 17:40
    klokare commented #503
  • Jun 29 16:46
    klokare commented #503
  • Jun 29 16:13
    klokare opened #503
  • Jun 12 03:56
    GaikwadPratik edited #502
  • Jun 12 03:55
    GaikwadPratik opened #502
  • Jun 02 15:05

    CMogilko on v6.2.2

    (compare)

  • Jun 02 15:03

    CMogilko on master

    upgrade version golang.org/x/sy… (compare)

  • Jun 02 15:03
    CMogilko closed #501
  • Jun 01 06:29
    MehrunesSky opened #501
  • Apr 07 15:52
    GaikwadPratik closed #493
  • Apr 07 15:14
    Gerrit91 edited #500
  • Apr 07 15:09
    Gerrit91 edited #500
  • Apr 07 15:05
    Gerrit91 edited #500
  • Apr 07 15:02
    Gerrit91 edited #500
  • Apr 07 14:58
    Gerrit91 edited #500
  • Apr 07 14:52
    Gerrit91 edited #500
  • Apr 07 14:42
    Gerrit91 commented #493
  • Apr 07 13:45
    Gerrit91 edited #500
  • Apr 07 13:44
    Gerrit91 edited #500
William McGann
@tehbilly
Instead of using .All() I'd probably loop na cursor of results, append each one to a slice of pointers to a struct/object that matches the output you want to display, then render that.
I'm on my phone so can't jot down a good example, but I will be later on and I'd be happy to write something up really quickly.
dalan
@dalanmiller
That would be super helpful. How does one append to a slice?
William McGann
@tehbilly
Looking for the best example isn't going well. I'm getting my daughter clean and to sleep, then I'll hop online. Apologies for the delay
Daniel Cannon
@dancannon
Hey dalanmiller just got in, are you still around?
@dalanmiller ^^
dalan
@dalanmiller
Still around :D, howdy Mr. @dancannon
I'm pretty new to Go, so I'm a bit perplexed on a few levels.
Daniel Cannon
@dancannon
Good to hear :) ill try to help out where I can
So reading back it seems like you have fixed your performance issues
Are you still looking to append to a slice?
dalan
@dalanmiller
Nah, I ended up just using cursor.All(&rows) but I was trying to use .ToJSON() to see if that could improve performance by just getting a JSON string straight from RethinkDB but was coming across issues.
For instance, in my last code snippet up there, why is that an inappropriate place for .ToJSON()?
Daniel Cannon
@dancannon
Were there performance issues when you were using All? I am a little unsure why that is returning an error. Perhaps it has to be coerced to an array first?
That error comes from the database directly so unless the query is being built incorrectly it is not a gorethink specific issue (I think)
dalan
@dalanmiller
No the .All always seemed to work, was just working with 1000s of documents
I think the query is just built incorrectly
Daniel Cannon
@dancannon
Hmm ok, regarding All I think @tehbilly was correct about using Next instead as this means you dont have to load everything into memory and then marshal to JSON.
William McGann
@tehbilly
My thought was to use a cursor, and a json encoder to encode each result into the response writer and flush. I still have a toddler in my lap, though. Heh
Daniel Cannon
@dancannon
Ok I think adding . CoerceTo("array") before the ToJSON() might help
Yeah I think that would make sense, it might take a little bit of code to work but it should be a bit more efficient
I just got home and its 2:30am haha...
Early morning debugging...
William McGann
@tehbilly
The best kind of debugging!
dalan
@dalanmiller
Ahh okay
Lemme try that
William McGann
@tehbilly
I just got my first snore, so I'm thinking I'll be at the desktop in ten to fifteen
dalan
@dalanmiller

So now I have this query:

cursor, err := r.DB("pi_baker").Table("observations").OrderBy(
            r.OrderByOpts{
                Index: r.Desc("datetime"),
            }).Limit(100).Pluck("datetime", "temp").CoerceTo("array").ToJSON().Run(session)

So now cursor is still a Cursor but I assume the result is a byte[]?

Daniel Cannon
@dancannon
So its actually string, so instead of using All you should instead use One
Daniel Cannon
@dancannon
@dalanmiller Im going to head off, if you have any more questions post them here and ill get back to you tomorrow
dalan
@dalanmiller
Thanks so much @dancannon, really appreciate it (also @tehbilly!). I think that about did it. I'm just wrestling with Martini now on changing a silly content-type
Daniel Cannon
@dancannon
@dalanmiller I would recommend looking into using https://github.com/codegangsta/negroni instead (which is made by the same author)
Its a bit more "idiomatic"
dalan
@dalanmiller
Yeah I think I'm going to jump ship
I think abstracting away net/http isn't a good practice
Daniel Cannon
@dancannon
Well yeah for the most part I just stick with net/http
dalan
@dalanmiller
Haha, I saw that argument too ;)
Daniel Cannon
@dancannon
maybe with a custom mux/router package if I want to do something more advanced
dalan
@dalanmiller
I think I need some training wheels for now.
Daniel Cannon
@dancannon
Yeah that makes complete sense
William McGann
@tehbilly
You've got something very fun to use as training wheels! Ironically, finally at my desktop, ha.
For the sake of completeness, would you like me to put together an example of what I was talking about? A good way to stream large numbers of responses without taking forever.
William McGann
@tehbilly
func fastHandler(w http.ResponseWriter, req *http.Request) {
    res, err := r.DB("pi_baker").Table("observations").
        OrderBy(r.Desc("datetime")).
        Limit(5760).
        Pluck("datetime", "temp", "humidity").
        Run(session)

    if err != nil {
        log.Fatalln(err.Error())
    }

    w.Header().Set("Content-Type", "application/json")
    encoder := json.NewEncoder(w)
    var row map[string]interface{}

    for res.Next(&row) {
        err = encoder.Encode(row)
        if err != nil {
            // Oh no! Something bad happened!
            // Since we're not buffering the response, we aren't going to send an
            // error code and message. Maybe you can log the error and continue.
        }
    }
}
Untested in terms of performance, but to the best of my knowledge that would stream the results as individual json objects, newline delimited. So you could hack in array formatting to make it a valid json object ([,commas between each item, and ]) or make your handler consume it line-by-line.
The call to .Encode() will write each object to the http connection, not storing it in memory. Probably the best way to handle actually massive amounts of data.
William McGann
@tehbilly
The simpler way to handle it, with some memory usage and whatnot, but should show what more standard go patterns are like and would send a valid json object:
var results []map[string]interface{} = make([]map[string]interface{}, 0)
    var row map[string]interface{}

    for res.Next(&row) {
        results = append(results, row)
    }

    jsonBytes, err := json.Marshal(results)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        // Logging goes here
        return
    }

    w.Header().Set("Content-Type", "application/json")
    w.Write(jsonBytes)
Tim Shannon
@timshannon
Just curious if the dependency on the hailocab protobuf fork is intentional, or accidental. https://github.com/dancannon/gorethink/blob/master/ql2/ql2.pb.go
Daniel Cannon
@dancannon
@timshannon Ah that's accidental. My protoc I'd built against that fork. I'll push an update.
Tim Shannon
@timshannon
Thanks, I just ran into it in my local repositories, and I don't have a copy of that protobuf fork. Just wanted to check to see if it was needed or not. I now see that hailocab has some interesting forks to explore :)