Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Jeff Martin
    @jjmartin

    with regards to

    maybe someone can answer this later - i'm using the spex.stream.read and when my reciever function returns a promise, the next reciever is called with pretty much all the rest of the data in the stream rather than a single record... is there a way to force it to 1 record per call, and still return a promise?

    Jeff Martin
    @jjmartin
    i'm using spex and pg-promise stream with the pg-query-stream... i'm wanting to write each record to a aws dynamoDb table - when i have the receiver return the promise, the first call returns 1 record, and the next returns all the rest. if i do something synchronous like add the data for the dynamo call to an array - i get one record at a time, but i'm trying to limit how much memory the app is using during the stream, so collecting them all in an array (and having the data param collect the whole contents of the query) isn't limiting how much memory it uses
    Vitaly Tomilov
    @vitaly-t

    You didn't follow on my comment from the other thread:

    There is no such thing as a record within the stream terminology. There is only data, in chunks. Stream providers may implement data groups according to their individual logic.
    Have you tried using option readSize? The result depends on your stream reader.

    Jeff Martin
    @jjmartin
    i tried readSize and it didn't seem to have an effect
    spex.stream.read(stream, writeDetailCollectSummary, false, 1)
    i also tried setting the highWaterMark on the pg-query-stream and also that seemed to have no effect
    Vitaly Tomilov
    @vitaly-t
    that means your stream provider sends all the data back just as it becomes available.
    Jeff Martin
    @jjmartin
    is the pg-query-stream the provider or the DB?
    Vitaly Tomilov
    @vitaly-t
    neither, it is the stream library that you are using.
    both pg-query-stream and SPEX are stream consumers. Stream provider is the library you are using to read the data.
    Jeff Martin
    @jjmartin
    oh well thats the pg-promise library
    Vitaly Tomilov
    @vitaly-t
    No :)
    pg-promise got nothing to do with streams :)
    Jeff Martin
    @jjmartin
    then its jsut whatever those use by default- i'm not specifying anything else
    Vitaly Tomilov
    @vitaly-t
    what library are you using to stream the data? What is the source?
    Jeff Martin
    @jjmartin
    redshift...
    Vitaly Tomilov
    @vitaly-t
    redshift is a hosting provider, it is not a library. what library?
    which require are you using to include your streaming library?
    Jeff Martin
    @jjmartin
    "dependencies": { "async": "^1.5.0", "aws-sdk": "^2.2.15", "bluebird": "^3.0.0", "cd-amqp-exchange": "^1.2.0", "lodash": "^4.6.1", "log4js": "^0.6.33", "pg-promise": "^4.0.4", "pg-query-stream": "^1.0.0", "promise-retry": "^1.1.0", "spex": "^0.4.2" },
    Vitaly Tomilov
    @vitaly-t
    got you. in this case pg-query-stream is the provider.
    and evidently, it delivers all the data it got at the moment
    Jeff Martin
    @jjmartin
    yea.. but only when my receiver returns a promise
    Vitaly Tomilov
    @vitaly-t
    of course, the asyncronous window gives it a chance to consume more data
    Jeff Martin
    @jjmartin
    right - maybe i need to go to whoever owns that to ask more about setting limits in pg-query-stream (was pretty sure that was where i needed to set it)
    Vitaly Tomilov
    @vitaly-t
    but it still should be limited by the maximum stream buffer
    quite possibly, I'm not sure - maybe it does support a limit...
    Jeff Martin
    @jjmartin
    i'm not seeing in any docs where exactly i set that - - tried highWaterMark - but that seemed to have no effect
    Vitaly Tomilov
    @vitaly-t
    I've just looked at it...
    var QueryStream = module.exports = function(text, values, options) {
      var self = this
      this._reading = false
      this._closing = false
      options = options || { }
      Cursor.call(this, text, values)
      Readable.call(this, {
        objectMode: true,
        highWaterMark: options.highWaterMark || 1000
      })
      this.batchSize = options.batchSize || 100
      this.once('end', function() {
        process.nextTick(function() {
          self.emit('close')
        })
      })
    }
    Jeff Martin
    @jjmartin
    i used that pg-query-stream because thats what the pg-promise docs reccomended
    Vitaly Tomilov
    @vitaly-t
    Try playing with highWaterMark and batchSize options ;)
    Jeff Martin
    @jjmartin
    that looks like batchSize - hmm - maybe i have an older version - dont' remember seeing that - ok well i'll check that bit out - thanks
    Vitaly Tomilov
    @vitaly-t
    n.p.
    Jeff Martin
    @jjmartin
    btw - i really like everything you have been doing with pg-promise - its been our go-to for redshift stuff since ive been working with AWS...
    Vitaly Tomilov
    @vitaly-t
    :+1:
    Jeff Martin
    @jjmartin
    @vitaly-t well i got things back to where i could test directly with setting the batchSize and the stream still fills up when i have a promise in the spex receiver i see where it is calling 1 row on the cursor but its not seeming to limit how many times it will push into its buffer...
    i had started working on a different solution to limit how much time/memory i was using (trying to get this all running in an AWS lambda) and i think i'm just going to go back to that
    Jeff Martin
    @jjmartin
    i guess i see what the issue is - the _read in the QueryStream is called with the highwaterMark number which i think is supposed to limit how much should be in the buffer of the stream... but that is never checked in the QueryStream code..
    Vitaly Tomilov
    @vitaly-t
    @jjmartin This doesn't surprise me at all. The level of support that BrianC has been providing for this leaves a lot to hope for.
    Jeff Martin
    @jjmartin
    heh ok
    well i was going to leave an issue there anyway - i don't know if i will get to a pull request on it - but it will be there - i saw yours there too... if you end up re-implementing it for pg-promise or whatever - would love a note
    Vitaly Tomilov
    @vitaly-t
    the whole stream? - not likely :)
    the library is already too large as it is :)
    Jeff Martin
    @jjmartin
    =)
    Vitaly Tomilov
    @vitaly-t
    I have just completely finished the new helpers addition: https://github.com/vitaly-t/pg-promise/tree/master/lib/helpers
    And I am hoping that this is the last one :)
    Jeff Martin
    @jjmartin
    btw i found your examples/best practices project just through clicking in github - i liked it - might be nice to link it in your wiki/learn by example..
    Vitaly Tomilov
    @vitaly-t
    i might do that, cheers!
    Vitaly Tomilov
    @vitaly-t
    @jjmartin it's always been listed on the project's WiKi main page: https://github.com/vitaly-t/pg-promise/wiki
    Vitaly Tomilov
    @vitaly-t
    I have started making some big changes in the library's support for TypeScript 2.0