Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
Tomasz Pluskiewicz
@tpluscode
for example, I looked at rdfxml-streaming-parser.js and it does not implement it although still works when passed to Sink#import().
Ruben Taelman
@rubensworks
I think is meant to enable applications that don't want to use in flow-mode, but want to pull in elements lazily, one element at a time. But I wasn't around when the spec was defined. @bergos and/or @RubenVerborgh may be able to give a better asnwer.
Oh, rdfxml-streaming-parser.js doesn't implement it? It's possible that many parsers don't implement it then. To be looked into...
Tomasz Pluskiewicz
@tpluscode
unless something is just wrong with the current state of typings
read appears to be a base method from Readable which your parser does extend
Ruben Taelman
@rubensworks
Indeed, it should correspond to this: https://nodejs.org/api/stream.html#stream_readable_read_size
Tomasz Pluskiewicz
@tpluscode
yes, sorry, I was not reading the error correctly. Pull Request on its way
bugger. I knew editing in GH's web interface was a bad idea
Blake Regalia
@blake-regalia
We need to revisit the streams interface and probably align with whatwg
Ruben Verborgh
@RubenVerborgh
yes the definition of read was to allow reading in a pull-based manner, such that consumers only read as much as they need to or can handle
@blake-regalia Agreed, if performance is not impacted more than a couple of %
Thomas Bergwinkl
@bergos
i don't think we should switch to whatwg streams. only fetch in the browser is using it. even node-fetch (12m weekly downloads) uses node streams.
i guess more people would be happy about async iterator support which is easy to implement by using the latest version of readable-stream.
if there are still some people which would like to use whatwg streams, implementing wrappers for both directions should not be that difficult. 1/2: https://github.com/bergos/nodeify-fetch/blob/master/lib/WhatwgReadable.js just needs a separate package.
about the spec: if we change something, it should be only adding async iterator to the stream interface.
Blake Regalia
@blake-regalia
interesting thought -- I caught up a bit on matteo colina's (father of node.js streams), and he had some things to say about the divergence here: https://changelog.com/jsparty/103#transcript-45
Blake Regalia
@blake-regalia
i do feel that rdfjs should be more web-oriented than node.js-oriented, but i also agree that the change is painful and all that. also, fetch is not the only thing using whatwg streams in the browser, there is also File
Thomas Bergwinkl
@bergos
"web-orientated" is from my point of view the wrong term. you can say "whatwg-orientated", but i would guess more people are actually using node.js streams in the browser than whatwg streams.
i interpret matteo colinas comments to support my suggestion: "we should in fact not aim to be compatible between each other directly, but instead use a common lingua franca, which would tend to be, in my opinion, async iterators"
so async iterators is what people want in the end.
Thomas Bergwinkl
@bergos
"if somebody wants to be involved in helping out and delivering some form of WHATWG streams compatibility, I will be very happy to help them develop this for Node.js"
i guess he means adding whatwg stream support to the node.js streams, but having bridge would be already a start.
and for me most important: "this also means that Node became very big, and everybody started using those"
Thomas Bergwinkl
@bergos
using whatwg streams would require a lot of work outside the rdfjs scope. there are so many packages for node.js streams and i'm not aware of any for whatwg streams.
Martynas Jusevicius
@namedgraph_twitter
what does this mean for the browser support?
Blake Regalia
@blake-regalia
It means rdfjs libs will not be spec'd to work with fetch/FileReader pipe methods in the browser, out-of-the-box. Although, you can still adapt them without third-party tools as I describe here: https://github.com/blake-regalia/graphy.js/issues/20#issuecomment-575740629
Martynas Jusevicius
@namedgraph_twitter
well that sucks :)
to me personally it's the browser that matters, not Node.js
Blake Regalia
@blake-regalia
Actually, what I linked to is not supported by the spec either since write and end are not defined by the interface. It would have to use .import on the writable steam but I'm not sure if thats compatible with whatwg readables
Blake Regalia
@blake-regalia
In that case, you would need to bring-your-own whatwg-to-nodejs stream adapter anytime you want to use an rdfjs lib in the browser, given that you don't rely on features provided by the lib itself
Martynas Jusevicius
@namedgraph_twitter
once again i get convinced of the state of the JS ecosystem: trash
discourages me to work on it again and again and again
namedgraph_twitter @namedgraph_twitter ducks
Thomas Bergwinkl
@bergos
@blake-regalia i think your comment leaves a wrong impression of the situation. add const fetch = require('nodeify-fetch') on top of your code. done. now your code works in node.js and in the browser with any rdfjs compatible parser.
or directly use https://www.npmjs.com/package/@rdfjs/fetch and you don't have to take care of media type + parser handling anymore.
that's the level most people will start. many may even start at more high level libs.

It means rdfjs libs will not be spec'd to work with fetch/FileReader pipe methods in the browser, out-of-the-box

my libs are working out of the box

Blake Regalia
@blake-regalia
@bergos -- require('nodeify-fetch') is a CommonJS way of loading a third-party tool, meaning that the user must transpile or bundle before deploying to the web -- both of which mean that some rdfjs lib does not work out-of-the-box
my comment does not leave the wrong impression -- it is objective
Thomas Bergwinkl
@bergos
does any rdfjs lib work without bundling? almost any web application gets transpiled and bundled. so let's skip that part. yes, some may need to install third party tools, but we should always put it in context. streams is the stuff most libs will hide from the developers (matteo also mentioned that). so experts will takes care of that problem (which is just a one liner). about the comment, i was refering to the github comment which gives the impression that it's a complicated and almost not solvable problem for average users: https://github.com/blake-regalia/graphy.js/issues/20#issuecomment-575740629
Tomasz Pluskiewicz
@tpluscode
well, it seems to be a good practice to build code which runs in the browser without bundling. hence all of the web components specifications.
you might still bundle for optimizations but build-less is at least better for development cycle
unfortunately the bigger problem is that all of rdfjs code is indeed written with commonjs. first we'd need to rewrite as modules and only then start thinking about streams
Blake Regalia
@blake-regalia

@bergos

does any rdfjs lib work without bundling?

good point, I imagine that since Stream extends EventEmitter, they must all be bundled anyway...

I totally understand your points about not wanting to switch to whatwg (and I agree with them) -- but i still think it's really important to have this discussion
Blake Regalia
@blake-regalia
as for the GH comment, I was merely stating an approach to work with node.js streams in the browser without any 3rd party libs
Thomas Bergwinkl
@bergos
totally agree on having that discussion and happy to hear opinions, but this is the public rdfjs channel and we should be careful that people that are not so deep into that topic get a wrong impression of using rdfjs in the browser requires dirty hacks.
that was my main concern
Thomas Bergwinkl
@bergos
i think it's not mentioned anywhere, but in the past we had a look from three differention positions: low level rdfjs libs developers, high level rdfjs libs developers, users of rdfjs libs. in my opinion the last group should not care about streams. we have the option to implement high level libs (e.g. rdf-fetch) or we give them a simpler interface (e.g. async iterators). it's easy to provide both options with node.js streams or whatwg streams. that's why we should focus more on the other two groups. they (should) know how to handle streams and should be aware that there are two standards and how to bridge them. so what's easier to use for that group? i use streams a lot (e.g. https://github.com/zazuko/barnard59 etl pipelines described in rdf with dynamic code loading completely based on streams), switching to whatwg would mean to start from scratch. not only my code but so many libraries i rely on. i made a quick search and could not even find a base class to start implementing whatwg streams. in other cases i would agree to align to more offical specs, but in this case it's even hard to say if we would be ahead or totally of track.