2 replies
For example, if I had an asynchronous file I/O surrogate, could I implement a synchronous
read(Array[Byte], Int, Int) method?
in dotty you could give https://github.com/rssh/dotty-cps-async a shot, that might work, too.
It was a fun refactoring, I think I also understand it better now too!! :)
Right now we hide most of this from the user BUT we are also getting customers that anticipate much larger data sets.
Anyone have any advice for how to build out our data model so it performs in the browser ?
We are debating using js.Object and native all the data so there is no deserialization. Then have all the Operations on those objects be via impliciits...
the generation of the serialized json on the server is very strict so we aren't worried about getting bad json or having to report on it.
anywho curious to hear people's thoughts
going with native js.Objects would be the next step, after that some kind of on-demand downloading/parsing, and after that maybe some kind of streaming/pagination, maybe event-based parsing (SAX-style) to perform computations without needing a data structure at all (uPickle supports this via Visitors).
The ideal architecture would depend on exactly how much larger "much larger" is, and can't really be answered in the abstract
@fizzy33 Try jsoniter-scala. It can parse white space separated rows or JSON array rows from input streams without holding a whole input in the memory. Also it is one of most efficient JSON parsers for Scala, please see results of benchmarks here or test immediately in your browser using this page.
the dependency copy-to-clipboard links on https://www.scala-js.org/libraries/facades.html are partly outdated; e.g. scalajs-dom copies 0.9.5 while latest release is 1.1.0
and in return when I have a
Blob, can I just asInstanceOf[Meta] its contents?
i.e. I would like a translation to
{ blockSize: 12, lastModified: 356 }
I'm trying 'Literal object construction using an Scala object interface' from here, but it seems wrong:
import scalajs.js
import org.scalajs.dom
trait Meta extends js.Object {
val blockSize: Int = js.native
val lastModified: Long = js.native
}
object Meta {
def apply(blockSize: Int, lastModified: Long): Meta =
js.Dynamic.literal(blockSize = blockSize, lastModified = lastModified).asInstanceOf[Meta]
}Compiler says 'Members of non-native JS traits must either be abstract, or their right-hand-side must be js.undefined'
Compiles if I add
@js.native. Ok?
It's complaining about the
Long now: 'method fromLong in object Any is deprecated (since forever): A Long is converted to Double to be cast to js.Any. This is almost certainly not what you want. Use .toDouble explicitly if you need it.' But I do want a Long
Or as a BigInt.
I have no idea what that means :) But I think for my meta data I should not use
Blob but ArrayBuffer, which is simpler for me to get back (I understand that Blob#arrayBuffer may not work in Safari). In general, I prefer binary formats. Just moving ahead with trial-and-error. "All you never wanted to know about JavaScript"
thanks everyone @ritschwumm @plokhotnyuk @aappddeevv
Unfortunately the way the data set is traversed all the data is needed. The data set could be cut down by 50% but that won't get us/me close enough on the performance envelope.
I have tried json-iter and it is better and could.does solve my immediate need but not the growing data set need. This is the library I PoC'ed with to test how much performance I could get with traditional intermediate AST parsing.
Our model is defined by a domain language that we generate the database tables AND the scala case classes from. We may just go the wrote and implement a version of the model that works in browser off of raw json objects. So we would
JSON.parse(jsonStr).asInstanceOf[JsNativeModel]
and just take care
we can validate the model for dev and debug purposes but in the production use case we don't
If you can't reduce the initial data load (virtualization, using local storage as a cache, etc.) and it all has to be there from the first moment, I always found that with even with a few thousand points, to your point, I skipped the creation of any scala data structures and used native traits and ensure that the path from data ingest in the browser to the "display/use" component had fewest transformations possible. There's always some deserialization going on and it sounds like no matter which way you go, that's the way it is. That's why I also built scalajs-reaction that way...very thin layer over js react components in order to avoid conversions.
I think the only other choice is if you can gradually add the data as it streams in, then you have a shot at having something displayed before it all arrives. And have an indicator to show as its loading. Sort of like a fuzzy picture being sent before the hi-res arrives. Good luck though.