Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jun 08 20:43
    @sjrd banned @sw7240614
  • Sep 29 2019 08:35
    @sjrd banned @8bitpitRBX_twitter
  • May 23 2019 13:44
    @sjrd banned @BIGCRYPTO001_gitlab
  • May 01 2019 14:31
    @sjrd banned @Certificates1_gitlab
  • Apr 22 2019 15:31
    @sjrd banned @lerbatax
  • Jan 31 2019 23:04
    sjrd commented #3555
  • Jan 31 2019 22:56
    sjrd review_requested #3555
  • Jan 31 2019 22:56
    sjrd opened #3555
  • Jan 31 2019 21:06

    sjrd on 0.6.x

    Remove dead code: specific coll… Adapt the signature of `js.Arra… Merge pull request #3554 from s… (compare)

  • Jan 31 2019 21:06
    sjrd closed #3554
  • Jan 31 2019 20:54
    erikvanoosten commented #2665
  • Jan 31 2019 16:33
    sjrd commented #3538
  • Jan 31 2019 15:54

    sjrd on master

    Fix analyzer cycle detection to… Add toString methods to analyze… Do not provide linked ClassInfo… and 1 more (compare)

  • Jan 31 2019 15:54
    sjrd closed #3552
  • Jan 31 2019 15:54

    sjrd on master

    Remove Logger.success It is un… Make level helpers final Clean-up ScalaConsoleLogger cod… and 1 more (compare)

  • Jan 31 2019 15:54
    sjrd closed #3553
  • Jan 31 2019 15:33
    sjrd commented #2665
  • Jan 31 2019 15:30
    erikvanoosten commented #2665
  • Jan 31 2019 10:41
    sjrd review_requested #3554
  • Jan 31 2019 10:41
    sjrd opened #3554
Eric K Richardson
@ekrich
@sjrd I changed the test names as we discussed. I think it is pretty good. scala-js/scala-js#4266
Glen Marchesani
@fizzy33
So I am looking for advice. I/we have a scala js app in the wild. The model for some of our customers is 100K rows (and growing). That data is all in the browser right now. 100K rows going through any kind of json deserialization takes a long long time.
Right now we hide most of this from the user BUT we are also getting customers that anticipate much larger data sets.
Anyone have any advice for how to build out our data model so it performs in the browser ?
We are debating using js.Object and native all the data so there is no deserialization. Then have all the Operations on those objects be via impliciits...
the generation of the serialized json on the server is very strict so we aren't worried about getting bad json or having to report on it.
anywho curious to hear people's thoughts
Li Haoyi
@lihaoyi
which JSON library are you using? e.g. uPickle is a lot faster than Circe or Play-JSON in the browser, even though on the JVM the difference is not nearly as great
going with native js.Objects would be the next step, after that some kind of on-demand downloading/parsing, and after that maybe some kind of streaming/pagination, maybe event-based parsing (SAX-style) to perform computations without needing a data structure at all (uPickle supports this via Visitors).
The ideal architecture would depend on exactly how much larger "much larger" is, and can't really be answered in the abstract
Domas Poliakas
@domaspoliakas
I’m a little confused - if you’re sending data from the server then surely you can’t have no deserialisation?
ritschwumm
@ritschwumm
@fizzy33 do you need the complete set of data in the browser? because "much larger than 100k rows" will get you in trouble where shaving off a constant factor will not help any more, probably.
Andriy Plokhotnyuk
@plokhotnyuk
@fizzy33 Try jsoniter-scala. It can parse white space separated rows or JSON array rows from input streams without holding a whole input in the memory. Also it is one of most efficient JSON parsers for Scala, please see results of benchmarks here or test immediately in your browser using this page.
Hanns Holger Rutz
@Sciss
the dependency copy-to-clipboard links on https://www.scala-js.org/libraries/facades.html are partly outdated; e.g. scalajs-dom copies 0.9.5 while latest release is 1.1.0
Hanns Holger Rutz
@Sciss

if I want to store a JS object in a Blob, created from Scala, is all I have to do extends js.Any (and @JSExportAll)? Like

@JSExportAll case class Meta(blockSize: Int, lastModified: Long) extends js.Any

val b = new Blob(js.Array(Meta(12, 345L)))

?

and in return when I have a Blob, can I just asInstanceOf[Meta] its contents?
i.e. I would like a translation to { blockSize: 12, lastModified: 356 }
Hanns Holger Rutz
@Sciss

I'm trying 'Literal object construction using an Scala object interface' from here, but it seems wrong:

import scalajs.js
import org.scalajs.dom

trait Meta extends js.Object {
  val blockSize: Int = js.native
  val lastModified: Long = js.native
}

object Meta {
  def apply(blockSize: Int, lastModified: Long): Meta =
    js.Dynamic.literal(blockSize = blockSize, lastModified = lastModified).asInstanceOf[Meta]
}

Compiler says 'Members of non-native JS traits must either be abstract, or their right-hand-side must be js.undefined'

Compiles if I add @js.native. Ok?
It's complaining about the Long now: 'method fromLong in object Any is deprecated (since forever): A Long is converted to Double to be cast to js.Any. This is almost certainly not what you want. Use .toDouble explicitly if you need it.' But I do want a Long
Sébastien Doeraene
@sjrd
There's no such thing as a Long in JS, so if your intent is to store that object in a blob, you're out of luck. You'll need to store it as two Ints or something.
Or as a BigInt.
Hanns Holger Rutz
@Sciss
Ok. Well, two Ints should be fine
Sébastien Doeraene
@sjrd
(a JS bigint, that is)
Eric K Richardson
@ekrich
Wouldn't a better approach be to send JSON back and then materialize that into an object on the JS side?
Hanns Holger Rutz
@Sciss
I have no idea what that means :) But I think for my meta data I should not use Blob but ArrayBuffer, which is simpler for me to get back (I understand that Blob#arrayBuffer may not work in Safari). In general, I prefer binary formats. Just moving ahead with trial-and-error. "All you never wanted to know about JavaScript"
Hanns Holger Rutz
@Sciss
how is nio.ByteBuffer implemented? There is not by chance a secret interface directly to Uint8Array without having to convert to/from regular arrays?
Hanns Holger Rutz
@Sciss
thanks; I see that allocateDirect is the trick to have it back'ed by typed arrays, and I could probably find a low-level array-to-array copy using the underlying typedArray() or dataView().
aappddeevv
@aappddeevv
@fizzy33 The other thing to ask is whether you use all 100K rows at one time. It's not clear that you are. If you are showing a table, then table virtualization might be helpful.
Glen Marchesani
@fizzy33

thanks everyone @ritschwumm @plokhotnyuk @aappddeevv
Unfortunately the way the data set is traversed all the data is needed. The data set could be cut down by 50% but that won't get us/me close enough on the performance envelope.

I have tried json-iter and it is better and could.does solve my immediate need but not the growing data set need. This is the library I PoC'ed with to test how much performance I could get with traditional intermediate AST parsing.

Our model is defined by a domain language that we generate the database tables AND the scala case classes from. We may just go the wrote and implement a version of the model that works in browser off of raw json objects. So we would JSON.parse(jsonStr).asInstanceOf[JsNativeModel]
and just take care
we can validate the model for dev and debug purposes but in the production use case we don't
aappddeevv
@aappddeevv
If you can't reduce the initial data load (virtualization, using local storage as a cache, etc.) and it all has to be there from the first moment, I always found that with even with a few thousand points, to your point, I skipped the creation of any scala data structures and used native traits and ensure that the path from data ingest in the browser to the "display/use" component had fewest transformations possible. There's always some deserialization going on and it sounds like no matter which way you go, that's the way it is. That's why I also built scalajs-reaction that way...very thin layer over js react components in order to avoid conversions.
Glen Marchesani
@fizzy33
got ya, very useful insight thanks @aappddeevv
aappddeevv
@aappddeevv
I think the only other choice is if you can gradually add the data as it streams in, then you have a shot at having something displayed before it all arrives. And have an indicator to show as its loading. Sort of like a fuzzy picture being sent before the hi-res arrives. Good luck though.
Glen Marchesani
@fizzy33
yeah agreed, right now 95% of the performance comes from the json deserialization. We already optimized the network / data load path (we keep all data in browser and only send diffs).
Hanns Holger Rutz
@Sciss
when I call slice on an Uint8Array, sjs somehow calls Iterable.slice and returns Iterable[Short]. What to do?
Hanns Holger Rutz
@Sciss
well, no worries, I actually need subarray
still to point out that the iterable method shadows the native method
Eric K Richardson
@ekrich
On Scala fiddle it seemed to give me a List.
Eric K Richardson
@ekrich
Hanns Holger Rutz
@Sciss

What to do with this:

val mode = org.scalajs.dom.raw.IDBTransaction.READ_WRITE
println(mode)
ERROR: undefined
Error
    at $c_sjsr_UndefinedBehaviorError.eval [as fillInStackTrace__jl_Throwable] (eval at  (https://scalafiddle.io/resultframe?theme=light:46:37), :2514:14)
    at $c_sjsr_UndefinedBehaviorError.eval [as fillInStackTrace__jl_Throwable] (eval at  (https://scalafiddle.io/resultframe?theme=light:46:37), :5336:67)
    at $c_sjsr_UndefinedBehaviorError.eval [as init___T__jl_Throwable__Z__Z] (eval at  (https://scalafiddle.io/resultframe?theme=light:46:37), :2533:10)
    at $c_sjsr_UndefinedBehaviorError.eval [as init___T__jl_Throwable] (eval at  (https://scalafiddle.io/resultframe?theme=light:46:37), :5343:58)
    at $c_sjsr_UndefinedBehaviorError.eval [as init___jl_Throwable] (eval at  (https://scalafiddle.io/resultframe?theme=light:46:37), :5339:67)
    at $throwClassCastException (eval at  (https://scalafiddle.io/resultframe?theme=light:46:37), :217:46)

According to https://developer.mozilla.org/en-US/docs/Web/API/IDBTransaction , the constants are deprecated, and apparently no longer defined (tested Firefox and Chromium). Perhaps it should at least be indicated in the docs?

mvillafuertem
@mvillafuertem
Hi. Unit testing Scala.js: What the best way to Read test data from file residing in test/resources?
Hanns Holger Rutz
@Sciss
anyone aware of a correct implementation of java.text.SimpleDateFormat ?
Sébastien Doeraene
@sjrd
If there is exists one, it's in scala-java-locales.
To read resources, the typical recommendation is to use source generation. If only for tests and you can test on Node.js, you can also use the functions of the fs module.
4 replies
Hanns Holger Rutz
@Sciss
is there any nice plotting library that works with Scala.js 1.x (Scala 2.13)?
I just need simple 1D or X/Y plots, e.g. from a Vector[Double]