Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Oct 18 2019 17:14
    @greggirwin banned @SmackMacDougal
Oldes Huhuman
@Oldes
I think that the reason is, that map! is not the best type for something like JSON, because it requires hash tables.. and as JSON file is usually transferred to map of maps, there is wasted quite a lot of memory and computation to build hashtable for each map.
@GiuseppeChillemi if you want to just parse the file to get the info from it... use just plain parse and you will have much better times. the structure is not too complicated.
Oldes Huhuman
@Oldes
Maybe map! could internally use a block and not hashes for small maps with just few keys.
@ne1uno it will not be related to url parsing, because all urls are in this file stored as plain strings
GiuseppeChillemi
@GiuseppeChillemi
@Oldes My bookmarks are mixed: I use them as History. I save in folders "bursts" of search results about topics, so it increases 30/50 entries each time.
GiuseppeChillemi
@GiuseppeChillemi
I have also saved the map to Red file and LOADed it. It takes 4 seconds vs. 80.
The slowness is related to parsing and conversion. because at LOADing to a MAP I imagine the hash tables are recalculated.
hiiamboris
@hiiamboris
Try Redbin
Oldes Huhuman
@Oldes
I think it is because of rehashing... and stressed GC. I'm not sure if it's possible to improve it without not using map!. But as I said... it's easy to use pure parse just to get info you need without ballast which is in this file.
And my opinion is, that 01:20 minutes is bad.
Boleslav Březovský
@rebolek
Rewriting JSON codec in R/S would help if anyone wants to give it a try :)
GiuseppeChillemi
@GiuseppeChillemi
@hiiamboris
SAVED-AS-MAP -> LOADED-TO-MAP = 4 seconds
JSON -> LOADED-AS-JSON = 80 seconds.
Boleslav Březovský
@rebolek
@GiuseppeChillemi
and SAVED-AS-REDBIN -> LOADED-TO-MAP = ?
Petr Krenzelok
@pekr
That's a juge difference. The thing is, what are we exactly comparing (whatn happens behind the scenes)
GiuseppeChillemi
@GiuseppeChillemi
The slowdown resides in the JSON codec.
@rebolek I will try once at home. How doto I save to REDBIN?
Oldes Huhuman
@Oldes
Using redbin is imho not a solution... as I guess Giuseppe wants to get data from JSON, not from prepared file format.
Boleslav Březovský
@rebolek
@GiuseppeChillemi
save/as %delme #(a: b) 'redbin
@Oldes I know, I'm just nterested how faster would Redbin be over text (Red).
GiuseppeChillemi
@GiuseppeChillemi
@rebolek
I have connected remotely to my home machine, could not wait!
Oldes Huhuman
@Oldes
Yes... that will be interesting test case for such a huge data source
GiuseppeChillemi
@GiuseppeChillemi
But I have a bad result... very bad!
Boleslav Březovský
@rebolek
for the JSON loading, I guess only R/S would help
GiuseppeChillemi
@GiuseppeChillemi
>> save/as %bmarks.redbin b 'redbin
*** Internal Error: internal limit reached: 20000
*** Where: encode
*** Stack: save
Petr Krenzelok
@pekr
There is no parse in R/S, no?
Boleslav Březovský
@rebolek
aha :)
@pekr no, you would need to write the parser manually using a loop
Oldes Huhuman
@Oldes
@rebolek what if you use block instead of map in json codec?
Boleslav Březovský
@rebolek
but JSON is pretty simple format, so it wouldn't be hard
@Oldes you can try to patch the JSON codec to test it
Oldes Huhuman
@Oldes
I will wait for someone else... I don't need it so much.
Boleslav Březovský
@rebolek
me neither...I use JSON for web apis where the size is in kB or tens of kBs
Oldes Huhuman
@Oldes
Problem is, that map! is required as JSON have also block values.
So proper solution would be not using hash tables for tiny maps... like used in the bookmarks file.
hiiamboris
@hiiamboris

*** Internal Error: internal limit reached: 20000

Oh no! You killed Redbin!

hiiamboris
@hiiamboris
reported.
GiuseppeChillemi
@GiuseppeChillemi
@hiiamboris Radio Killed the RedBin
Gregg Irwin
@greggirwin
We can take this to red/red or another room. It's a good technical topic. We don't have a red/codecs room. Should we?
We do have https://github.com/red/red/wiki/%5BHOWTO%5D-SYSTEM-CODECS,-LOAD-AS-and-SAVE-AS, where we can make notes, or we can add codec specific notes on their individual pages.
JSON's is here
Boleslav Březovský
@rebolek
Codecs room makes sense to me.
Petr Krenzelok
@pekr
+1
Gregg Irwin
@greggirwin
GiuseppeChillemi
@GiuseppeChillemi
@greggirwin REQ: Red Advocacy Gitter Chat
(please!)
Gregg Irwin
@greggirwin
We can do most of that in red/red, where the bulk of the community will see it. Advocacy is a very general topic, but we can point to the wiki page on it as well.
GiuseppeChillemi
@GiuseppeChillemi
Advocacy often is the place "strong" conversations, so it would be useful to keep red/red free from them.
Gregg Irwin
@greggirwin
Well, let's see how it goes. New rooms usually come up after the need is determined by pushing a current room too far. :^)
GiuseppeChillemi
@GiuseppeChillemi
I can help to shorten the time for the creation of the group, but you should attract both Doc and Vladimir to join some Red advocacy discussion on Red/Red. I have a special skill to upset them (but I respect both either for their great knowledge, commitment and also for the spirits that moves their hearts).
Gregg Irwin
@greggirwin
LOL! :^)