red/chit-chat

A place for the Red community to talk about whatever they want, to keep other rooms clean.

Oldes Huhuman

@Oldes

I think that the reason is, that map! is not the best type for something like JSON, because it requires hash tables.. and as JSON file is usually transferred to map of maps, there is wasted quite a lot of memory and computation to build hashtable for each map.

@GiuseppeChillemi if you want to just parse the file to get the info from it... use just plain parse and you will have much better times. the structure is not too complicated.

Oldes Huhuman

@Oldes

Maybe map! could internally use a block and not hashes for small maps with just few keys.

@ne1uno it will not be related to url parsing, because all urls are in this file stored as plain strings

GiuseppeChillemi

@GiuseppeChillemi

@Oldes My bookmarks are mixed: I use them as History. I save in folders "bursts" of search results about topics, so it increases 30/50 entries each time.

GiuseppeChillemi

@GiuseppeChillemi

I have also saved the map to Red file and LOADed it. It takes 4 seconds vs. 80.

The slowness is related to parsing and conversion. because at LOADing to a MAP I imagine the hash tables are recalculated.

hiiamboris

@hiiamboris

Try Redbin

Oldes Huhuman

@Oldes

I think it is because of rehashing... and stressed GC. I'm not sure if it's possible to improve it without not using map!. But as I said... it's easy to use pure parse just to get info you need without ballast which is in this file.

And my opinion is, that 01:20 minutes is bad.

Boleslav Březovský

@rebolek

Rewriting JSON codec in R/S would help if anyone wants to give it a try :)

GiuseppeChillemi

@GiuseppeChillemi

@hiiamboris
SAVED-AS-MAP -> LOADED-TO-MAP = 4 seconds
JSON -> LOADED-AS-JSON = 80 seconds.

Boleslav Březovský

@rebolek

@GiuseppeChillemi
and SAVED-AS-REDBIN -> LOADED-TO-MAP = ?

Petr Krenzelok

@pekr

That's a juge difference. The thing is, what are we exactly comparing (whatn happens behind the scenes)

GiuseppeChillemi

@GiuseppeChillemi

The slowdown resides in the JSON codec.

@rebolek I will try once at home. How doto I save to REDBIN?

Oldes Huhuman

@Oldes

Using redbin is imho not a solution... as I guess Giuseppe wants to get data from JSON, not from prepared file format.

Boleslav Březovský

@rebolek

@GiuseppeChillemi

save/as %delme #(a: b) 'redbin

@Oldes I know, I'm just nterested how faster would Redbin be over text (Red).

GiuseppeChillemi

@GiuseppeChillemi

@rebolek
I have connected remotely to my home machine, could not wait!

Oldes Huhuman

@Oldes

Yes... that will be interesting test case for such a huge data source

GiuseppeChillemi

@GiuseppeChillemi

But I have a bad result... very bad!

Boleslav Březovský

@rebolek

for the JSON loading, I guess only R/S would help

GiuseppeChillemi

@GiuseppeChillemi

>> save/as %bmarks.redbin b 'redbin
*** Internal Error: internal limit reached: 20000
*** Where: encode
*** Stack: save

Petr Krenzelok

@pekr

There is no parse in R/S, no?

Boleslav Březovský

@rebolek

aha :)

@pekr no, you would need to write the parser manually using a loop

Oldes Huhuman

@Oldes

@rebolek what if you use block instead of map in json codec?

Boleslav Březovský

@rebolek

but JSON is pretty simple format, so it wouldn't be hard

@Oldes you can try to patch the JSON codec to test it

Oldes Huhuman

@Oldes

I will wait for someone else... I don't need it so much.

Boleslav Březovský

@rebolek

me neither...I use JSON for web apis where the size is in kB or tens of kBs

Oldes Huhuman

@Oldes

Problem is, that map! is required as JSON have also block values.

So proper solution would be not using hash tables for tiny maps... like used in the bookmarks file.

hiiamboris

@hiiamboris

*** Internal Error: internal limit reached: 20000

Oh no! You killed Redbin!

hiiamboris

@hiiamboris

reported.

GiuseppeChillemi

@GiuseppeChillemi

@hiiamboris Radio Killed the RedBin

Gregg Irwin

@greggirwin

We can take this to red/red or another room. It's a good technical topic. We don't have a red/codecs room. Should we?

We do have https://github.com/red/red/wiki/%5BHOWTO%5D-SYSTEM-CODECS,-LOAD-AS-and-SAVE-AS, where we can make notes, or we can add codec specific notes on their individual pages.

JSON's is here

Boleslav Březovský

@rebolek

Codecs room makes sense to me.

Petr Krenzelok

@pekr

Gregg Irwin

@greggirwin

Done: https://gitter.im/red/codecs

GiuseppeChillemi

@GiuseppeChillemi

@greggirwin REQ: Red Advocacy Gitter Chat

(please!)

Gregg Irwin

@greggirwin

We can do most of that in red/red, where the bulk of the community will see it. Advocacy is a very general topic, but we can point to the wiki page on it as well.

GiuseppeChillemi

@GiuseppeChillemi

Advocacy often is the place "strong" conversations, so it would be useful to keep red/red free from them.

Gregg Irwin

@greggirwin

Well, let's see how it goes. New rooms usually come up after the need is determined by pushing a current room too far. :^)

GiuseppeChillemi

@GiuseppeChillemi

I can help to shorten the time for the creation of the group, but you should attract both Doc and Vladimir to join some Red advocacy discussion on Red/Red. I have a special skill to upset them (but I respect both either for their great knowledge, commitment and also for the spirits that moves their hearts).

Gregg Irwin

@greggirwin

LOL! :^)

Where communities thrive

People

Repo info

Activity