These are chat archives for ProtoDef-io/node-protodef

12th
Mar 2017
Hans Elias J.
@hansihe
Mar 12 2017 13:44
what's the point of compareToValue in switch?
heh I guess that's kind of a hack though
mhsjlw
@mhsjlw
Mar 12 2017 15:55
lmao
Hans Elias J.
@hansihe
Mar 12 2017 17:42
rom1504: i'm writing a JS target for the compiler, thinking about what kinda code to generate
I think having a constructor function like that should make it easy for V8 to inline stuff
good/bad idea?
Romain Beaumont
@rom1504
Mar 12 2017 17:44
current js serialization/deserialization code has offset and sizes everywhere, do you plan to do away with that ? (and how ?)
Hans Elias J.
@hansihe
Mar 12 2017 17:44
nope, I would probably do that
Romain Beaumont
@rom1504
Mar 12 2017 17:44
ie this interface https://github.com/ProtoDef-io/node-protodef/blob/master/doc/newDatatypes.md (minus the typeArgs and context which are removed by compilation)
Hans Elias J.
@hansihe
Mar 12 2017 17:44
I just don't know what kinda interface buffers in node have off the top of my head
oh right, thanks
that looks like what I was looking for
Hans Elias J.
@hansihe
Mar 12 2017 17:46
the compiler itself is (mostly) at feature parity with the elixir one now btw
only thing it needs is compile targets
Romain Beaumont
@rom1504
Mar 12 2017 17:48
basically js code that I would write manually if I wanted to encode/decode the test datatypes
might give some idea of what your generated code might look like
Hans Elias J.
@hansihe
Mar 12 2017 17:50
oh, thanks
that really helps
Romain Beaumont
@rom1504
Mar 12 2017 17:50
basically maybe you want to directly put the native function in the generated code instead of 'native_types["i8"]'
I'm not sure if that's better or not
Hans Elias J.
@hansihe
Mar 12 2017 17:51
that's one of the things I want to make configurable
but I'll just make everything a function call initially
Romain Beaumont
@rom1504
Mar 12 2017 17:51
(the sizeOf function is missing in my gist though)
ok
Hans Elias J.
@hansihe
Mar 12 2017 17:52
I am not sure if it's even worth it to inline stuff manually actually
v8 shouldn't have any trouble doing that optimization
what do you think?
Romain Beaumont
@rom1504
Mar 12 2017 17:53
did you manage to implement mapper in this one ?
oh
Hans Elias J.
@hansihe
Mar 12 2017 17:54
I haven't done it yet, no
Romain Beaumont
@rom1504
Mar 12 2017 17:54
do you mean the function(native_types) ?
Hans Elias J.
@hansihe
Mar 12 2017 17:54
yeah
Romain Beaumont
@rom1504
Mar 12 2017 17:54
yes it might be useful
I'm not exactly understanding why but that kind of things make v8 faster sometimes
Hans Elias J.
@hansihe
Mar 12 2017 17:55
that should put the decoder/encoder objects in the same stack frame as the functions itself, something which should make it easy to inline stuff
decoder/encoder functions
*
Romain Beaumont
@rom1504
Mar 12 2017 17:55
yeah that makes sense
Hans Elias J.
@hansihe
Mar 12 2017 17:55
I guess that can be changed relatively easy later on
Hans Elias J.
@hansihe
Mar 12 2017 17:56
oh, yep
mhsjlw
@mhsjlw
Mar 12 2017 17:57
oh nice ! you're adding optional backends and stuff ?
yeah, :+1:
*for different languages
Hans Elias J.
@hansihe
Mar 12 2017 17:57
yeah
frontends are decoupled as well
mhsjlw
@mhsjlw
Mar 12 2017 17:57
oh, that's going to be interesting
Hans Elias J.
@hansihe
Mar 12 2017 17:58
so if we want another representation other than protocol.json, then it's trivial to add
mhsjlw
@mhsjlw
Mar 12 2017 17:58
yeah that would be interesting
i would like explore a protobuf-like syntax
just because writing switches and comparisons in json is a pain in the ass
Robin Lambertz
@roblabla
Mar 12 2017 17:58
@rom1504 I just thought of something that could probably make node-protodef much faster
The thing that takes the biggest time in node-protodef is object creation for the context
thing is, we don't really need to create new objects all the time
Romain Beaumont
@rom1504
Mar 12 2017 17:59
well...
Robin Lambertz
@roblabla
Mar 12 2017 17:59
Instead, we should have a flat Map, and use prefixes or whatever
Something like a Trie would work as well.
Romain Beaumont
@rom1504
Mar 12 2017 18:01
idk. Remember I managed to make it 10 times faster by just getting rid of typeArgs by compiling it ?
mhsjlw
@mhsjlw
Mar 12 2017 18:02
Have you guys just thought of using sweetjs
and just throwing everything else out ?
:P
Romain Beaumont
@rom1504
Mar 12 2017 18:02
it's something about doing things dynamically that makes it slow. I don't really understand the inner workings of v8 to really understand why
Robin Lambertz
@roblabla
Mar 12 2017 18:02
the main thing that made it slow when I last profiled it was that we create objects everywhere
mhsjlw
@mhsjlw
Mar 12 2017 18:02
also -> what happens when you compile ProtoDef through google closure ?
Robin Lambertz
@roblabla
Mar 12 2017 18:02
My guess is, a new object is like a malloc() in C
which, after a while, gets slow
Romain Beaumont
@rom1504
Mar 12 2017 18:03
switch have field access, container have loops, both unnecessary
Robin Lambertz
@roblabla
Mar 12 2017 18:03
furthermore, if our context object always have the same shape
(since we'd just store things from the root)
I think v8 would be able to optimize it much better
It's just an idea i had while fiddling with rust-protodef
I'm currently implementing a proper context, and I'm having a hard time
Romain Beaumont
@rom1504
Mar 12 2017 18:05
well node-protodef has benchmarks
so we could try
Robin Lambertz
@roblabla
Mar 12 2017 18:05
brb
this may or may not be relevant
but that is how js engines like js makes property access on objects fast if I understand it right
you might be able to optimize for that?
Romain Beaumont
@rom1504
Mar 12 2017 18:06
yeah
that kind of thing is important but hard to understand / do right https://github.com/vhf/v8-bailout-reasons
basically v8 optimize functions, but sometimes it gives up
honestly
all the module that do that kind of things in npm
do it by compiling
for example all the json schema validator that are fast (like 100x faster than the naive ones) compile schemas
I think it's not really very surprising that if you do very dynamic code the interpreter cannot figure out a way to optimize it
mhsjlw
@mhsjlw
Mar 12 2017 18:11
So
for TOML (just an example) : packet_message = ["container", [{"name"=>"player_id", "type"=>"i8"}, {"name"=>"message", "type"=>"string"}]]
that would be packet_message in toml style
but this is still effectively the json format with just some more steps
that's why i think an actual protobuf-like language would be usefulk
*useful
Romain Beaumont
@rom1504
Mar 12 2017 18:12
only difference with json is =>
in that example
Hans Elias J.
@hansihe
Mar 12 2017 18:13
okey, so here are my thoughts on this
it can not be a format where comments are not a first-class part of the data
because in my mind at least, it would be a goal to have documentation as part of the format?
Romain Beaumont
@rom1504
Mar 12 2017 18:14
yes
but how are comments relevant ?
to me descriptions/notes are data
but yeah if you need to put it in quotes, it's not going to work
I think we found a solution about that
Hans Elias J.
@hansihe
Mar 12 2017 18:15
yeah, my point was that it would need to be a format that makes it easy to have text as a part of the data
Romain Beaumont
@rom1504
Mar 12 2017 18:15
like putting description in a separate .md file
Hans Elias J.
@hansihe
Mar 12 2017 18:15
I guess I worded it strangely
Romain Beaumont
@rom1504
Mar 12 2017 18:15
ah
Hans Elias J.
@hansihe
Mar 12 2017 18:16
json does not quite qualify here, no human would really want to write documentation directly into a json string
Romain Beaumont
@rom1504
Mar 12 2017 18:16
yeah if we want to edit the file directly, it's not convenient to edit doc in json
Hans Elias J.
@hansihe
Mar 12 2017 18:16
separate .md file would work, but then it makes it harder to automatically generate docs that show comments inline with fields
Romain Beaumont
@rom1504
Mar 12 2017 18:17
yes
inline comments need to be in the protocol def file
Hans Elias J.
@hansihe
Mar 12 2017 18:17
if we want wiki.vg level of documentation, that would be a goal
mhsjlw
@mhsjlw
Mar 12 2017 18:17
yes please ^
Romain Beaumont
@rom1504
Mar 12 2017 18:17
is there a format that makes it possible to have a nested structured and to edit text ?
mhsjlw
@mhsjlw
Mar 12 2017 18:17
hmm
Romain Beaumont
@rom1504
Mar 12 2017 18:17
*structures
mhsjlw
@mhsjlw
Mar 12 2017 18:17
yaml ? <- no
Romain Beaumont
@rom1504
Mar 12 2017 18:18
(latex is not the answer :p)
Hans Elias J.
@hansihe
Mar 12 2017 18:18
postscript
mhsjlw
@mhsjlw
Mar 12 2017 18:18
i feel like we should just steal the capn'proto format
:D
Hans Elias J.
@hansihe
Mar 12 2017 18:19
if we want to have a custom format, I would be willing to write a parser/printer for it, but i'm not sure we would
Romain Beaumont
@rom1504
Mar 12 2017 18:20
the current advantage of json is also that random people from #mcdevs start fiddling with it without having to write a complicated grammar/parser for language X
mhsjlw
@mhsjlw
Mar 12 2017 18:20
@hansihe my problem with this is getting people to learn it
or , if we do, we just steal good things from other schemas
idk, i'm not really that good at this stuff.. sorry
but i will still try to be good at it :D
Romain Beaumont
@rom1504
Mar 12 2017 18:21
anyway
Hans Elias J.
@hansihe
Mar 12 2017 18:21
to me a prerequisite with a custom format would be the ability to convert to and from a variant of the current protocol.json format losslessly
Romain Beaumont
@rom1504
Mar 12 2017 18:21
I think the advantage of changing of format is not really if that the syntax would look better or not. I think something more important would be to have something easier to handle semantically
mhsjlw
@mhsjlw
Mar 12 2017 18:22
i don't think that needs to be packaged with everything else though
just a crappy javascript script :P
idk
Romain Beaumont
@rom1504
Mar 12 2017 18:22
for example if we could avoid having to do some of the compiler passes by writing something easier to handle, it would be nice
mhsjlw
@mhsjlw
Mar 12 2017 18:22
that's why we're trying to accomplish
so switch statements are actually reable
*readable
Hans Elias J.
@hansihe
Mar 12 2017 18:23
i think the main advantage of a custom syntax is that it would enable us to put documentation as a first-class citizen in the data
it might even eliminate the need to actually generate documentation, it could be very readable directly in the text file
mhsjlw
@mhsjlw
Mar 12 2017 18:23
what if we all just pool in some example protocol files that we would like it to look like / function ?
Hans Elias J.
@hansihe
Mar 12 2017 18:23
and yeah, making stuff easier to handle for compilers would also be a goal
mhsjlw
@mhsjlw
Mar 12 2017 18:23
then write out an entire schema and see if there are things that already do it
Hans Elias J.
@hansihe
Mar 12 2017 18:24
but i'm mainly thinking about documentation here to be honest, I think that removing the friction from writing/updating/reading docs is really important
but yeah, i'm still undecided if it's actually a good idea with a custom format
we might be overcomplicating things a lot
Romain Beaumont
@rom1504
Mar 12 2017 18:26
there was an issue where we enumerated all the possibilities but I can't find it...
it's very likely in the wrong repo
yeah no really can't find it...
@roblabla do you know where it is ?
@hansihe do you think putting field notes in the json file would really be that bad ?
definitely descriptions should be in .md or something
but field notes are short
well
Hans Elias J.
@hansihe
Mar 12 2017 18:29
not necessarily field notes
i think the main issue is splitting the two up
Romain Beaumont
@rom1504
Mar 12 2017 18:29
yes I was about to say that
conceptually it's bad
it recreated the "packet" concept again
*recreates
type is a more generic concept
Hans Elias J.
@hansihe
Mar 12 2017 18:30
it might not seem like much, but when editing stuff that really does create a lot of friction
yeah, that too
another advantage with a proper format would be that it could make plaintext diffs between versions a lot more useful
Robin Lambertz
@roblabla
Mar 12 2017 18:42
I tried to come up with custom formats
And there's one very important thing IMO : it should be JSON-compatible
Like YAML
This way, everything we currently have works
And we can change it as we go.
Thing is, it's hard to come up with a good format with that constraint
Which sucks
Romain Beaumont
@rom1504
Mar 12 2017 18:44
having several format could be okay
as long as they are maintained
Robin Lambertz
@roblabla
Mar 12 2017 18:44
sure
hmm
Romain Beaumont
@rom1504
Mar 12 2017 18:45
I mostly mean as long as the transformer from one to the other is maintained
Robin Lambertz
@roblabla
Mar 12 2017 18:45
Oh, another way to go around this is by saying that each "pretty format" feature needs an equivalent JSON
yeah
so then we still work with the JSON format
we just implement a transformation from custom to JSON
Romain Beaumont
@rom1504
Mar 12 2017 18:46
or the other way round ;)
Robin Lambertz
@roblabla
Mar 12 2017 18:46
yup
That's actually something I experimented with early in protodef's life in the form of sweetjs macros
I had a prototype where we had a bunch of sweetjs macros generating the final json
Hans Elias J.
@hansihe
Mar 12 2017 18:47
that's what I was thinking about doing, yeah
Robin Lambertz
@roblabla
Mar 12 2017 18:47
SweetJS ended up being a bit of a bitch to write though.
And that was years ago. I was a much suckier programmer then :D
Anyway, we'd need to know all the features we want to implement to make something coherent
Romain Beaumont
@rom1504
Mar 12 2017 18:49
btw
I don't know if you followed
but wiki.vg is dying
well "changing host"
Robin Lambertz
@roblabla
Mar 12 2017 18:49
wut ?
who's taking over ?
Romain Beaumont
@rom1504
Mar 12 2017 18:49
tktech is retiring, he asked if somebody wants to host it
seems like he found somebody
Robin Lambertz
@roblabla
Mar 12 2017 18:50
dayum
Romain Beaumont
@rom1504
Mar 12 2017 18:50
but my point is there will be a transition period
wouldn't hurt to replace it now :P
Robin Lambertz
@roblabla
Mar 12 2017 18:50
yeah, I got your point
rom1504 @rom1504 starts his world domination plan
Robin Lambertz
@roblabla
Mar 12 2017 18:50
Well I thought long and hard about this
I think if we could generate wikitext from protodef files, it might help a good deal
Thing is, wiki.vg has a bunch of info we don't have (like prose about how the auth works 'n stuff)
(On the other hand, we do have a bunch of stuff wiki.vg doesn't have like list of blocks, entities, etc...)
Romain Beaumont
@rom1504
Mar 12 2017 18:52
if we magically had a lua protodef-to-wikitext compiler, then we could let people edit the protodef file that would be stored on a wiki page, while displaying the current thing in wiki.vg
Robin Lambertz
@roblabla
Mar 12 2017 18:53
given the fact that wiki.vg is established, I guess if we could auto-generate the current wiki.vg page from our stuff, and then ask whoever maintains wiki.vg to Lock the page and redirect the edit page to github
then we'd get a pretty satisfying situation
furthermore, the fact that we actually maintain different versions of the format is great :D
I got to add it to mcdata readme
well
Robin Lambertz
@roblabla
Mar 12 2017 18:55
my god
@rom1504 what have we done :P
Romain Beaumont
@rom1504
Mar 12 2017 18:55
he is creating c++ item classes using java code from minecraft data json file
idk
why not
Robin Lambertz
@roblabla
Mar 12 2017 18:56
I know I mean, we created an entire community of people using our data format. Projects are sprouting everywhere
It's scawwy
Romain Beaumont
@rom1504
Mar 12 2017 18:56
haha yeah
@roblabla well thing is it's an "optional parse"
I'm not sure if we want that
(the pchunk protodef thing)
Robin Lambertz
@roblabla
Mar 12 2017 18:58
thing is, we should implement a better way to parse optionally
You know how we used to not read packets we didn't care about ?
That was much better :P
Romain Beaumont
@rom1504
Mar 12 2017 19:01
that was supposed to be translated in switch implementation
which would have made it even more dynamical
kind of opposite to the compiler concept
well well
maybe not