These are chat archives for ProtoDef-io/node-protodef

2nd
Apr 2017
Hans Elias J.
@hansihe
Apr 02 2017 16:24
unions now work
so now it's fully functional for generating size_of
the cool part is that the js backend is completely done, it shouldn't require modifications to generate write and read implementations
and it's only just over 100 loc
it's gonna be really easy to write backends for other languages
Robin Lambertz
@roblabla
Apr 02 2017 16:35
@hansihe is this pushed on protodefc ?
Hans Elias J.
@hansihe
Apr 02 2017 16:53
roblabla: nope, I can do it now though
Robin Lambertz
@roblabla
Apr 02 2017 16:53
That'd be cool. I'm curious to try it out :P
Hans Elias J.
@hansihe
Apr 02 2017 16:54
oh, it's not at that stage yet
right now you call a function with a spec string and outputs javascript
there is no handling of namespaces or stuff like that
Robin Lambertz
@roblabla
Apr 02 2017 16:55
That's plenty enough to try stuff out :). And I'm mostly curious of how to adapt it to other backends ;)
that's some of the tests
that's the actual code generation logic for js
it's in the size_of.rs file, but it's not specific for it
I will move it into another file
there is also the builder.rs file, but that's just a simple way of generating js code with proper indentation and formatting
I would have used a crate for that, but there is nothing established that does what I want, so I just rolled my own
the reason why there is so little language specific code is because I moved all of the actual generation logic into the imperative_base backend
it's not really a backend on it's own, but it outputs a sort of ast that can be easily translated to code for imperative languages
should be possible to use it for functional languages like elixir too
Robin Lambertz
@roblabla
Apr 02 2017 16:59
So it's like another intermediate layer between the IR and the actual JS ?
Hans Elias J.
@hansihe
Apr 02 2017 16:59
yeah, more or less
there are many layers now
spec text -> spec ast -> ir -> imperative ast -> js
it might be overkill, but the way it's done is easily extensible, there is not much code repetition, and adding new features is still relatively easy
so I don't think it's too bad
Robin Lambertz
@roblabla
Apr 02 2017 17:01
Well, it looks ok for now at least. I'm managing to navigate the codebase fairly easily :)
mhsjlw
@mhsjlw
Apr 02 2017 17:03
im very excited to contribute backends
Hans Elias J.
@hansihe
Apr 02 2017 17:03
with this latest change doing that should be really easy for 80% of languages
mhsjlw
@mhsjlw
Apr 02 2017 17:03
but I mostly want the rust and elixir backend personally
I do know someone who would benefit from a Java backend
Hans Elias J.
@hansihe
Apr 02 2017 17:04
I don't want to start accepting backends before I have both read and write working for js though
java backend is my next priority after the js one
mhsjlw
@mhsjlw
Apr 02 2017 17:04
obviously, yes
okay sounds good
I'd be happy to work on that with you :+1:
Hans Elias J.
@hansihe
Apr 02 2017 17:04
awesome
Hans Elias J.
@hansihe
Apr 02 2017 17:37
it's pretty cool actually, I think it might be easier to write a compiler backend than an interpreter for a lot of languages
mhsjlw
@mhsjlw
Apr 02 2017 17:42
I'm just waiting for Rustler to get on trending (and Reddit) after your talk :)
although with the same token I want to work on Protodef
lol
Hans Elias J.
@hansihe
Apr 02 2017 17:46
heh, I kinda doubt it will
it's a very niche library
mhsjlw
@mhsjlw
Apr 02 2017 17:47
well , I find it useful :)
and I'm sure others do as well
I think it actually is not specific at all
it lets you literally use anything from rust inside of elixir
an interesting but complicated problem I think
it would be interesting to auto generate bindings for a crate
although probably not useful
Hans Elias J.
@hansihe
Apr 02 2017 18:03
reason why I say it's specific is because
erlang and especially elixir is a tiny programming language
rust is a small programming language
so it's like small^2
mhsjlw
@mhsjlw
Apr 02 2017 19:05
makes me sad :(
rust and elixir do a better job than every other language I've written
Well
maybe Haskell is good
Robin Lambertz
@roblabla
Apr 02 2017 19:10
@hansihe well, at least the intersection of people using (or wanting to use) rust and elixir is probably "okay" in terms of % of each language's userbase.
Hans Elias J.
@hansihe
Apr 02 2017 19:10
yeah, probably
although a lot of elixir people originate from ruby
and many don't have any experience with statically typed languages at all
Robin Lambertz
@roblabla
Apr 02 2017 19:11
is rustler elixir-specific, or is it usable in the general erlang ecosystem ?
mhsjlw
@mhsjlw
Apr 02 2017 19:11
which is funny because of how different they are
Hans Elias J.
@hansihe
Apr 02 2017 19:11
it's perfectly usable in erlang
mhsjlw
@mhsjlw
Apr 02 2017 19:11
rustler makes no effort to work with erlang :(
but it works ^
Robin Lambertz
@roblabla
Apr 02 2017 19:12
hmm
roblabla just thought of something pretty nice
Robin Lambertz
@roblabla
Apr 02 2017 19:14
Namespaces gave us a pretty easy way to handle imports in a nice way
mhsjlw
@mhsjlw
Apr 02 2017 19:17
in protodef?
Robin Lambertz
@roblabla
Apr 02 2017 19:17
yup
mhsjlw
@mhsjlw
Apr 02 2017 19:17
ah you and rom want that package manager thing
I'll agree to it if we can write a registry :)
and work on a spec
Robin Lambertz
@roblabla
Apr 02 2017 19:18
This should allow protodef to be trivially extensible (varints and other types)
mhsjlw
@mhsjlw
Apr 02 2017 19:19
@hansihe can you please add support for constants and combination well?
Robin Lambertz
@roblabla
Apr 02 2017 19:19
My idea is to simply use github for storage, and then have a static website hitting github's search API.
mhsjlw
@mhsjlw
Apr 02 2017 19:19
*as
then we can remove our raknet native types
Robin Lambertz
@roblabla
Apr 02 2017 19:19
Using tags to find protodef types
Hans Elias J.
@hansihe
Apr 02 2017 19:20
i'm against a package manager
i'm for having a way to fetch types from some repository
mhsjlw
@mhsjlw
Apr 02 2017 19:20
so am i
just github modules
or whatever they're called
Hans Elias J.
@hansihe
Apr 02 2017 19:20
mhsjlw: constants are planned, what do you mean by combination?
Robin Lambertz
@roblabla
Apr 02 2017 19:20
@hansihe my idea of a package manager is very much just a way to fetch types from some repo
But then we do need a way to properly structure a "type repo"
and a way to find them :)
Hans Elias J.
@hansihe
Apr 02 2017 19:23
way to find them could be a list in a file in the spec repo or something
Robin Lambertz
@roblabla
Apr 02 2017 19:25
@hansihe the idea would be that you would add "protodef-type" to your topics on your github repo
mhsjlw
@mhsjlw
Apr 02 2017 19:25
Test
Robin Lambertz
@roblabla
Apr 02 2017 19:25
and then I'd just build a pretty website that list them.
Hans Elias J.
@hansihe
Apr 02 2017 19:25
oh, yeah
mhsjlw
@mhsjlw
Apr 02 2017 19:25
nothing
Hans Elias J.
@hansihe
Apr 02 2017 19:25
that's a good idea
mhsjlw
@mhsjlw
Apr 02 2017 19:25
having 3 u8s and combination into a string with custom separators or something
ip to string
Hans Elias J.
@hansihe
Apr 02 2017 19:25
doesn't even need a website, just a link
Robin Lambertz
@roblabla
Apr 02 2017 19:26
well, static html with client-side JS doing calls to the gh search
Then we'd just have to specify how your repo needs to look so people who want to use bitbucket or whatever private infra still can do it
But the "canonical place" for our search engine would be github
quick, dirty, and perfect :D
@hansihe I'll probably work on adding a "JSON" backend to protodefc so we can transform protodef back to a protodef JSON format
mhsjlw
@mhsjlw
Apr 02 2017 19:28
uh
why not just tag your repo with protodef tag
Hans Elias J.
@hansihe
Apr 02 2017 19:29
oh, my idea is to handle that a bit differently
mhsjlw
@mhsjlw
Apr 02 2017 19:29
or like, exploit GitHub's ' repositories'
Robin Lambertz
@roblabla
Apr 02 2017 19:29
@mhsjlw "topic" is the github way to call a tag. They name it topics because git already has something called "tags"
mhsjlw
@mhsjlw
Apr 02 2017 19:29
*dependent
Robin Lambertz
@roblabla
Apr 02 2017 19:30
but tags are for things like tagging a commit for versioning or things like that
mhsjlw
@mhsjlw
Apr 02 2017 19:30
then you can see who is using your definitions
ex. 3 projects depend on the varint repository
and this project depends on varint and nbt
Robin Lambertz
@roblabla
Apr 02 2017 19:30
Having access to your dependents would be nice, but I think we can think about it when we grow :P
Hans Elias J.
@hansihe
Apr 02 2017 19:30
i guess you could have it as a backend, but both the protocol_spec and protocol_json frontends will be able to convert both from and to ir
the idea is to be able to go between them losslessly, without losing any docs or anything
mhsjlw
@mhsjlw
Apr 02 2017 19:30
GitHub could do that for us
Robin Lambertz
@roblabla
Apr 02 2017 19:31
THe idea is that what I propose is exactly 0-cost
mhsjlw
@mhsjlw
Apr 02 2017 19:31
@hansihe how would you handle it?
oh
I don't like that
I want this to be the standard
how else are we going to add new breaking features?
Robin Lambertz
@roblabla
Apr 02 2017 19:32
@mhsjlw what ?
@hansihe Well, my opinion is that the IR should have access to the docs and whatever, and then protocol_spec and protocol_json have both a back and frontend. It should be easier that way, I think ?
Hans Elias J.
@hansihe
Apr 02 2017 19:32
yeah, the IR has the docs in it
I guess we are thinking about the exact same thing
Robin Lambertz
@roblabla
Apr 02 2017 19:33
Yeah, I think we are :P
Oh and if we can go from JSON to protocol lang, I'm going to love this so much
Hans Elias J.
@hansihe
Apr 02 2017 19:33
except that I think of the combination of format <-> ir as a frontend
Robin Lambertz
@roblabla
Apr 02 2017 19:33
yup
Ah well
I see format -> ir as frontend
Hans Elias J.
@hansihe
Apr 02 2017 19:33
although calling the ir -> format part a backend is more technically correct, but yeah
Robin Lambertz
@roblabla
Apr 02 2017 19:34
and ir -> format as backend
Then "format" can be a language, a docgen or just another format
Hans Elias J.
@hansihe
Apr 02 2017 19:34
yep, your terminology is more correct
I agree
gonna switch to that probably
I just didn't think that far
Hans Elias J.
@hansihe
Apr 02 2017 19:51
serialization works too now
only deserialization left then, that's slightly harder
serialization was easy because the control flow was exactly the same as for size_of
Robin Lambertz
@roblabla
Apr 02 2017 19:56
Is there a generic "frontend" trait ?
or something ?
erm
s/frontend/backend
Hans Elias J.
@hansihe
Apr 02 2017 19:57
nope
not yet at least
there isn't really a compelling reason to have one
except for having a specified interface
which I guess is a reason in and of itself, but yeah
Robin Lambertz
@roblabla
Apr 02 2017 19:58
For documentation purposes
Hans Elias J.
@hansihe
Apr 02 2017 19:58
yep
Robin Lambertz
@roblabla
Apr 02 2017 19:58
For getting new backends :P
Hans Elias J.
@hansihe
Apr 02 2017 19:58
but for now i'm doing it this way
it's easy to change later :)
slightly related, I love refactoring in rust
my process is 1. make change 2. fix compiler errors 3. it works
Robin Lambertz
@roblabla
Apr 02 2017 20:00
I'm eager for the day the Language Server will support refactoring operations
So moving types around and it fixes your whole codebase
So once I have a TypeContainer, what function do I call to get the JS output ?
ir is of type TypeContainer here
Robin Lambertz
@roblabla
Apr 02 2017 20:02
ah ok
Hans Elias J.
@hansihe
Apr 02 2017 20:03
the to_js call just converts the js ast -> js string
so the compiler is actually spec text -> spec ast -> ir -> imperative ast -> js ast -> js text
holy lasagna
Robin Lambertz
@roblabla
Apr 02 2017 20:08
Haha, but JS ast seems to just be imperative AST transformations ?
Hans Elias J.
@hansihe
Apr 02 2017 20:08
js is what saves me from string concatination hell in my logic
js ast*
Robin Lambertz
@roblabla
Apr 02 2017 20:09
XD
Hans Elias J.
@hansihe
Apr 02 2017 20:09
js ast -> js text is 1:1, there is no logic there
it's just for convenience
Robin Lambertz
@roblabla
Apr 02 2017 20:12
lol
struct Expr(pub String)
I smell hax.
CertainLach
@CertainLach
Apr 02 2017 20:13
Hi, does protodef supports other languages? (Not only js)
Robin Lambertz
@roblabla
Apr 02 2017 20:13
We're working on that. there's a python implementation in the work from @chibill
And an existing elixir impl written by @hansihe
Hans Elias J.
@hansihe
Apr 02 2017 20:15
roblabla: "by design"
Robin Lambertz
@roblabla
Apr 02 2017 20:15
XD
Hans Elias J.
@hansihe
Apr 02 2017 20:15
I didn't want to implement a JS ast down to the expression granularity level
Robin Lambertz
@roblabla
Apr 02 2017 20:15
Yeah, I expected that
Hans Elias J.
@hansihe
Apr 02 2017 20:15
that would be both tedious to use, and would take longer to implement
Robin Lambertz
@roblabla
Apr 02 2017 20:16
(And be completely useless)
Hans Elias J.
@hansihe
Apr 02 2017 20:16
yep
Robin Lambertz
@roblabla
Apr 02 2017 20:16
TBF, the current JS ast is fairly generic. We could probably reuse builder for most impls.
Hans Elias J.
@hansihe
Apr 02 2017 20:17
yeah, that's true
could make it generic for many languages fairly easily
Robin Lambertz
@roblabla
Apr 02 2017 20:17
With a few alteration. FunctionDecl for instance isn't part of Block in C :P
Robin Lambertz
@roblabla
Apr 02 2017 20:17
@F6CF this is our work-in-progress generic compiler
It's written in rust, but compiles to JS
(And only supports a limited subset of protodef for now)
Hans Elias J.
@hansihe
Apr 02 2017 20:18
or any other language you want to write a backend for
Robin Lambertz
@roblabla
Apr 02 2017 20:18
All credits go to @hansihe for being awesome :P
CertainLach
@CertainLach
Apr 02 2017 20:19
That's sad) Because i have node.js client for node.js server, and i want to rewrite server to rust)
Hans Elias J.
@hansihe
Apr 02 2017 20:19
there is gonna be a rust backend for it
Robin Lambertz
@roblabla
Apr 02 2017 20:19
@F6CF I might have something that could bring you half the way there, maybe
CertainLach
@CertainLach
Apr 02 2017 20:20
?
sorta broken and doesn't support the whole range of protodef either
But the idea is that you give it a protodef file, and it compiles to rust code. SO it might be able to generate a lot of boilerplate code you'd have to write yourself otherwise
Like the struct definitions and things like that.
Actually, I should fix my current version and publish it, it supports a whole lot more stuff.
CertainLach
@CertainLach
Apr 02 2017 20:23
Thank you, i try it!
Robin Lambertz
@roblabla
Apr 02 2017 20:24
@F6CF it's very rough around the edges, mind you. I built it mostly as a learning experiment and totally not ready for prime-time. But if you're willing to get your hands dirty, maybe it'll help :)
Spongedown is amazing xD
Hans Elias J.
@hansihe
Apr 02 2017 23:04
deserialization works as well now
next up is namespace support/support for whole protocol spec files I think
that would make it actually usable, which is pretty exciting
once that's done, priority should be on the protocol_json frontend
so that it can be tried out with a protocol.json from minecraft-data