These are chat archives for nlf/dlite

17th
Mar 2016
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:34
I've had issues with routes and multiple bridges on the same subnet when using xhyve separately from dlite
You can see that issue though in the logs pretty clearly however
Hi beautiful people
Nathan LaFreniere
@nlf
Mar 17 2016 16:36
hi there
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:37
Thanks for fixing that bug, you took a similar approach as I did to debug it, although mine was sloppy;)
Nathan LaFreniere
@nlf
Mar 17 2016 16:38
:) it took me a bit to realize it was because logs -f wasn't actually a stream
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:38
Can you believe that?
Nathan LaFreniere
@nlf
Mar 17 2016 16:38
it's a bit odd
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:39
Why in the world is that the case? That's what was tripping me up.
Nathan LaFreniere
@nlf
Mar 17 2016 16:39
it made sense for docker pull though, that one i knew wasn't a stream and was just a chunked response with line delimited json
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:39
I mean, why not just always use ws:// then?
Nathan LaFreniere
@nlf
Mar 17 2016 16:39
they don't even use websockets, it's a plain tcp socket
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:40
Oh, it's not over http?
Nathan LaFreniere
@nlf
Mar 17 2016 16:40
it's initiated by http, but after the initial request comes in it's all straight tcp
doesn't use websocket framing
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:40
Oh, so it's just an upgrade tunnel
Nathan LaFreniere
@nlf
Mar 17 2016 16:41
yup
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:41
That's what I meant by ws://
I was just off in my terminology
Nathan LaFreniere
@nlf
Mar 17 2016 16:41
ah, gotcha. yeah it's a very similar thing
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:42
I still don't understand the purpose of upgrade when connect existed
I must not be hip enough
They're practically the same semantics
Nathan LaFreniere
@nlf
Mar 17 2016 16:43
iirc it's because upgrade is designed to work through proxies
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:43
Ssl works through proxies just fine though
;)
Nathan LaFreniere
@nlf
Mar 17 2016 16:43
sure does
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:44
So I still don't understand
Lol
Connect is trivial to implement too
Insert random "I'm old" and "these technologies, they're always a-changing" remark at will
Nathan LaFreniere
@nlf
Mar 17 2016 16:46
maybe it's an opt-in thing, upgrade says "hey please try to do this" whereas connect says "do this or fail"? i dunno
it's a guess, but i don't feel like reading RFCs to confirm it lol
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:46
Yeah, that's about all I've seen in the differences
I unfortunately have read them
Good times (not)
Nathan LaFreniere
@nlf
Mar 17 2016 16:47
i have plenty of rfc reading to do in my future already
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:47
Good times
Nathan LaFreniere
@nlf
Mar 17 2016 16:47
turns out no one has already implemented a usermode nfs server in go
not that i was able to find anyway
except for one that uses an old version of the unfs3 c code as a backend
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:48
Are you looking into that over plan9port?
Nathan LaFreniere
@nlf
Mar 17 2016 16:48
yeah
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:48
Isn't there a better network fs protocol by now
Nathan LaFreniere
@nlf
Mar 17 2016 16:48
it's that or i have to update the p9 code to support 9p2010 and then also write a client because as far as i can tell the support built in to the kernel lacks it
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:48
Nfs is so chatty with small files
Nathan LaFreniere
@nlf
Mar 17 2016 16:48
9p is no better
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:49
I've noticed
I was using it with docker-machine-driver-xhyve before dlite
Nathan LaFreniere
@nlf
Mar 17 2016 16:49
and since the only devices i can emulate at the xhyve level are block devices, i can't really make a passthrough that way
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:50
At least it works well with dlite without hackery
Sure would be nice to be able to use a shared fs
Nathan LaFreniere
@nlf
Mar 17 2016 16:50
yeah
it really would be
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:51
Ceph?
It works well.
Not sure about it's osx support however
Nathan LaFreniere
@nlf
Mar 17 2016 16:51
it's also distributed, not shared
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:51
It's both
Ish
Nathan LaFreniere
@nlf
Mar 17 2016 16:51
oh is it? i've only seen it used in a distributed way
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:52
You can use it ala nfs
With cephfs
Honestly it's going to be hard to beat a stupid simple rsync upon change notification
Nathan LaFreniere
@nlf
Mar 17 2016 16:53
hmm.. if there are go bindings that build in osx i could try that
yeah
i'm actually going to be building an rpc channel from host to guest anyway
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:54
It's going to be needed, I see why you are.
Nathan LaFreniere
@nlf
Mar 17 2016 16:54
doing the shutdown over ssh fails miserably if the ssh key doesn't work
and also with the rpc channel there i can take care of file notifications a lot more nicely
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:54
Yeah, real host integration is impossible without such a thing
What we really need is a nicer lxcish interface
Nathan LaFreniere
@nlf
Mar 17 2016 16:55
and i considered rsync, but that file duplication is a real pain
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:55
For osx
I agree with that, I also feel dirty using it
Do I smell today? Oh no that's just my file replication setup, we're good.
Nathan LaFreniere
@nlf
Mar 17 2016 16:56
yeah basically
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:56
So with ceph
We could just throw that into a container
Then mount it on both
Nathan LaFreniere
@nlf
Mar 17 2016 16:57
if i can find a decent network filesystem where the server has been implemented as a go library i'd be golden
Trevor Joynson
@akatrevorjay
Mar 17 2016 16:57
That is a different use case however
Since local would only exist remote
But I don't think I'd mind that
I did a lot of testing a few years ago for replicated vm disk storage, ceph was nice. Not the MOST performant, but not near the worst either. This was with large disk images, do lots of random io.
So*
Nathan LaFreniere
@nlf
Mar 17 2016 17:00
isn't ceph designed as more of a "create a new shared filesystem" vs "share this filesystem i already have" though?
Trevor Joynson
@akatrevorjay
Mar 17 2016 17:00
Exactly
But you could mount it on the host as well
Kinda hokey idea tbh
But it could work, just a different use case
Nathan LaFreniere
@nlf
Mar 17 2016 17:01
yeah.. i'm not sure i like that, we'd still have the problem of duplicated data
Trevor Joynson
@akatrevorjay
Mar 17 2016 17:01
I was thinking you would just store your project there
So no duplication
Nathan LaFreniere
@nlf
Mar 17 2016 17:02
ah, but dlite isn't a single project thing :) i share the user's whole home directory to make volume binding in docker as intuitive as possible
without putting their entire home directory in ceph, we'd lose out on that simplicity
Trevor Joynson
@akatrevorjay
Mar 17 2016 17:02
Yeah I know
Could do both
;)
Nathan LaFreniere
@nlf
Mar 17 2016 17:03
if i'm going to implement one shared fs that's enough for me
Trevor Joynson
@akatrevorjay
Mar 17 2016 17:03
Truth.
Nathan LaFreniere
@nlf
Mar 17 2016 17:04
especially if i have to do it from scratch, which is looking like the case
Trevor Joynson
@akatrevorjay
Mar 17 2016 17:05
Is linking against existing libs possible with eg choice?
Cgo *
I don't know how mature go's bindings are yet
Nathan LaFreniere
@nlf
Mar 17 2016 17:05
yup, if you can find a usermode nfs 4 server written in c i can bind against that
all of xhyve is actually a cgo binding right now
i thought about trying to bind nfs-ganesha, but it's a big code base
and does way more than i need
and in truth, my go is a lot stronger than my c, so it would be a lot simpler for me if i didn't have to use c libs
Nathan LaFreniere
@nlf
Mar 17 2016 17:12
if i knew more about hardware programming i'd probably be trying to port all of the xhyve code to go and just bind to the frameworks through cgo
some day
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:31
I've done s lot if cython bindings
A lot*
Maybe I could help out
My go isn't as good as my c
Nathan LaFreniere
@nlf
Mar 17 2016 19:32
between the two of us, we might be able to make it look like we know what we're doing lol
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:33
And my c leaves something to be desired ;)
I'm down.
Nathan LaFreniere
@nlf
Mar 17 2016 19:33
so my primary concern right now is the file share
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:33
I've been wanting a new project anyway to be honest
;)
Nathan LaFreniere
@nlf
Mar 17 2016 19:33
i need something that's faster and also can be embedded
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:34
And handles permission mapping
Nathan LaFreniere
@nlf
Mar 17 2016 19:34
right, i need to be able to hack permissions stuff up
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:34
Since uids vary in containers
Nathan LaFreniere
@nlf
Mar 17 2016 19:34
i'm blatantly violating standards currently
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:34
Yeah you did what we all do currently with docker
Nathan LaFreniere
@nlf
Mar 17 2016 19:34
basically when the guest changes ownership of a file, i write it to an extended attribute instead of actually changing it on the host
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:35
That's a good solution
Nathan LaFreniere
@nlf
Mar 17 2016 19:35
and then i report the ownership back to the guest as being what's in the extended attribute
falling back to actual permissions if no xattr is set
in an ideal world, i'd have an nfs4 server that could do that
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:35
That's exactly what I was going to say we could do
Nathan LaFreniere
@nlf
Mar 17 2016 19:35
and bind it to go
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:35
You're on it man
Nathan LaFreniere
@nlf
Mar 17 2016 19:36
that's what i'm doing with 9p right now, but 9p doesn't support statfs calls so du and the like report the disk is full
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:36
Using nfs4's built-in uidmap I assume?
Nathan LaFreniere
@nlf
Mar 17 2016 19:36
if i can, yeah that'd be nice
if not it'll have to be hacked in
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:36
We definitely should be able to
Nathan LaFreniere
@nlf
Mar 17 2016 19:36
i would think so
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:36
That's a pretty common need
Nathan LaFreniere
@nlf
Mar 17 2016 19:37
nfs does uid mapping, so we should be able to make it just work
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:37
Exactly
Nathan LaFreniere
@nlf
Mar 17 2016 19:37
9p doesn't so it was an awful hack
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:37
I'm surprised it works at all
Plan9 on osx that is
Nathan LaFreniere
@nlf
Mar 17 2016 19:37
:)
a 9p server is easy, it's just rpc
pretty similar to nfs actually
just a lot more basic
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:38
Yeah but it's super hacky
The way it's used currently through virtio I mean
Nathan LaFreniere
@nlf
Mar 17 2016 19:38
yeah i agree
it's also really slow
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:38
It's a patch strewn across github repos
Nathan LaFreniere
@nlf
Mar 17 2016 19:39
yeah, once i have a different solution i plan to remove it
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:39
Why isn't that mainstream yet BTW?
Your stuff isn't hacky
Nathan LaFreniere
@nlf
Mar 17 2016 19:39
because the primary maintainer of xhyve does this thing where he's active for a week, then disappears for 4 months
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:39
I meant the lib9p changes felt that way to me
Nathan LaFreniere
@nlf
Mar 17 2016 19:39
so things rarely get merged
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:39
Ah
And there's multiple forked repos
Nathan LaFreniere
@nlf
Mar 17 2016 19:40
yeah i know what you mean, the person who implemented the initial lib9p stuff literally just copied the directory into the xhyve repo
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:40
Yes!
ugh, that was terrible
Nathan LaFreniere
@nlf
Mar 17 2016 19:40
yeah it's not pleasant that's for sure
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:40
It took me a minute to understand what was even going on
I did all of this before I knew about dlite
Nathan LaFreniere
@nlf
Mar 17 2016 19:41
and then cgo is a bit ridiculous so to use it you have to flatten out the source
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:41
Trying to get 9p sharing with dm-d-xhyve
Yes.
You understand the plight.
Nathan LaFreniere
@nlf
Mar 17 2016 19:41
oh i very much do
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:42
Go is great, but parts of it make me want to commit seppuku
I'm driving home, if you're avail in a bit I'll be in
On *
Nathan LaFreniere
@nlf
Mar 17 2016 19:42
package management and directories being their own packages are what drive me crazy
just let me put my files in a directory, it's ok
yeah i'll be around
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:43
Lol
Glide at least makes that part usable
Nathan LaFreniere
@nlf
Mar 17 2016 19:43
i've been using govend to vendor stuff and go 1.6 so the built in vendoring is enabled by default
works for me
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:44
The concept of main.go is strange to me as well
I'll rephrase, the restrictions on it
Glide integrates with the 1.5/1.6 vendoring and all that too
Never tried govend
Go is in a nice momentum state rn
Nathan LaFreniere
@nlf
Mar 17 2016 19:46
govend only works with the built in vendoring. all it does is find your imports and stick them in the vendor directory. nothing more
Trevor Joynson
@akatrevorjay
Mar 17 2016 19:48
Ah. Glide is the same then, just provides stupid shortcuts and a file to manage version shas
They are one and the same from the sounds of it
Nathan LaFreniere
@nlf
Mar 17 2016 19:48
yeah super similar
Trevor Joynson
@akatrevorjay
Mar 17 2016 20:42
Hey so this is probably a bad idea but
What
What about making our own fuse fs
Could we be any faster than NFS? We could take advantage of our insanely high speed link
If we could do say rdma, we could maybe even do zerocopy somehow
But it would just be a pass-through
Take advantage of the host's fs cache, no need for the vm to care if it's quick and simple enough
I guess really we could just do it right and not have it be fuse as well, but fuse with go might be quick enough, and definitely easier to manage
Nathan LaFreniere
@nlf
Mar 17 2016 20:55
i've considered it, but hadn't gotten far enough to see how feasible it is
i mean, i don't see why not
Trevor Joynson
@akatrevorjay
Mar 17 2016 20:55
:thumbsup:
Maybe it would be slow as balls too
Nfs has only had you know
Years of development on iy
Nathan LaFreniere
@nlf
Mar 17 2016 20:56
right
also there's already a client readily available for the vm
Trevor Joynson
@akatrevorjay
Mar 17 2016 20:57
I wish the gitter Mobile App let you edit
Yeah
Trevor Joynson
@akatrevorjay
Mar 17 2016 21:28
Wow there are some cool Go fuse filesystems out there! Bazil is amazing!
Nathan LaFreniere
@nlf
Mar 17 2016 21:41
yeah :) i looked through their code a bit