These are chat archives for bjz/gfx-rs

2nd
Jul 2014
Dzmitry Malyshau
@kvark
Jul 02 2014 00:32 UTC
@csherratt I assume we can close #30?
Coraline Sherratt
@removed~csherratt
Jul 02 2014 01:09 UTC
Yep. I closed it.
Brendan Zabarauskas
@brendanzab
Jul 02 2014 01:38 UTC
@kvark I like #48! sorry for just nodding and stuff! I think the layers are holding us back
@csherratt Bindless is awesome. One of my biggest irritations with OpenGL (not that I have don a huge amount)
Brendan Zabarauskas
@brendanzab
Jul 02 2014 02:36 UTC
@kvark Could you explain the Environment (render::envir::Storage) to me again?
Brendan Zabarauskas
@brendanzab
Jul 02 2014 05:09 UTC
Type safe handle thingy: http://is.gd/jJdTvZ
Coraline Sherratt
@removed~csherratt
Jul 02 2014 13:19 UTC
@bjz what
Neat
Dzmitry Malyshau
@kvark
Jul 02 2014 13:51 UTC
@bjz Environment structure captures all shader parameters (note: vertex attributes are not parametes, they are inputs, and they are provided by the Mesh). It can be used with multiple shaders, and all it allows all the operations to be one-way (not requiring feedback from the render thread), while providing maximum performance (parameter name is mentioned only once, and is not accessed on the fast rendering path).
@bjz Honestly, I woke up today not being so definite about Environment as I was when the idea hit me. In particular, the fact that the environment has the state (even though it's very limited - only parameter values can change, not their count or types; also, the environment allows shader program to not have the state) makes me think again about how is that better than just having the program with the state. All in all, any suggestions to make it better are welcome!
Dzmitry Malyshau
@kvark
Jul 02 2014 14:21 UTC
@bjz I created the Ready to Roll milestone, which you can point to if anyone asks you when gfx-rs is ready for their project.
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:12 UTC
neat
@kvark what do you think of the handle thing?
Dzmitry Malyshau
@kvark
Jul 02 2014 15:23 UTC
@bjz I was going to ask you about it, but didn't want to flood the channel before you read my other notes ;) So, what problem does this handle thing solve exactly (i.e. why/where do we need it)? Also, how does it interact with @photex EntityData thing?
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:27 UTC
I just get worried seeing all these typedefs with no enforcement - but maybe it's not a big deal
:P
So Environments bunch together parameters so that you don't have to request each one individually?
Dzmitry Malyshau
@kvark
Jul 02 2014 15:29 UTC
@bjz most of these typedefs are for sure temporal, and were supposed to be refactored by the resource management system (#22)
@bjz yep, pretty much. No feedback is needed from the render thread in order to assign and pass parameters
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:31 UTC
awesome!
just trying to get a handle on it
hah
'handle'
Dzmitry Malyshau
@kvark
Jul 02 2014 15:31 UTC
lol
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:32 UTC
I am just trying to get the ergonomics straightened out is all :)
so why would you need to alter environments after they are built?
Dzmitry Malyshau
@kvark
Jul 02 2014 15:35 UTC
@bjz the environment structure is immutable, but you can change the values. E.g. you can use the same environment for the whole render phase of drawing stuff into the shadow map, and per model you'd like to set the uniform buffer (or value) of the model matrix.
brendanzab @bjz wonders if, due to Environments being coarser grained, they would be more amenable to being reference counted
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:37 UTC
@kvark oh yes, of course
Dzmitry Malyshau
@kvark
Jul 02 2014 15:37 UTC
@bjz reference counted on which thread? If on the user one, then how does it send a message in the destructor?
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:38 UTC

Re. altering, I mention because you said:

In particular, the fact that the environment has the state (even though it's very limited - only parameter values can change, not their count or types;

@kvark ah yeah - I dunno. maybe I dunno what I'm talking about :P
Dzmitry Malyshau
@kvark
Jul 02 2014 15:39 UTC
@bjz just raw crazy ideas are great, please continue to generate them ;)
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:41 UTC
I definitely like the idea of Evironments
Batching stuff up is really nice
What does the Mark do? That is unclear to me.
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:46 UTC

gfx-rs: A bindless graphics API abstraction for Rust

??

Dzmitry Malyshau
@kvark
Jul 02 2014 15:46 UTC
@bjz Please consider the following alternative. create_program() returns a descriptor, in which the user can get variable handles (not add them like with Environment, because program descriptor already knows what program needs). Then the user can just set these values per program handle in the same way it is done now per environment handle: renderer.set_program_uniform(program_handle, uniform_var, value). That would make program state mutable, but it would simplify things quite a bit. In particular, we'd need one less parameter for the draw call. The big downside though is that the user will need to do this for every program he has. So having multiple models of possible different shaders (that share most of the parameters) becomes a huge burden. We can leave Environment at least to get this stuff more convenient... Just something to thing about (for @csherratt as well)
@bjz Mark is supposed to be a semi-unique hidden value to protect variable handles to be used on foreign environments. Once the variable handle is generate (by, for example, env.add_uniform()), it has to be used with this environment only, and Mark is how I enforced it.

gfx-rs: A bindless graphics API abstraction for Rust

This surely captures the juice, but I feel that something is missing

Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:49 UTC
So why doesn't the program state have to be mutable right now?
@kvark agreed, re. the tag line
And why would the program state have to be mutable in an Environmentless world?
@bvssvni o/
Dzmitry Malyshau
@kvark
Jul 02 2014 15:53 UTC
@bjz Because the user may assign parameter values on it, and the following draw calls may actually depend on them? So, technically, it wouldn't change any mutability modifies on the structs or their fields. It's just the perception of the program by the user - if he has an ability to change program state, then the program is considered mutable.
Brendan Zabarauskas
@brendanzab
Jul 02 2014 15:57 UTC
So could you define 'program state' for me? Just the parameters defined for it?
Dzmitry Malyshau
@kvark
Jul 02 2014 16:00 UTC
@bjz Yeah, I suppose that's it.
Dzmitry Malyshau
@kvark
Jul 02 2014 16:08 UTC
@bjz On the other hand... having many different programs should be discouraged, so perhaps forcing the user to manage parameters for each program is not such a big deal?.. I wish we could make it less binding though, but I feel that would require the user to pass all the parameters on every draw call, which is a bit too heavy...
Brendan Zabarauskas
@brendanzab
Jul 02 2014 16:10 UTC
Have you thought of how expensive each option is? Ergonomically, what should we be encouraging users to do?
brendanzab @bjz has to get back to work now
Brendan Zabarauskas
@brendanzab
Jul 02 2014 16:16 UTC
EFL ftw!
Dzmitry Malyshau
@kvark
Jul 02 2014 16:20 UTC
@bjz @csherratt Ok, let me think about it. First of all, our options:
  1. Assign program parameters per program, which will have it's mutable state exposed. Cost is low, ergonomics is pretty good. Chance to shoot yourself in the foot (SYF chance) is high.
  2. Have Environment structure to contain all the parameters, it is partially-mutable, and can be used with multiple programs at will (that's what we have now). Cost is pretty low due to shortcut cache, ergonomics is a bit worse (due to the need to care about environment), and the SYF chance is normal (parameter values can still leak if user forgets to set them).
  3. Pass all the parameters with the draw call, thus making the program state immutable from the user perspective. Cost is higher, but depends on the implementation of the parameter descriptor. Cloning Environment would involve 3 heap allocations for the vectors, for example. Ergonomics is the same as in 2, and the SYF chance is low (user is in full control of the parameters)
Brendan Zabarauskas
@brendanzab
Jul 02 2014 16:25 UTC
Could the vectors be wrapped in Options?
ie. for option 3?
Brendan Zabarauskas
@brendanzab
Jul 02 2014 16:35 UTC
Would cause some overhead with null-checks, but reduce the need for allocation
hm
Corey Richardson
@cmr
Jul 02 2014 16:59 UTC
Vec::new() does no allocation.
So if it's empty, you don't pay anything, don't need an Option
Brendan Zabarauskas
@brendanzab
Jul 02 2014 17:02 UTC
ooooh
nice!!
o/ @cmr
Dzmitry Malyshau
@kvark
Jul 02 2014 17:04 UTC
@cmr it's nice, but our vecs are likely to be non-empty
@bjz wrapping them for options is doable, but we'd have to limit the number of elements then
Brendan Zabarauskas
@brendanzab
Jul 02 2014 17:13 UTC
hmm, indeed
Dzmitry Malyshau
@kvark
Jul 02 2014 17:13 UTC
@bjz Personally, the bindless nature and SYF chance are the most important things for me, in regards to the shader parameter design, so I'd prefer Option-3 to be shaping up. In past, my Environment thing was just a dictionary of string -> uniform, that the program took values from upon binding. As I said in the #19 description, this is not quite what we need here, but at least it was option-3 in a sense.
Brendan Zabarauskas
@brendanzab
Jul 02 2014 17:13 UTC
so not allocating on empty is not really an advantage
option 3 sounds the most compelling/attractive to me
immutability and low SYF factor is awesome
I think we could probably figure out the allocation issues - that is an engineering problem
Dzmitry Malyshau
@kvark
Jul 02 2014 17:20 UTC
@bjz Agreed. Well, the good thing is that we at least have some solution, so someone can do the terrain sample in parallel with our future design changes. I've reopened #19 to figure out how to implement Choice-3..
@bjz Another question is, do we want the environment to be bound strictly 1:1 to a program? If we do, then program creation can return an environment struct with all the parameters laid out, so the user only needs to change the values. It would also render the Shortcut irrelevant, and no caching is going to be needed (sounds like all positive signs here...)
brendanzab @bjz wonders if we can get some sort of hash of the shaders to ensure they are not re-compiled
Dzmitry Malyshau
@kvark
Jul 02 2014 17:25 UTC

hash of the shaders to ensure they are not re-compiled

That's relatively easy but irrelevant to our current situation (not a focus for Ready to Roll milestone).

Brendan Zabarauskas
@brendanzab
Jul 02 2014 17:25 UTC
yeah - bit of a micro-optimisation
Dzmitry Malyshau
@kvark
Jul 02 2014 17:28 UTC
@bjz before I start adding new labels (issue tags)... is there a naming convention you'd like to follow?
@bjz funny thing about this micro-optimization is that using a hash of the shader code brings in a small SYF factor, in case of a hash collision...
@bjz I've noticed that piston labels start with uppercase (like "Easy"), but that is a bit inconsistent with the built-in labels, which are all lower-case... We should either remove the existing ones entirely, or make new ones that follow the suit, unless you explicitly want our custom labels to differ from the built-ins.
Corey Richardson
@cmr
Jul 02 2014 17:39 UTC
What is SYF?
Brendan Zabarauskas
@brendanzab
Jul 02 2014 17:42 UTC

@cmr

Chance to shoot yourself in the foot (SYF chance)

@kvark yeah I've never really used labels all that much. we could follow rust-lang/rust?
Coraline Sherratt
@removed~csherratt
Jul 02 2014 18:25 UTC
Anyone have ideas on fallbacks opengl versions?
I just got a bug report that said I should support a fallback to 2.1. I know GFX is currently assuming 3.2.
Dzmitry Malyshau
@kvark
Jul 02 2014 18:28 UTC
from @kimundi? yeah, I saw the IRC discussion... he's video card is so weak/old... We may have a compile-time configuration to work with GL 2.1, I believe, but it needs to be implemented.
kvark @kvark is gonna be fired soon for spending so much time on github
Brendan Zabarauskas
@brendanzab
Jul 02 2014 18:40 UTC
please don't get fired!
Dzmitry Malyshau
@kvark
Jul 02 2014 18:40 UTC
@bjz on the other hand, I'll be free to work full-time on personal projects! until my wife kills me, of course ;)
Brendan Zabarauskas
@brendanzab
Jul 02 2014 18:41 UTC
haha
we would have to make some money then
Dzmitry Malyshau
@kvark
Jul 02 2014 18:41 UTC
@bjz that would bail me out for sure!
Dzmitry Malyshau
@kvark
Jul 02 2014 18:50 UTC
@csherratt created #55 for the GL-2.1 stuff
@bjz Rust has labels split into categories in format "X- lowercase-name", and they seem to cleaned up the built-in ones. Seems a bit too complex for us to go that way
Chip Collier
@photex
Jul 02 2014 19:17 UTC
@kvark I just realized that I still need to preallocate the generations field
otherwise that whole part of the setup breaks down
in particular the add method
in fact, if you're just pushing 0 onto a generations vec... why even bother with the generation
this whole thing kinda depends on preallocation... I think a version that only uses new() shouldn't be a part of this.
we can keep the allocator named something other than new though
Dzmitry Malyshau
@kvark
Jul 02 2014 19:20 UTC
@photex You can treat generations outside of the allocated range as 0. So if add needs to access it, it will extend the allocation vector. I'd like to keep it from pre-allocating 2^16 words for each handle type...
Chip Collier
@photex
Jul 02 2014 19:21 UTC
but that's part of the trouble, when you add something, and you've previously removed another item but haven't swapped it, then when or how would you know where the next allocation will be
or what it's generation should be
Dzmitry Malyshau
@kvark
Jul 02 2014 19:21 UTC
@photex let me see...
Chip Collier
@photex
Jul 02 2014 19:21 UTC
2^16 words is an excessive default for certain, so perhaps no default, and always require the user of Manager to specify how much of a thing they would like
Dzmitry Malyshau
@kvark
Jul 02 2014 19:22 UTC
@photex so add is not a problem. Swapping is only done in remove, so that one is?
Chip Collier
@photex
Jul 02 2014 19:22 UTC
using swap_remove always meant that using push made sense when adding
removing something unfortunately also invalidates the handle as we have it now
because the index of the handle will not be valid (the item it referred to was part of a swap)
so say we always push when adding, unless generations were preallocated, you couldn't happen to know that you just pushed an item onto a spot that had been used before (aka it's a different generation)
if you remove something, you need to potentially track it's index so that the next add took place in that spot
Dzmitry Malyshau
@kvark
Jul 02 2014 19:26 UTC
@photex No. If something was used there, the generations for it were allocated already. add can just check if the generations have enough room, and extend with 0es if it doesn't
Chip Collier
@photex
Jul 02 2014 19:26 UTC
alternatively using push if that value was None perhaps
so an Option for insert index
if it's Some() you're just inserting the item into that position
but even then you'd have to have a queue of indices ready to be reused
Dzmitry Malyshau
@kvark
Jul 02 2014 19:28 UTC
@photex I'm lost, sorry, I don't see why a simple check for generation.len() wouldn't solve the pre-allocation issues
Chip Collier
@photex
Jul 02 2014 19:28 UTC
@kvark I'm jumping around a bit
Dzmitry Malyshau
@kvark
Jul 02 2014 19:28 UTC
if the index is in allocated range - use it, if it's not - allocate more and be done with it
Chip Collier
@photex
Jul 02 2014 19:28 UTC
it's a matter of the add/remove pair of operations
say you add 3 items
and remove the 0th
instead of pushing on add
you would want to insert the new item at 0
then the handle would be (0, 1)
next item added would be (4, 0)
etc
but if say add 5 items, and remove the 2nd and 3rd
you'd want the next sequence of adds be (1, 1) (2, 1) (5, 0)
and so on
this is why I first tried swap_remove
you only ever add to the back
but that invalidates handles
so it doesn't work
instead we need a queue of free indices
does that sound about right?
kvark @kvark is shocked about how complicated it became
photex @photex literally just laughed out loud
Chip Collier
@photex
Jul 02 2014 19:35 UTC
an alternative to a queue could be a map from external to internal indices
so we could swap remove and not invalidate a handle
just update it's internal index
this was partly why I called this EngineData... because I was going to use this for components as well in the game
the underlying vec would always hold contiguous data that you can iterate over in a cache friendly manner
Dzmitry Malyshau
@kvark
Jul 02 2014 19:41 UTC
@photex Ok, I need to re-think the whole thing... If you feel that just having 2 arrays (data and generations) is not enough, it would be nice if you explained it in a concise manner in the PR itself.
Chip Collier
@photex
Jul 02 2014 19:41 UTC
well... yeah, I mean, I'm not sure this structure is really suited for the kinds of data that gfx is dealing with
Dzmitry Malyshau
@kvark
Jul 02 2014 19:42 UTC
It feels to me that the original code should have never worked in the first place..
Chip Collier
@photex
Jul 02 2014 19:42 UTC
which is uints and possibly some pointers
the original code was never used
I just started on it
in the voyager repo
mostly in regards to using it for scene graph and game object/entity type data
systems would use this to efficiently perform it's work in update
I think that perhaps for gfx our handles don't need to be this complicated at all
and for the most part... that's precisely what GL is providing for us anyway with the uints returned from almost everything
Dzmitry Malyshau
@kvark
Jul 02 2014 19:45 UTC
@photex Yes but we need generations to avoid accidental mis-using something that was re-used
Chip Collier
@photex
Jul 02 2014 19:45 UTC
so these handles most likely need to be some kind of correlating id or tag
right... I think that this is getting simpler again
lunch time... be back in a bit