These are chat archives for SimonDanisch/FixedSizeArrays.jl

26th
Mar 2015
Kristoffer Carlsson
@KristofferC
Mar 26 2015 10:43
Is the Quaternion import really needed?
Simon
@SimonDanisch
Mar 26 2015 10:44
that was a quick fix to be honest
Kristoffer Carlsson
@KristofferC
Mar 26 2015 10:44
Oke, np
Starting to look good btw :)
Simon
@SimonDanisch
Mar 26 2015 10:45
I'm not really sure how to do this properly. In glabstraction i use quaternions and fixedsize arrays, so either of them then overwrites the normalize function
thanks =) Yeah, I was finally able to integrate it into glabstraction :)
Kristoffer Carlsson
@KristofferC
Mar 26 2015 10:46
Nice, I really like that you added mutable versions
Simon
@SimonDanisch
Mar 26 2015 10:47
yeah, they should be nice for the matrix code. I've also added a dense array for the mutable types, hope that solves the problems we have been having with mutable fixedsizearrays
Kristoffer Carlsson
@KristofferC
Mar 26 2015 10:49
Do you have any ideas what should happen if you try to for example multiply a Matrix4z4 with a normal array, like Matrix4x4(rand(4,4)) * rand(4,4)
Simon
@SimonDanisch
Mar 26 2015 10:50
good question... I wonder how expensive a good conversion is (the current conversion is not fast)... If it comes for free automatic conversion might be a clean way of doing this
Kristoffer Carlsson
@KristofferC
Mar 26 2015 10:50
Yeah, O
Simon
@SimonDanisch
Mar 26 2015 10:51
I also want some view type...
Kristoffer Carlsson
@KristofferC
Mar 26 2015 10:51
I've been experimenting with some pointer converts, like pointer_to_array(convert(Ptr{T}, pointer_from_objref(v))+sizeof(T), SIZE)
Simon
@SimonDanisch
Mar 26 2015 10:52
to extract vectors and arrays out of vectors and arrays..preferably both julia and fixedsize arrays
yes that's where I started... but they gave me some problems, and I really, really wanted to get this running to finally integrate it with glabstraction... so I went for the trivial solution for now
how did your experiments go?
I've actually haven't tested your version yet
Kristoffer Carlsson
@KristofferC
Mar 26 2015 10:55
julia> @time for i = 1:10^6
Voigt.unsafe_typevec!(e)
end
elapsed time: 0.087788188 seconds (122 MB allocated, 9.88% gc time in 6 pauses with 0 full sweep)
e is a FSA vector of size 6
Simon
@SimonDanisch
Mar 26 2015 10:58
hmm
Kristoffer Carlsson
@KristofferC
Mar 26 2015 10:59
Haven't done any real testing tbh
Simon
@SimonDanisch
Mar 26 2015 11:00
i see.. well the native code that gets emitted seems to be very long... so we might be able to get something better
Kristoffer Carlsson
@KristofferC
Mar 26 2015 11:00
Yes, for sure
Simon
@SimonDanisch
Mar 26 2015 11:07
it's probably the best we have for now...
the other way around can be made practically free it seems!
Kristoffer Carlsson
@KristofferC
Mar 26 2015 11:11
Yeah I saw that on the mailing list
Simon
@SimonDanisch
Mar 26 2015 11:14
hm one would think if one way works, than the other should as well... But it's probably the difference between stack allocated and heap allocated
so the array is already on the heap, getting a stack allocated object from it isn't very expensive then.. But for the other way around, you need to heap allocate memory
Kristoffer Carlsson
@KristofferC
Mar 26 2015 11:15
Yes, I think you need to [a,]
Simon
@SimonDanisch
Mar 26 2015 11:16
but for your problem this is okay right?
i mean, for the mentioned array*fixedarray
*matrix
Kristoffer Carlsson
@KristofferC
Mar 26 2015 11:16
Yes, for me, it works fine
Could always just swap it out if there is a better combination of pointer cast magic to achieve the same thing
Simon
@SimonDanisch
Mar 26 2015 11:17
yes!
but this means, we could just define a convert for * right away...and it should be fast even...There was just one problem with the array <-> fsarray conversion
Vector3([1,2,3,4]) should preferably work... but Vector3 is abstract. I'm not really sure how to handle this, because you don't have abstract in the type hierarchy, which means we can't dispatch against it that easily... we would need triangular dispatch for that I think
we have to put in an if with two different function calls then.. because just calling FSA = Vector3 and then FSA{eltype(array)}, yields compiler errors...
Simon
@SimonDanisch
Mar 26 2015 11:24
function convert{FSA <: FixedArray}(::Type{FSA}, v::Array{T})
    if FSA.abstract
        unsafe_load(Ptr{FSA{eltype(v)}}}(pointer(v)))
    else
        unsafe_load(Ptr{FSA}(pointer(v)))
    end
end
this function can't be compiled, because it checks syntax statically before removing the dead code
(it seems)
it might be a bug, though
but that's the reason why I reverted to the trivial solution for now
Kristoffer Carlsson
@KristofferC
Mar 26 2015 11:53
Vector3 is not abstract, right?
Kristoffer Carlsson
@KristofferC
Mar 26 2015 11:59
Something like:
function convert{FSA <: FixedArray, T}(::Type{FSA}, v::Array{T})
        unsafe_load(convert(Ptr{FSA{T}}, pointer(v)))
end
julia> convert(Vector4, [1,2,3,4])
Vector4{Int64}(1,2,3,4)
Kristoffer Carlsson
@KristofferC
Mar 26 2015 12:30
julia> @time for i = 1:10^6
       convert(Vector4, [1,2,3,4])
       end
elapsed time: 0.052423517 seconds (76 MB allocated, 29.78% gc time in 3 pauses with 0 full sweep
Kristoffer Carlsson
@KristofferC
Mar 26 2015 12:35
Maybe there is something wrong with it but it seems very fast compared to the current
julia> @time for i = 1:10^6
       convert(Vector4, [1,2,3,4])
       end
elapsed time: 1.54273512 seconds (640 MB allocated, 3.74% gc time in 29 pauses with 0 full sweep)
Simon
@SimonDanisch
Mar 26 2015 12:42
Vector4 is abstract because the type parameter is missing... So your version wouldn't work for Vector4{Float32}, right?
Which it should... and I can't really differentiate between these two cases via multiple dispatch, as I'd need to know the concrete type... which is not known to fixedsizearrays
Kristoffer Carlsson
@KristofferC
Mar 26 2015 12:49
Not abstract in the sense
julia> Vector4.abstract
false
Simon
@SimonDanisch
Mar 26 2015 12:50
oh thanks, i dodn't realize this
Kristoffer Carlsson
@KristofferC
Mar 26 2015 12:51

So you want to be able to do:

 convert(Vector4{Float32}, [1,2,3,4])

?

You could always
julia> convert(Vector4, float32(rand(4)))
Vector4{Float32}(0.42428526f0,0.7441575f0,0.62335825f0,0.40632445f0)
Simon
@SimonDanisch
Mar 26 2015 12:53
yes
I want this more for the constructors, as they fall back to convert, which means I need to define less functions
so Vector4{Float32}([1,2,3,4]) should preferably work via the convert
this is especially important if you have type aliases, as they're typed
typealias Vec4f0 Vector4{Float32}
for example
Kristoffer Carlsson
@KristofferC
Mar 26 2015 12:55
Yeah, I see
Simon
@SimonDanisch
Mar 26 2015 12:56
function convertt{FSA <: FixedArray, T}(::Type{FSA}, v::Array{T})
    if isbits(FSA)
        concrete_convert(FSA, v)
    else
        abstract_convert(FSA, v)
    end
end
seems to work
wait.. I don't need to call out..
and it doesn't work :D
it simply doesn't compile...
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:03
Is it possible to just dispatch on everything FSA can dispatch on.. something like:
function convert{P, T, NDim, SIZE}(FSA::Type{FixedArray{T, NDim, SIZE}}, v::Array{P})
    unsafe_load(convert(Ptr{FSA{T, NDim, Size}}, pointer(T(v))))
end
Doesnt seem like it because
julia> Type{Vector3{Int}} <: Type{FixedArray{Int, 1, (3,)}}
false

julia> Vector3{Int} <: FixedArray{Int, 1, (3,)}
true
Simon
@SimonDanisch
Mar 26 2015 13:06
yup
that is the core of the problem\
all methods in fixedsizearrays must be of the form method{FSA <: FixedArray}(x::FSA)
we could open an issue... if the argument types are known, then all statements that could be eliminated shouldn't be evaluated
but I'm not sure if that's actually practical... as you still have to find the end of the if clause
and if you would allow invalid code inside the if, this would be impossible, i guess?
Simon
@SimonDanisch
Mar 26 2015 13:11
what an annoying problem :D
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:14
You get the ERROR: error compiling convertt: too many parameters for type Vector3?
Simon
@SimonDanisch
Mar 26 2015 13:14
yupp
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:19
Cant we dispatch on the value of isbits.. :P
or something like that
Maybe I am talking nonsense, I will test
Simon
@SimonDanisch
Mar 26 2015 13:28
stagedfunction convertt{FSA <: FixedArray, T}(::Type{FSA}, v::Array{T})
    isbits(FSA) && return :(unsafe_load(Ptr{FSA}(pointer(v))))
    :(unsafe_load(Ptr{FSA{eltype(v)}}(pointer(v))))
end
this works ;)
i had taht version befor, but with FSA.abstract, which is not working as intended as you pointed out
Simon
@SimonDanisch
Mar 26 2015 13:34
so it gave the same error like the other method :D
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:35
This also works :P

function convert1{FSA <: FixedArray, T}(::Type{FSA}, v::Array{T})
    convert2(Val{isbits(FSA)}, FSA, v)
end

function convert2{FSA <: FixedArray, T}(::Type{Val{true}}, ::Type{FSA}, v::Array{T})
    v_conv = map(eltype(FSA), v)
    unsafe_load(convert(Ptr{FSA}, pointer(v_conv)))
end

function convert2{FSA <: FixedArray, T}(::Type{Val{false}}, ::Type{FSA}, v::Array{T})
    unsafe_load(convert(Ptr{FSA{T}}, pointer(v)))
end
julia> convert1(Vector3, [1,2,3])
Vector3{Int64}(1,2,3)

julia> convert1(Vector3{Float32}, [1,2,3])
Vector3{Float32}(1.0f0,2.0f0,3.0f0)
Simon
@SimonDanisch
Mar 26 2015 13:37
the stagedfunction version seems to be faster though
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:38
I get this with it:
julia> convertt(Vector3{Float32}, [1,2,3])
Vector3{Float32}(1.0f-45,0.0f0,3.0f-45)
Simon
@SimonDanisch
Mar 26 2015 13:41
ah yeah, that's because I don't convert :D
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:41
Yeah, I was thinking that
At some point you have to do the actual conversion
Simon
@SimonDanisch
Mar 26 2015 13:43
yeah, that was just prototyping...;)
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:43
;)
My version is 5 times slower than yours even when it doesn't convert
Simon
@SimonDanisch
Mar 26 2015 13:46
yup... i tested both versions without conversion ;)
```Julia
stagedfunction convertt{FSA <: FixedArray, T}(::Type{FSA}, v::Array{T})
    conversion = isbits(FSA) ? :(unsafe_load(Ptr{FSA}(pointer(convert(Array{eltype(FSA)}, v))))) : :(unsafe_load(Ptr{FSA{eltype(v)}}(pointer(v))))
    quote
        checkbounds(v, $(size(FSA)...))
        $conversion
    end
end
so the final version should look something like this
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:51
very nice
Should this give an error?
julia> b = convertt(Vector3, [1,2,3,4])
GeometryTypes.Vector3{Int64}(1,2,3)
Simon
@SimonDanisch
Mar 26 2015 13:52
hmmm... yeah I've thought shortly about that :D
probably not
or this functionality should be exposed via a different apoi?
if you have that, you probably also ant to specify an offset
maybe we should throw an error for the conversion and rather overload the getindex and stindex functions for vector with fixedsizearraays as eltype
so maybe assert, that length(Vector) == length(FSA)
this way, you can also transform Vector{T} into e.g. matrix4x4
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:56
that's nice
Simon
@SimonDanisch
Mar 26 2015 13:57
yeah.. I wanted to put quite a few functions into arrays_of_fixedsize.jl
to do a lot of these little tricky operations
e.g have Vector3[1,2,3, 1,2,3, 1,2,3] produce Vector{Vector3}
and making it easy to get the x dimension of Vector{Vector3} and such things
Kristoffer Carlsson
@KristofferC
Mar 26 2015 13:59
yeah, guess that would be nice for your opengl stuff?
Simon
@SimonDanisch
Mar 26 2015 13:59
definitely... and for images, which are all Matrix{ColorValue <: FixedVector}
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:14

And for the other way, you want both

julia> convert(Vector{Float64}, Vector3([1,2,3]))
julia> convert(Vector, Vector3([1,2,3]))

?

This seems broken now
julia> convert(Vector{Float64}, Vector3([1,2,3]))
3-element Array{Int64,1}:
 1
 2
 3
Simon
@SimonDanisch
Mar 26 2015 14:15
yeaah... trying to implement that right now
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:22
a 4x4 FSA * a 4x4 Array should give a FSA or Array?
I guess an array?
Simon
@SimonDanisch
Mar 26 2015 14:23
good question
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:23
Since you can have a 4x4 FSR * 4 x 100000 Array
Doesnt make sense to put that in an FSA, you could always convert yourself after if you want to
In some way, an FSR is more pure than an Array and something pure * something dirty = dirty :P
Simon
@SimonDanisch
Mar 26 2015 14:24
yeah :D
makes sense i guess
well we probably should go for the solution, which allows to be faster in the end... so converting from array to fsa is cheaper, which is in favor of what you propose!
```Julia
function convertt{DA <: DenseArray, FSA <: FixedArray}(::Type{DA}, b::FSA)
pointer_to_array(Ptr{eltype(FSA)}(pointer_from_objref(b)), size(FSA))
end
function convertt{T,N, FSA <: FixedArray}(t::Type{Array{T, N}}, b::FSA)
convert(t, convertt(Array, b))
end
function convertt{DA <: DenseArray, FSA <: FixedArray}(::Type{DA}, b::FSA)
    pointer_to_array(Ptr{eltype(FSA)}(pointer_from_objref(b)), size(FSA))
end
function convertt{T,N, FSA <: FixedArray}(t::Type{Array{T, N}}, b::FSA)
    convert(t, convertt(Array, b))
end
this should do the trick!
missing: boundcheck and conserving the same shape
here we can actually decide, if the type parameter is missing ;)
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:28
julia> convertt(Vector{Float64}, Vector3([1,2,3]))
3-element Array{Float64,1}:
 1.39951e14
 1.0       
 2.0
Simon
@SimonDanisch
Mar 26 2015 14:28
whps
ah yeah forgot the offset
ah haha, and bound check and preserving the siye is not reallz needed!
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:31
Because the Vector will always be the correct shape?
or Array I mean
Simon
@SimonDanisch
Mar 26 2015 14:31
and i didn't forget the offset! I think the behavior of pointer_from_objref has changed in the newer versions of julia?
at least its working fine for me
yeah pointer to array together with size(FSA) is all we need ;)
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:33
This works for you?
 convertt(Vector{Float64}, Vector3([1,2,3]))
Simon
@SimonDanisch
Mar 26 2015 14:37
yes
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:37
you are using what is currently on master for FSA?
Simon
@SimonDanisch
Mar 26 2015 14:39
let me make sure
i don't think so...
I delete the old conversion methods, inserted the new ones and get this:
convert(Vector{Float64},Vector3{Float32}([1,2,3])) = [1.0,2.0,3.0]
convert(Vector{Float64},Vector3([1,2,3])) = [1.0,2.0,3.0]
convert(Vector,Vector3{Float32}([1,2,3])) = Float32[1.0f0,2.0f0,3.0f0]
convert(Vector{Float64},Vector3{Float32}([1,2,3])) = [1.0,2.0,3.0]
convert(Vector{Float64},Vector3([1,2,3])) = [1.0,2.0,3.0]
convert(Vector,Vector3{Float32}([1,2,3])) = Float32[1.0f0,2.0f0,3.0f0]
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:43
This is when I am getting weird stuff
julia> convertt(Vector,Vector3{Float32}([1,2,3]))
3-element Array{Float32,1}:
 7.48003    
 4.58225e-41
 1.0        

julia> convertt(Vector,Vector3{Float64}([1,2,3]))
3-element Array{Float64,1}:
 6.93898e-310
 1.0         
 2.0         

julia> convertt(Vector,Vector3{Int8}([1,2,3]))
3-element Array{Int8,1}:
 -48
  -4
  -4
How old is your julia?
Mine is quite old now so maybe it has changed like you said
Simon
@SimonDanisch
Mar 26 2015 14:44
(4 days old master)
yeah I think someone wnted to change that
removing the offset from the pointer returned by fromobjref
guess the PR got through
ah snap, and this seems not to work for mutable vectors...
the error is surprising though :D
its again too much parameters...
ah yeah..and its not a surprise
type is never isbits == true
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:47
yeah
true
what's the check then... :P
Simon
@SimonDanisch
Mar 26 2015 14:49
good question...
guess it will and up to be relatively ugly with Type.parameters
all(isdefined, FSA.parameters)
probably
aaaaah no! =)
totally simple!
isleaftype
:D
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:53
Ofc!
haha :D
Simon
@SimonDanisch
Mar 26 2015 14:53
I always forget about that function, because its such an unintuitive name
works with mutable and immutable =) good... I thought for a second the pointer stuff just works with isbits, which would not allow for mutable vectors
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:54
Cool!
Guess im compiling a new julia then, so the pointerstuff works for me too :P
on my work laptop.. inside my linux VM zzzz so slow
Simon
@SimonDanisch
Mar 26 2015 14:56
yeah try it out! bad thing is, we can't really write code for a specific commit :D i thing we only have if v"3.1" vs v"4.0"
guess we just have to warn about this in the readme
think.. damn I'm writing really sloppy today :D
Kristoffer Carlsson
@KristofferC
Mar 26 2015 14:59
You could do something ugly like look at the date of the commit from versioninfo()
Simon
@SimonDanisch
Mar 26 2015 14:59
haha yeah
Kristoffer Carlsson
@KristofferC
Mar 26 2015 15:03
this is what versioninfo use
GIT_VERSION_INFO.date_string
Base.GIT_VERSION_INFO.date_string
Kristoffer Carlsson
@KristofferC
Mar 26 2015 15:16
julia> date_str = Base.GIT_VERSION_INFO.date_string
"2015-03-26 06:44 UTC"

julia> (Date(date_str[1:10]) - today()).value
0
rofl
put that in a const and check if < 0 maybe
Kristoffer Carlsson
@KristofferC
Mar 26 2015 15:55
To get all the .x in a Vector{Vector3} for example, couldn't you just return a strided array pointing to the first x and then have the stride same as the length of the FSA?
Simon
@SimonDanisch
Mar 26 2015 15:57
Yeah probably! I haven't looked into how to make this fast yet ;)