do you want to implement an algorithm for an existing backend?
give more feat to an existing backend
give more feat to an existing backend
for example, the linear layer has:
impl<B: IBackend + LayerOps<f32>> ComputeInputGradient<f32, B> for Linear {
fn compute_input_gradient(&self,
backend: &B,
weights_data: &[&SharedTensor<f32>],
output_data: &[&SharedTensor<f32>],
output_gradients: &[&SharedTensor<f32>],
input_data: &[&SharedTensor<f32>],
input_gradients: &mut [&mut SharedTensor<f32>]) {
// Gradient with respect to input data
backend.gemm(&self.one,
Transpose::NoTrans,
output_gradients[0],
Transpose::NoTrans,
weights_data[0],
&self.zero,
input_gradients[0])
.unwrap();
}
}
impl<B: IBackend + LayerOps<f32>> ComputeParametersGradient<f32, B> for Linear {
fn compute_parameters_gradient(&self,
backend: &B,
output_data: &[&SharedTensor<f32>],
output_gradients: &[&SharedTensor<f32>],
input_data: &[&SharedTensor<f32>],
parameters_gradients: &mut [&mut SharedTensor<f32>]) {
// gradient w.r.t. weights
backend.gemm(&self.one,
Transpose::Trans,
output_gradients[0],
Transpose::NoTrans,
input_data[0],
&self.zero,
parameters_gradients[0])
.unwrap();
}
}I am confused on computing gradients here since there is no derivatives of activation functions
it other is regarding the input sample given
iirc
@onFireForGod_gitlab
the activation functions are modeled as separate layers in juice
as such there is no point in having them interleaved with this level
(it mmight make sense for performance reasons, but that's yet to be benched and decided)
specifically, if tensors could have typed dimensions
like
Tensor2<Const<2>, Dynamic>
so each dimension could be a dynamic or constant dimension
then all operations which reshape tensors in in ways where the output size can be determined statically we could have statically verified
I'd wait another week before you start so I can actually get the one-repo integration done
Do you have a link to the const generics RFC?
The only issue that might pop up, is the fact that the backends tend to work with fixed dimensions, as such it might be nice from the interface pov, but not necessarily make it any easier to implement any layer on top of an existing backend
I am more interested in the user improvements than the backend improvements
from that perspective, wrapping coaster in an a const generics api might be sufficient as well
tracking RFC for const generics: rust-lang/rust#44580
unfortunately, it still doesnt look ready enough for something like this, even for initial implementation
when its ready I might also have other interests at the time, but I definitely put this as a high priority because having a pure-rust data-parallelism abstraction over CUDA, opencl, and native is important
and extending the API to support things like knowing the output dimensions of matrix multiplications at compile time is critical
@vadixidav I understand your point and I for sure would prefer it from an end user API perspective :) I am just trying to convey that this is some heavy lifting and that the internal representation on a per backend basis is and will not be statically determined - CPU based execution might be the exception here
*be able to be determined
it can continue to be dynamic
so it wont affect it much
unfortunately what landed in nightly is very buggy
so I dont think its useful enough yet for me to attempt a refactor