.NET based ecosystem of open-source software for mathematics, science, and engineering.
Source=System.Linq
StackTrace:
at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)
at System.Linq.Enumerable.TryGetLastTSource
at System.Linq.Enumerable.LastTSource
at Tensorflow.Keras.Layers.Dense.build(Tensors inputs)
HResult=0x80004003
Message=Value cannot be null. (Parameter 'first')
Source=System.Linq
StackTrace:
at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument) in //src/libraries/System.Linq/src/System/Linq/ThrowHelper.cs:line 12
at System.Linq.Enumerable.ZipTFirst,TSecond,TResult in //src/libraries/System.Linq/src/System/Linq/Zip.cs:line 14
at Tensorflow.Keras.Engine.Functional.run_internal_graph(Tensors inputs, Boolean training, Tensors mask)
at Tensorflow.Keras.Engine.Functional.Call(Tensors inputs, Tensor state, Nullable
1 training)
at Tensorflow.Keras.Engine.Layer.<>c__DisplayClass1_0.<Apply>b__0(NameScope scope)
at Tensorflow.Binding.tf_with[T](T py, Action1 action)at Tensorflow.Keras.Engine.Layer.Apply(Tensors inputs, Tensor state, Boolean training)
at Tensorflow.Keras.Engine.Model.train_step(Tensor x, Tensor y)
at Tensorflow.Keras.Engine.Model.train_step_function(OwnedIterator iterator)
at Tensorflow.Keras.Engine.Model.FitInternal(Int32 epochs, Int32 verbose)
at Tensorflow.Keras.Engine.Model.fit(NDArray x, NDArray y, Int32 batch_size, Int32 epochs, Int32 verbose, Single validation_split, Boolean shuffle, Int32 initial_epoch, Int32 max_queue_size, Int32 workers, Boolean use_multiprocessing)
at MyraTest3.Chatbot.Model.TrainModel(Int32[,] xTrain, Int32[] yTrain)
@Oceania2018 I'm new to neural networks and tensorflow. I'm not sure what I would unit test.
Would this help?:
at Tensorflow.Gradients.math_grad._SumGrad(Operation op, Tensor[] grads)
at Tensorflow.Gradients.math_grad._MeanGrad(Operation op, Tensor[] grads)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
at Tensorflow.ops.<>c_DisplayClass1_1.<RegisterFromAssembly>b_3(Operation oper, Tensor[] out_grads)
at Tensorflow.gradients_util.<>c_DisplayClass0_2.<_GradientsHelper>b_6(NameScope scope1)
at Tensorflow.Binding.tf_withT
at Tensorflow.gradients_util.<>c_DisplayClass0_0.<_GradientsHelper>b_0(NameScope scope)
at Tensorflow.Binding.tf_withT
at Tensorflow.gradients_util._GradientsHelper(Tensor[] ys, Tensor[] xs, Tensor[] grad_ys, String name, Boolean colocate_gradients_with_ops, Boolean gate_gradients, Int32 aggregation_method, Tensor[] stop_gradients, Graph src_graph)
at Tensorflow.Optimizer.compute_gradients(Tensor loss, List
1 var_list, Nullable1 aggregation_method, GateGradientType gate_gradients, Boolean colocate_gradients_with_ops, Tensor grad_loss)at Tensorflow.Optimizer.minimize(Tensor loss, IVariableV1 global_step, List
1 var_list, GateGradientType gate_gradients, Nullable1 aggregation_method, Boolean colocate_gradients_with_ops, String name, Tensor grad_loss)
{
Here's a fun one. Conv2DTranspose has default value for
bool use_bias = true,
However in the actual concrete class:
if (use_bias)
throw new NotImplementedException("");
case int[] shape2:
if (shape.ndim != shape2.Length)
return false;
int[] newDims;
newDims = new int[shape.dims.Length];
for (int i = 0; i < shape.dims.Length; i++)
{
newDims[i] = (int)shape.dims[i];
}
return Enumerable.SequenceEqual(newDims, shape2);
@Craigjw @SuperDaveOsbourne @AndreasHogstrandUltromics_gitlab Sadly after over an year of banging my head on the wall I've finally given up and moved across to using TorchSharp. Unfortunately for non trivial models there is some kind of a bug in the C++ build of Tensorflow itself. Which is not there in the Python Version.
What I've found is that regardless of which version you use and I have tried TF1.3, TF 1.15.5, TF 2.3, TF 2.6 The C++ build of Tensorflow has some kind of weird bug in it. I've tried both the nuget packaged versions provided here. As well as compiling from source using bazel. It makes no difference. The bug is "silent corruption of data". What this means is when building non trivial models (More complex than the samples or the unit tests) or in other words deep networks of more than 2 or 3 layers your model will train upto a certain point. After which it will not train anymore. You can try with Adam, ADAGrad, RMSProp or whatever you want. Your model will not train beyond a certain level. From what I can see it is related to floating point precision of the weight updates from gradients.
@Craigjw May I suggest just building it all in a console app. Much easier and a lot less pain. Also don't bother with the Eager mode if your planning on using GPU. Actually don't bother either way. Which unfortunately means that all of Keras is out the window. Here is the reason. When running in eager mode you have to use Gradient Tape. While the Python version of Gradient Tape might be computationally efficient, the Dot net translated version is anything But. Since most of this framework is tested on CPU rather than GPU this issue is hidden and not that obvious. However if your planning on doing any real training you will need to use GPU and lemme put it this way. I'm running a threadripper with 4 X 3090 cards. The Gradient Tape implementation is single threaded and I can barely manage to get one GPU at just over 2% Cuda utilization.
Your alternative is to use Graph mode which does work quite well. This however means Tensorflow Native which is really not that much of a big deal. Most of the operations are fully implemented except in the area of convolutions. Conv2D is implemented and does work. However Conv2dTranspose is not implemented to any functional level. Having said that its also not that much of a big deal cause you can get close to the same result using dilated conv2D followed by a standard dense to expand the shape. I've tested this approach and it does work decently.
@AndreasHogstrandUltromics_gitlab May I suggest using NetMQ for your transport along with MessagePack and MessagePack.Annotations for serialization. I can get near wire speed (1Gbps) serialization of floats from multiple agents to the central Neural Network (Multi Agent reinforcement learning scenario). Note: the Numpy implementation in Tensorflow.Net is extremely performant and blazing fast for converting float shape structures. Much faster than the methods used in TorshSharp so I'm continuing to use the Numpy implementation even through I'm now using TorchSharp as the neural network backend
In my use case something is not working right and some obscure bug in Tensorflow is getting triggered by my use case and therefore the model wont train. Your case may be different.
Or if you really want Eager Mode. TorchSharp is by default Eager and there is no Graph mode vs Eager as such.
The differences in implementation coverage is not huge at this stage. TorchSharp has pretty much 100% coverage of Pytorch functionbality. Tensorflow.Net has high coverage of the Python version, however a few areas are missing such as with convolutions only conv2d is fully implemented.
If your trying to port across some python models that have been done in Tensorflow and Convolution Transpose is not in the picture then your better off with Tensorflow.Net. On the other hand if the models you are porting across are Torch based then TorchSharp will be better.
So in summary: Lets say your trying to implement say a GAN or a VQVAE and your not an expert in designing these so copying network structures from tutorials or other Python implementations. Your going to find a ton more Tensorflow 1.X examples done in graph mode that you can build upon. On the other hand trying to find examples of that in Pythorch you wont find as many.
ALso tensorflow.Net Graph mode works like a breeze and is slightly more performant than TorchSharp. TorchSharp gives you more flexibility in some areas like say having parts of your model on the GPU and parts on CPU. Think of Graph mode vs Eager as. Precompiled GPU kernels vs on the fly GPU kernels. Of course pre-compiled will out perform on the fly.
REgardless of what you choose. If your planning to use Numpy style operations as well. The implementation of Numpy in Tensorflow.Net is Extremely high performance for reshaping data.
First the helper function that does the equivalent of Keras Dense:
public static Tensor FullyConnectedDense(Tensor _Input, int _Num_Units, TF_DataType dataType = TF_DataType.TF_FLOAT, bool Trainable = true, bool Normalize = false,
IInitializer InitializeVariant = null, string _Name = null)
{
if (InitializeVariant == null)
{
InitializeVariant = tf.truncated_normal_initializer(0, 1);
}
int in_dim = (int)_Input.shape[1];
string WeightsName;
string BiasName;
WeightsName = _Name.ConcatIfNotNullOrEmptyElseNull(@"_Weights");
BiasName = _Name.ConcatIfNotNullOrEmptyElseNull(@"_Bias");
if (in_dim == 0)
{
in_dim = -1;
}
ResourceVariable Weights;
ResourceVariable Bias;
Weights = tf.Variable(
InitializeVariant,
name: WeightsName,
dtype: dataType,
shape: new int[2] { in_dim, _Num_Units },
trainable: Trainable,
validate_shape: false);
Bias = tf.Variable(
InitializeVariant,
name: BiasName,
dtype: dataType,
shape: new int[1] { _Num_Units },
trainable: Trainable,
validate_shape: false);
Tensor layer = tf.matmul(_Input, Weights) + Bias;
return layer;
}