Hi @memora0101, you just need to add np.argmax()
over axis =1
to your prediction output. So, it returns if 0 index (Angry) is taking higher probability or the 1 index (Happy).
for sent in train_x[60:70]:
inputs = np.array(sent_to_tensor(sent, vocab_dict=Vocab))
inputs = inputs[None, :]
#### This get you 16 (your used batch size) input and and 16 target. Not needed.
# example_input = next(val_generator(batch_size=batch_size, shuffle=True))
# tmp_inputs, tmp_targets, tmp_example_weights = example_input
tmp_pred = training_loop.eval_model(inputs)
print(f'example input str: {sent}')
print(f'example input array:{inputs}')
print(f'Model returned sentiment probabilities: {tmp_pred.argmax(axis=1)}')
print('_________')
part of the output:
example input str: Great, Iâm glad I could help you!! Thanks for being a customer of ToyCityInc. â We hope your daughter has a very happy birthday!
example input array:[[ 37 46 47 48 12 71 9 96 249 250 251 47 252 177 253 202 254]]
Model returned sentiment probabilities: [1]
example input str: I appreciate it!
example input array:[[173]]
Model returned sentiment probabilities: [1]
example input str: I have a problem with one of the products that I have purchased at your store
example input array:[[244 107 255 256 29]]
Model returned sentiment probabilities: [0]
example input str: It is an iphone 10, it has been a week about now and the phone has started to overheat.
example input array:[[257 258 259 78 260 261]]
Model returned sentiment probabilities: [0]
If you want to work with eval:
for sent in range(5):
example_input = next(val_generator(batch_size=batch_size, shuffle=True))
tmp_inputs, tmp_targets, tmp_example_weights = example_input
tmp_pred = training_loop.eval_model(tmp_inputs)
print(f'example input array:{tmp_pred.argmax(axis=1)}')
print(f'Model returned sentiment probabilities: {tmp_targets}')
print('_________')
outputs:
example input array:[0 1 1 1 0 0 1 1 1 0 1 0 0 0 0 1]
Model returned sentiment probabilities: [1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0]
example input array:[1 0 0 1 1 1 1 1 0 0 0 0 0 0 0 1]
Model returned sentiment probabilities: [1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0]
example input array:[0 0 1 1 1 0 1 1 0 0 0 0 0 1 1 0]
Model returned sentiment probabilities: [1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0]
example input array:[1 0 1 1 1 1 1 1 0 1 0 0 0 1 1 0]
Model returned sentiment probabilities: [1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0]
example input array:[1 1 0 1 1 1 1 0 0 0 0 1 0 0 1 0]
Model returned sentiment probabilities: [1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0]
The model needs more data as it fits on inputs and fail on validation (overfits) or reduce the number of steps (I got 80% with 20 steps). Also, you can try to add layers to the model until you get the best results with this small amount of data.
I also get errors when trying to run the cell which uses a Beam Search in this colab:
beam_decoder = Search(
trax.models.Reformer, model_weights,
beam_size=4,
alpha=0.6, # For length normalization, set to 0.6 following Vaswani et al.
eos_id=1, # The stop token has id 1 in the vocabulary we use.
max_decode_len=146,
)
I get the following error:
AttributeError Traceback (most recent call last)
in ()
5 alpha=0.6, # For length normalization, set to 0.6 following Vaswani et al.
6 eos_id=1, # The stop token has id 1 in the vocabulary we use.
----> 7 max_decode_len=146,
8 )
/usr/local/lib/python3.7/dist-packages/trax/models/beam_search.py in init(self, model, weights, max_decode_len, beam_size, temperature, alpha, eos_id)
469 # Work around a jax error
470 # Ref: google/jax#1919 (comment)
--> 471 jax_partial_eval._thread_local_state.remat = True # pylint: disable=protected-access
472
473 def _get_initial_state(self, inputs, targets_prefix, batch_size):
AttributeError: module 'jax.interpreters.partial_eval' has no attribute '_thread_local_state'
# __init__ is implemented for you
def __init__(self, n_units, init_stdev=0.1):
# Set the number of units in this layer
self._n_units = n_units
self._init_stdev = init_stdev
# Please implement 'forward()'
def forward(self, x):
# Matrix multiply x and the weight matrix
dense = np.dot(x,self.weights)
return dense
# init_weights
def init_weights_and_state(self, input_signature, random_key):
# The input_signature has a .shape attribute that gives the shape as a tuple
input_shape = (input_signature.shape[-1], self._n_units)
# Generate the weight matrix from a normal distribution,
# and standard deviation of 'stdev'
w = trax.fastmath.random.normal(key=random_key, shape=input_shape) * self._init_stdev
self.weights = w
return self.weights
Returns:
parameters -- python dictionary containing your parameters "W1", "b1", ..., "WL", "bL":
W1 -- weight matrix of shape (layers_dims[1], layers_dims[0])
b1 -- bias vector of shape (layers_dims[1], 1)
...
WL -- weight matrix of shape (layers_dims[L], layers_dims[L-1])
bL -- bias vector of shape (layers_dims[L], 1)
"""
np.random.seed(3)
parameters = {}
L = len(layers_dims) - 1 # integer representing the number of layers
for l in range(1, L + 1):
### START CODE HERE ### (≈ 2 lines of code)
parameters['W' + str(l)] = np.random.randn(layers_dims[l],layers_dims[l-1])*np.sqrt(2./layers_dims[l-1])
parameters['b' + str(l)] = np.zeros((layers_dims[l],1))
### END CODE HERE ###
return parameters
Hi, I am very new to Trax and I am trying to make the Chatbot Coursera course work:
I am getting a layer error, my imports are
!pip install trax==1.3.5
!pip install t5==0.8.1
LayerError Traceback (most recent call last)
<ipython-input-46-83df29050cce> in <module>()
----> 1 model.init_from_file('/content/drive/My Drive/work/chatbot_model1.pkl.gz', weights_only=True, input_signature=shape11)
2
3 STARTING_STATE = model.state
9 frames
/usr/local/lib/python3.7/dist-packages/trax/layers/base.py in init(self, input_signature, rng, use_cache)
286 name, trace = self._name, _short_traceback(skip=3)
287 raise LayerError(name, 'init', self._caller,
--> 288 input_signature, trace) from None
289
290 def init_from_file(self, file_name, weights_only=False, input_signature=None):
LayerError: Exception passing through layer Serial (in init):
layer created in file [...]/models/reformer/reformer.py, line 236
layer input shapes: Traced<ShapedArray(int32[1,1])>with<DynamicJaxprTrace(level=1/0)>
File [...]/trax/layers/combinators.py, line 104, in init_weights_and_state
sublayer.init(inputs, use_cache=True))
LayerError: Exception passing through layer ReversibleSerial (in init):
layer created in file [...]/models/reformer/reformer.py, line 229
layer input shapes: (ShapeDtype{shape:(1, 1, 512), dtype:float32}, ShapeDtype{shape:(1, 1, 512), dtype:float32})
File [...]/trax/layers/combinators.py, line 105, in init_weights_andstate
outputs, = sublayer._forward_abstract(inputs)
LayerError: Exception passing through layer ReversibleHalfResidual (in _forward_abstract):
layer created in file [...]/models/reformer/reformer.py, line 113
layer input shapes: (ShapeDtype{shape:(1, 1, 512), dtype:float32}, ShapeDtype{shape:(1, 1, 512), dtype:float32})
File [...]/jax/interpreters/partial_eval.py, line 404, in abstract_evalfun , avalsout, = trace_to_jaxpr_dynamic(lu.wrap_init(fun, params), avals)
File [...]/jax/interpreters/partial_eval.py, line 1178, in trace_to_jaxpr_dynamic
jaxpr, out_avals, consts = trace_to_subjaxpr_dynamic(fun, main, in_avals)
File [...]/jax/interpreters/partial_eval.py, line 1188, in trace_to_subjaxpr_dynamic
ans = fun.call_wrapped(*in_tracers)
File [...]/dist-packages/jax/linear_util.py, line 166, in call_wrapped
ans = self.f(args, *dict(self.params, kwargs))
File [...]/dist-packages/jax/linear_util.py, line 166, in call_wrapped
ans = self.f(args, *dict(self.params, kwargs))
LayerError: Exception passing through layer ReversibleHalfResidual (in pure_fn):
layer created in file [...]/models/reformer/reformer.py, line 113
layer input shapes: (ShapeDtype{shape:(1, 1, 512), dtype:float32}, ShapeDtype{shape:(1, 1, 512), dtype:float32})
File [...]/trax/layers/base.py, line 588, in _do_custom_gradients
do_forward = fastmath.custom_grad(do_forward_vjp, _do_forward)
File [...]/trax/fastmath/ops.py, line 260, in custom_grad
return backend()'custom_grad'
File [...]/trax/fastmath/jax.py, line 158, in _customgrad
f = jax.custom_transforms(f_original)
AttributeError: module 'jax' has no attribute 'custom_transforms'
trax
has support for building "packed" version of a text dataset, for better performance on TPU. For example, this is done in a flax example: https://github.com/google/flax/blob/master/examples/wmt/input_pipeline.py#L72.