it appears that the accuracy is reduced by nearly the exact amount of the noise (i.e. 5% noise leads to ~95% accuracy, 10% equates to ~90% accuracy and so on)
This might change depending on the estimator used, other than that seems reasonable (I assume noise is generated from a normal distribution)
but how is it testing? and then how is it training?
The data from the strem is first used to get a prediction (test(X)) and the predicted value is compraed against the true value to estimate track the performance of the estimator. Then the same sample is used to train the estimator (train(X, y))
After training, can I send it another stream to be processed using the "trained" classifier?
Yes, there is a way to run another evaluation task with a previously trained model. You must first make sure to set the parameter restart_stream=False in EvaluatePrequential. This way the firsst evaluate call will train the model without restarting it at the end. If you call again evaluate with a new stream and the same model, the model will continue learning.
@jacobmontiel Jacob, if I don't want the model to continue learning, should I set restart_stream=True or False?
This parameter is not intended to be used like that. This parameter indicates if the model should be re-started (True) or not (False) at the end of the evaluation. If restart_stream=True it means that after the evaluation the model instance will remain in the last status from the evaluation. You can either continue treaining or use it only to get predictions. That is up to you to define (and code). However, as mentioned earlier, EvaluatePrequential always performs both tessting and training.
@jacobmontiel Hi Jacob, I had another problem. I wanted to predict all the target values in 20 time steps after time t, and when I used EvaluatePrequentia, the parameters in it didn't seem to do the job.I can only do this by changing the source code? I hope you can give me some help with this problem. Thank you
This is not currently supported by the EvaluatePrequential, we are working on a new feature for this case, but it might take some time until it is available. In the meantime the best option is, as you mention, to manually implement it. You can take a look into PR #222 for reference :-)
^M #################### [100%] [22165.92s]
Separating 5000 holdout samples.
Expected 2D array, got 1D array instead:
array=[].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a
single sample.
Processed samples: 1000000
Mean performance:
AdaptiveRandomForest-Holdout - Accuracy : 0.7894
AdaptiveRandomForest-Holdout - Training time (s) : 20975.01
AdaptiveRandomForest-Holdout - Testing time (s) : 278.86
AdaptiveRandomForest-Holdout - Total time (s) : 21253.87
AdaptiveRandomForest-Holdout - Size (kB) : 445215.2520