statistical models have errors, which seems normal. Have you seen that this model is not trained by us, and followed the pointer we give for it ? http://www.openu.ac.il/home/hassner/projects/cnn_agegender/
You can train your own model if you'd like to. However the model we point to is used by many with fair results.