Interesting new research looking at the performance of polynomial regression versus neural networks https://arxiv.org/abs/1806.06850
. Although they claim that just polynomial regression beats neural networks or are just as good, I'm skeptical as they didn't appear to optimize the neural networks they were using. But with the polynomial regression, they modified it with different number of coefficients and even resorted to PCA regression (which performs PCA to get largest variance components and then runs the regression). Interesting paper nonetheless getting at the whole "black box" of neural networks.