It was network related traffic, if the sketch did nothing other than be connected, I never had much of a problem even with higher interrupt disabling times. The research I did is pretty old now (2 years?). This was all the understanding I was trying to do with my NeoPixelbus and bitbang.
It seems to have gotten worse as the SDK versions have progressed over the years as well. Now, I'm noticing severe packet loss when they're disabled for ~700us which didn't happen before.
Much over that and it starts the network crash / watchdog reset cycle