I've had a minimized test case for several days, just trying to grind though HW.
Turns out it's totally a race condition inside IDA.
I'm not in a hurry to send it off so we can get a patch for IDA 6.10. Rather, I'll just try to hack around it as pleasantly as possible so we don't crash. At least now I know exactly what causes it and how to avoid it.