@iangilman@avandecreme first of all: thanks for your efforts in this issue. It's quite a tricky one. Maybe we should go for an unit-tests on this thing where one or more Point objects get transformed into all the coordinate systems (more than one time), one starting with web-coordinates (Web <-> Viewport <-> Image), another one on viewport coords (Viewport <-> Image). the latter one should deliver always the same coordinates since not affected by the "web-view" system. The first should apply rotation only on the step Web <-> Viewport (in both ways). Maybe this example could be a start for this: https://openseadragon.github.io/examples/viewport-coordinates/
One thing to keep in mind is the vector representation of width and height for rectangles. (Now I'll try to revive my poor math knowledge :) ) Since it's a vector, rotation will affect this one as well, leading to wrong width/height. If using two points for spanning a rectangle, and rotating both points will preserve width/height information (which then has to be calculated).