Hm... I have to confess that I'm not that familiar with the internal calculation of the different coordinate systems. But I think there could appear problems when transforming coordinates from one coordinate system to another and then once more. the rotation must not affect the second transformation because the point was already rotated in the first transformation. So... rotation of the coordinates should somehow be done once. Maybe indeed a flag "isRotated" would be a solution?!
I think the rotation should always be applied when going from element to viewport and viewport to element. Adding the logic inside pointFromPixel and pixelFromPoint should do the trick because all the elementToViewport and windowToViewport are using those methods underneath IIRC.
There might be some problems with the corresponding methods working on rectangles though.