@rgrinberg I was thinking about your problem a little more. There is an alternative to keeping references to all subscriptions. Namely passing in a cancellation token (well that's C# stuff, I'm sure Java has something similar). That way you can still cancel your work early if needed without the need to keep track of what observable finished or not.
Or store your Subscriptions in a weak reference. Anyone here able to confirm that's safe?