Question 1: consistency between MacOS and Linux. Removing the watched directory (an obvious corner-case) behaves differently on MacOS and Linux. What is the expectation? Maybe it should be documented as undefined behavior?
[Sample program] (https://github.com/utgheith/watch/blob/master/src/main/scala/main.scala)
event data/a data/b
Here is a stress test that leads to failure in 3 native environments: MacOS, Ubuntu/ext4, Ubuntu/ZFS. There seem to be 2 separate issues:
(1) a race condition that causes the watch thread to die with FileNotFoundException.
(2) events get reordered leading to premature watch cancelation on some subdirectories
WatchServiceonly drops directory
DELETEevents (at least in the stress tests I tried)
DELETEevents for sub directories of moved directories:
Once the call to watch returns, onEvent is guaranteed to receive a an event containing the path for: - Every file or folder that gets created, deleted, updated or moved within the watched folders - For copied or moved folders, the path of the new folder as well as every file or folder within it. - For deleted or moved folders, the root folder which was deleted/moved, but without the paths of every file that was within it at the original location
I apologize for all the spam but here is a simple test that shows that the problem is in
os.watch is likely masking some of those issues in its implementation but not all.
It seems that
readme.md is using travis-ci badges instead of the github actions ones. Clicking on the build status goes back to a January build (telling me that the switch occurred around this time). Tried to submit a pull request that fixes that but it gets stuck and never runs the tests. This seems to be related to an issue that I don't fully understand but requires special privileges to deal with anyway. Here is a link:
os.watchand I want to know your thoughts on them.
os.watchis the sort of thing that requires stress testing. Unit testing is still important obviously. Do you have your own stress tests? If not, is there value in developing some and running them as part of CI?
OVERFLOW(silently drop events, force the caller to re-examine the world, ...).
inotifyuses a finite system-wide queue for notifications and there is no guarantee that it will not overflow.
Same test as before but uses
inotify (not directly but by spawning a process running
inotifywatch). Works reliably on the same test that was failing before. This strongly hints that using
inotify directly will be more reliable. Also
inotify is explicit about how it handles moves (move_to, move_from, move_self, etc)
inotify-based implementation too, if you don't mind wrangling some JNA FFI code. That's what we ended up doing for the OS-X backend anyway so it's not unprecedented
Watcherimplementation (com-lihaoyi/os-lib#70). It's still rough around the edges and didn't have a change to stress-test it but it already passes the existing
os.watchtests reliably (even on Docker). Would be great if you took look at the overall design.
scalacheckas a dependency? It makes working with such tests easier.