Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 21 20:01
    ddfreyne commented #1483
  • Jan 21 19:59
    da2x commented #1483
  • Jan 21 19:57
    da2x opened #1483
  • Jan 21 17:24
    joerggollnick commented #1479
  • Jan 21 15:27
    da2x opened #1482
  • Jan 21 07:59
    ddfreyne opened #1481
  • Jan 21 07:58

    ddfreyne on rm-safe-level

    Remove safe_level support in ER… (compare)

  • Jan 21 07:57

    ddfreyne on rm-safe-level

    Remove safe_level support in ERB (compare)

  • Jan 21 00:09
    bobbens edited #1475
  • Jan 20 22:29
    da2x commented #1475
  • Jan 10 07:21
    cdlm commented #1480
  • Jan 10 01:53
    cup edited #1480
  • Jan 10 01:52
    cup opened #1480
  • Jan 08 17:06
    blinry commented #1471
  • Jan 08 11:14
    iay opened #1479
  • Dec 26 2019 10:14
    ddfreyne labeled #1477
  • Dec 26 2019 10:14
    ddfreyne labeled #1478
  • Dec 26 2019 10:14
    ddfreyne closed #1478
  • Dec 26 2019 10:14
    ddfreyne commented #1478
  • Dec 26 2019 03:41
    cup opened #1478
Denis Defreyne
@ddfreyne
@moll It uses a temporary file
Andri Möll
@moll
That would explain why it's triggering LiveReload before it's finished.
If that temporary file is placed in the same output directory.
Nanoc could place it in tmp and rename from there. That'd fix early reloads. Better though probably to hook into LiveReload and trigger it once compilation is finished, but I don't know whether the LiveReload library permits that.
Denis Defreyne
@ddfreyne
Hmm no, it’s written into /tmp/* and then moved into place
Andri Möll
@moll

Mkay, I'll run inotifywait next time I'm working on my site and confirm what Nanoc's doing. Perhaps a directory gets touched (by the kernel) first, triggers inotify and LiveReload catches that.

You're claiming Nanoc's renaming files over existing files, right? It's not a delete-then-move action?

Presumably at some point the output file has to be missing entirely, otherwise the web server wouldn't report a 404.
Denis Defreyne
@ddfreyne
Hmmm, it should be in place!
This is pretty weird.
Andri Möll
@moll

Curiously, I'm seeing a delete happen ...

19:51:36 DELETE public/articles/foo/index.html
19:51:36 CREATE public/articles/foo/index.html
19:51:36 MODIFY public/articles/foo/index.html
19:51:36 CLOSE_WRITE,CLOSE public/articles/foo/index.html

This with:

inotifywait --recursive --monitor public --format "%T %e %w%f" --timefmt "%T" -e modify -e attrib -e close_write -e moved_to -e moved_from -e move -e move_self -e create -e delete -e delete_self
And that triggered a race which LiveReload won, reloading before the create and coming up with "File not found".
Andri Möll
@moll
I think I know why this is happening. Nanoc's not using renaming, it's using hard linking.
https://github.com/nanoc/nanoc/blob/4.10.1/nanoc/lib/nanoc/base/services/item_rep_writer.rb#L59 calls FileUtils.ln with force: true. Force, as per Ruby's implementation, just unlinks it before creating a new link:
https://ruby-doc.org/stdlib-2.4.1/libdoc/fileutils/rdoc/FileUtils.html#method-c-ln
That explains the delete inotify event before create and explains the race condition.
Andri Möll
@moll
Why's Nanoc using linking or copying there in the first place? Wouldn't it be fine to move the temporary file? I suppose it's because you were trying to optimize by hard linking binary files.
..given that in one branch of the code temp_path is set to not a true temporary path, but the actual filename from the content directory.
Denis Defreyne
@ddfreyne
@moll The hardlinking’s purpose is to improve disk usage
Moving stuff in place can work under certain conditions, but not all
(e.g. the input file in content/ might be hardlinked to what is in output/!)
Andri Möll
@moll
Are you saying that non-binary items are hardlinked from content? From my brief reading I got the impression compiled items are just written to tmp, then hardlinked, then removed from tmp.
Denis Defreyne
@ddfreyne
Non-binary items are not hardlinked from content.
Andri Möll
@moll
Are compiled items stored somewhere aside from the output directory? I didn't notice any in my tmp...
Denis Defreyne
@ddfreyne
The intermediate compiled files are stored in the OS’ temporary directory (/tmp somewhere)
Andri Möll
@moll
For a long time or just during a single compile?
Denis Defreyne
@ddfreyne
During a single compilation
Andri Möll
@moll

Got it. Then it seems I guessed correctly before:

Wouldn't it be fine to move the temporary file? I suppose it's because you were trying to optimize by hard linking binary files.

I'm not sure how hard linking helps for compilable items then.
Denis Defreyne
@ddfreyne
For compileable items?
Do you mean textual, or binary?
Andri Möll
@moll
Presumably anything that's not identical to content, but particularly text files.
Denis Defreyne
@ddfreyne
I think what surprises me is that the DELETE + CREATE + MODIFY happens slow enough that the browser actually sees the file as missing
Andri Möll
@moll
It seems there's a pretty good opportunity for optimizing "compilation" of passed through binary items, too — not to compare them byte by byte with FileUtils.identical, but check if they're hard links. If I change the hard linked file in content, it's guaranteed to already match in output.
Yeah, I'm seeing it approximately every 10 updates. Fast computer, what can I say. :P
Denis Defreyne
@ddfreyne
Also makes me wonder whether Nanoc could create a hard link temporarily in ./tmp/nanoc (inside the site dir) and then move it in place (which would be atomic)…
Andri Möll
@moll
Still, how does a hard link help there?
There's one index.html compiled to /tmp. That's not around after the compile ends. Why not move it immediately?
Denis Defreyne
@ddfreyne
nanoc/nanoc#1321 Here’s the MR that introduced hard links
Andri Möll
@moll
I find his hard linking proposal to be sensible for files that are identical. For the changed items, I'm still puzzled.
Identical between content and output, that is.
Even the copy fallback seems wasteful if I understand the flow correctly. Given a compiled file (like Markdown -> HTML) temp_path referes to a file that's deleted after the method finishes. That file shouldn't be copied and then deleted, it should only be moved.
Denis Defreyne
@ddfreyne
I will have to give this a bit more thought, but I believe you are right
(Though if hardlinking fails, mv will likely be equivalent to a cp + rm anyway)
Andri Möll
@moll
For files on different filesystems, yep. However, as I currently understand it, a move itself (a.k.a rename), can be atomic while a hard link perhaps can't be.
Denis Defreyne
@ddfreyne
It looks like it!
Though for your particular problem, perhaps a 100ms delay somewhere would be enough
I am not sure how to introduce that, however.
Andri Möll
@moll
LiveReload does have a delay option (which Adsf sets to 0), but I can probably wait for the correct solution, too. :)
Denis Defreyne
@ddfreyne
adsf could definitely set it to something else (0.1 maybe)
Andri Möll
@moll
Perhaps making it configurable would be best.
Denis Defreyne
@ddfreyne
I’d suggest against that; if it turns out that configurability is needed because 0.1 does not work for everyone, we can revisit that
(Otherwise, you end up with something that is so configurable that the config file becomes overwhelming)