Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Nov 10 11:46

    ddfreyne on master

    Upgrade Nanoc (compare)

  • Nov 10 11:39

    ddfreyne on 4.11.14

    (compare)

  • Nov 10 11:39

    ddfreyne on nanoc-core-v4.11.14

    (compare)

  • Nov 10 11:38

    ddfreyne on nanoc-cli-v4.11.14

    (compare)

  • Nov 10 11:38

    ddfreyne on master

    Set version to 4.11.14 Add release notes for 4.11.14 (compare)

  • Nov 10 10:42
    josepegea commented #1462
  • Nov 10 09:33

    ddfreyne on fix-dep-bug

    (compare)

  • Nov 10 09:32

    ddfreyne on master

    Let dep tracker properly recogn… Merge pull request #1464 from n… (compare)

  • Nov 10 09:32
    ddfreyne closed #1464
  • Nov 10 09:32
    ddfreyne closed #1463
  • Nov 08 19:50
    ddfreyne edited #1464
  • Nov 08 19:50
    ddfreyne labeled #1464
  • Nov 08 19:50
    ddfreyne opened #1464
  • Nov 08 19:49

    ddfreyne on fix-dep-bug

    Let dep tracker properly recogn… (compare)

  • Nov 08 19:47
    ddfreyne opened #1463
  • Nov 08 19:47
    ddfreyne labeled #1463
  • Nov 08 19:43
    ddfreyne labeled #1462
  • Nov 08 19:43
    ddfreyne labeled #1462
  • Nov 04 16:13
    Fjan opened #1462
  • Nov 02 12:19

    ddfreyne on master

    Upgrade packages (and Nanoc to … (compare)

Andri Möll
@moll
Presumably at some point the output file has to be missing entirely, otherwise the web server wouldn't report a 404.
Denis Defreyne
@ddfreyne
Hmmm, it should be in place!
This is pretty weird.
Andri Möll
@moll

Curiously, I'm seeing a delete happen ...

19:51:36 DELETE public/articles/foo/index.html
19:51:36 CREATE public/articles/foo/index.html
19:51:36 MODIFY public/articles/foo/index.html
19:51:36 CLOSE_WRITE,CLOSE public/articles/foo/index.html

This with:

inotifywait --recursive --monitor public --format "%T %e %w%f" --timefmt "%T" -e modify -e attrib -e close_write -e moved_to -e moved_from -e move -e move_self -e create -e delete -e delete_self
And that triggered a race which LiveReload won, reloading before the create and coming up with "File not found".
Andri Möll
@moll
I think I know why this is happening. Nanoc's not using renaming, it's using hard linking.
https://github.com/nanoc/nanoc/blob/4.10.1/nanoc/lib/nanoc/base/services/item_rep_writer.rb#L59 calls FileUtils.ln with force: true. Force, as per Ruby's implementation, just unlinks it before creating a new link:
https://ruby-doc.org/stdlib-2.4.1/libdoc/fileutils/rdoc/FileUtils.html#method-c-ln
That explains the delete inotify event before create and explains the race condition.
Andri Möll
@moll
Why's Nanoc using linking or copying there in the first place? Wouldn't it be fine to move the temporary file? I suppose it's because you were trying to optimize by hard linking binary files.
..given that in one branch of the code temp_path is set to not a true temporary path, but the actual filename from the content directory.
Denis Defreyne
@ddfreyne
@moll The hardlinking’s purpose is to improve disk usage
Moving stuff in place can work under certain conditions, but not all
(e.g. the input file in content/ might be hardlinked to what is in output/!)
Andri Möll
@moll
Are you saying that non-binary items are hardlinked from content? From my brief reading I got the impression compiled items are just written to tmp, then hardlinked, then removed from tmp.
Denis Defreyne
@ddfreyne
Non-binary items are not hardlinked from content.
Andri Möll
@moll
Are compiled items stored somewhere aside from the output directory? I didn't notice any in my tmp...
Denis Defreyne
@ddfreyne
The intermediate compiled files are stored in the OS’ temporary directory (/tmp somewhere)
Andri Möll
@moll
For a long time or just during a single compile?
Denis Defreyne
@ddfreyne
During a single compilation
Andri Möll
@moll

Got it. Then it seems I guessed correctly before:

Wouldn't it be fine to move the temporary file? I suppose it's because you were trying to optimize by hard linking binary files.

I'm not sure how hard linking helps for compilable items then.
Denis Defreyne
@ddfreyne
For compileable items?
Do you mean textual, or binary?
Andri Möll
@moll
Presumably anything that's not identical to content, but particularly text files.
Denis Defreyne
@ddfreyne
I think what surprises me is that the DELETE + CREATE + MODIFY happens slow enough that the browser actually sees the file as missing
Andri Möll
@moll
It seems there's a pretty good opportunity for optimizing "compilation" of passed through binary items, too — not to compare them byte by byte with FileUtils.identical, but check if they're hard links. If I change the hard linked file in content, it's guaranteed to already match in output.
Yeah, I'm seeing it approximately every 10 updates. Fast computer, what can I say. :P
Denis Defreyne
@ddfreyne
Also makes me wonder whether Nanoc could create a hard link temporarily in ./tmp/nanoc (inside the site dir) and then move it in place (which would be atomic)…
Andri Möll
@moll
Still, how does a hard link help there?
There's one index.html compiled to /tmp. That's not around after the compile ends. Why not move it immediately?
Denis Defreyne
@ddfreyne
nanoc/nanoc#1321 Here’s the MR that introduced hard links
Andri Möll
@moll
I find his hard linking proposal to be sensible for files that are identical. For the changed items, I'm still puzzled.
Identical between content and output, that is.
Even the copy fallback seems wasteful if I understand the flow correctly. Given a compiled file (like Markdown -> HTML) temp_path referes to a file that's deleted after the method finishes. That file shouldn't be copied and then deleted, it should only be moved.
Denis Defreyne
@ddfreyne
I will have to give this a bit more thought, but I believe you are right
(Though if hardlinking fails, mv will likely be equivalent to a cp + rm anyway)
Andri Möll
@moll
For files on different filesystems, yep. However, as I currently understand it, a move itself (a.k.a rename), can be atomic while a hard link perhaps can't be.
Denis Defreyne
@ddfreyne
It looks like it!
Though for your particular problem, perhaps a 100ms delay somewhere would be enough
I am not sure how to introduce that, however.
Andri Möll
@moll
LiveReload does have a delay option (which Adsf sets to 0), but I can probably wait for the correct solution, too. :)
Denis Defreyne
@ddfreyne
adsf could definitely set it to something else (0.1 maybe)
Andri Möll
@moll
Perhaps making it configurable would be best.
Denis Defreyne
@ddfreyne
I’d suggest against that; if it turns out that configurability is needed because 0.1 does not work for everyone, we can revisit that
(Otherwise, you end up with something that is so configurable that the config file becomes overwhelming)
Andri Möll
@moll
If it's only to bounce back a delete-and-create, you could then go with 50ms probably, too.
I've used LiveReload generally for a long time and I remember it requiring quite a bit of tweaking to find the sweet spot for each project — sometimes compiling JavaScripts or CSS to the output directory took a long time and having the browser constantly reload during that time was annoying. :P
Ideally Nanoc should signal a reload only once it has actually finished its compile cycle.
But with LiveReload being part of adsf instead of Nanoc, that may be more difficult practice than in theory.
Denis Defreyne
@ddfreyne
Hm yeah, good point… that’d be good to have
Denis Defreyne
@ddfreyne
@moll If you’re willing to add support for that to adsf, I’d be happy to help!