playscale/stashed.io

Let's talk about build acceleration

midinerd

@midinerd_twitter

Hey Jean - is there a log I can check any details on the hit/misses?
I'm trying to figure out a moving target in our build system that might cause 'misses' from a filename changing, but the content being the same (i.e. this modifies the cmdline/pathing)

I'm fairly certain this is the reason, but if I could get some feedback from stashed to confirm it I'll definitely leverage that.

Jean-Sebastien Mouret

@jsmouret

sadly, if nothing is showing in the logs tab in the dashboard, there are no other options for now

we can plan to add an option to log more info (like a debug switch)

akaraivanov

@akaraivanov

Hi. I am testing stashed. I think cache hit is not very good. I did number of rebuild and clear cycles with stashed. Also increased disk cache size. Before the last rebuild I cleared logs, and statistics. Then I get in statistics:
Job hits: 1940
Job miss: 2
Blob hit: 10,531
Blob miss: 2,146

In logs, I get 340 warnings of the same type having reason: "/out:something\foo.rc.res.dep.obj", that appear to be related to compilation of resources.

Q1: how can I diagnose where are blob misses coming from?
Q2: is it possible to disable temporarily stashed to have more builds without it?

akaraivanov

@akaraivanov

On the next day, when my vm was restarted, I do not get Blob miss any more... Strange

Jean-Sebastien Mouret

@jsmouret

sorry, for the late reply

Q1: for common cases like using PDB+PCH at the same time, it just increases the `unsupported job' counter. for other cases, it should output an error/warning telling why it bailed in the Logs tab on the dashboard.

Q2: yes, you can disable all frontends at once in the dashboard

Looking at the stats above, very few jobs misses and plenty of blob misses is a symptom of a small cache size. stashed knows about the compilation request (job hit) but cannot find the matching obj files (blob miss), most likely because they got removed from the cache.

akaraivanov

@akaraivanov

Thanks for the answer! I will try increasing the cache size

bpaberg

@bpaberg

Hi, I'm also testing stashed. looks really cool. A question:
There is an old problem compile problem with building in parallel (/MP) and using #import so we do not have /MP on some projects.
With stashed this problem started happening again. Does stashed build in parallel even if /MP is not on?

I'm thinking that since I saw this:
"In this scenario, caching is disabled but compilation will still benefit from automatic multi-threading and the correct output will be generated."
here: https://github.com/playscale/stashed.io/wiki/Stats#job-unsupported

bpaberg

@bpaberg

Also, I tried to put the #import in StdAfx.h since that one should be compiled before everything else but then I got cache miss on all jobs in the project.

Jean-Sebastien Mouret

@jsmouret

as far as I know, stashed simply discards /MP

stashed splits the command line in multiple compilation requests that are indeed run in parallel whether /MP is on or not

(googling the #import problem, we don't do anything special about it...)

Jean-Sebastien Mouret

@jsmouret

looking at https://msdn.microsoft.com/en-us/library/bb385193.aspx it says that /MP is not compatible with #import, but I'd like to know why to understand how we can workaround that...

bpaberg

@bpaberg

yea, well /MP is not compatible with #import since compiling a cpp with import will generate a tlh and tli files so if multiple cpp files have the same #import and are compiled in parallell then they will clash on disk, getting a permissin denied error

tis is a limitation in cl and I guess that modern code does not use #import anymore

there is also a workaround that Microsoft mention, to make a specific pass (target) that have the #imports and will generate the tlh and tli files, then in another pass (target) use the generated files.

so I would guess that stashed should not really have to do anything about this cl-flaw apart from maybe mention it and the workaround in the faq

bpaberg

@bpaberg

btw, with stashed, there is a job miss on cpp-files that use #import, maybe it should be "job unsupported" instead?

Jean-Sebastien Mouret

@jsmouret

is there anything on the dashboard logs when that file is compiled?

bpaberg

@bpaberg

no, just that I get job miss for it in Stats

bpaberg

@bpaberg

there is a blob hit for it though, how can that be if is is a job miss?

Jean-Sebastien Mouret

@jsmouret

the job itself is stored as a blob, which is fetched ok, but it might be missing dependencies so it counts a job miss. we must be doing something wrong when finding dependencies/outputs for cpp files with #import.

bpaberg

@bpaberg

mm, i guess the tlb file is a dependency and tlh tli files are extra outputs

bpaberg

@bpaberg

There is a lot of stuff happening around #import, it can take many arguments on how to generate the tlh and tli files. I don't know if that matters to stashed though.
maybe stashed don't need to bother with all that. just identify it as "job unsupported" if #import is involved (through any header) and then have the workaround in the FAQ.
Thats pretty much what Microsoft do with cl /MP.

bpaberg

@bpaberg

If stashed was to learn how to cache with #import and if pch is used then I suppose the easiest a workaround is to put the #import directives in the StdAfx.h and the other files could simply include the generated tlh files instead of using #import.

Since the pch has to be compiled before everything else, even when compiling in parallel, the tlh and tli files will be generated once only and can be used by the other compile units.

akaraivanov

@akaraivanov

Is disk cache location configurable and how? Thanks

akaraivanov

@akaraivanov

If that is not configurable then I guess I can just do directory sym link from C:\.stashed to the desired location

Jean-Sebastien Mouret

@jsmouret

Sorry, the cache location is not configurable. You can select which drives to use but not the path.

akaraivanov

@akaraivanov

That would be just fine (I need the cache on a fast disk). How can I select the drive?

Jean-Sebastien Mouret

@jsmouret

In the dashboard, there is a Cache tab where you can disable unwanted drives

akaraivanov

@akaraivanov

Perfect. Thanks!

Ben Peck

@bpeck

We're having trouble getting any of our machines to start creating any local cache entries with UE4 builds.

Our studio has a convention of using a mapped virtual drive, eg mapping c: to z:. Could this be causing the problem.

ie everyone has their workspace mapped to a common base path on a z: drive. But they are mapping from an arbitrary physical drive like c: or d:

Jean-Sebastien Mouret

@jsmouret

Interesting... I don't think we have tried this. Stashed will try to use the local hard drive to cache first (which can be confusing with the mapped drive) but it should fallback on others. We need some time to investigate and we'll get back to you

KonanM

@KonanM

Ok interesting. It seems like there is excatly the same issue for my company. We also use virtual drives and stashed.io doesn't work...

willjoseph

@willjoseph

I'm running into warnings when stashed.io is installed only on my machine, and I run a UE4 build distributed across multiple IncrediBuild agents. EXEC : warning : Stashed service could not be reached from this remote environment. Job hit:miss rate for consecutive full rebuild is around 1:3, probably due to this warning. All agents are Windows 10 with no special firewall settings that I know of.

bitsrules

@bitsrules

I just installed stashed.io and it work possible because of my compiler flags. Here is the command I am using
cl file.cpp /MP /GS /TP /analyze- /W3 /Zc:wchar_t /I"Z:\F\dev\include" /I"C:\dev\vcpkg\installed\x86-windows\include" /Gm- /Od /Ob0 /Fd"proxy.dir\Debug\proxy.pdb" /Zc:inline /fp:precise /D "WIN32" /D "_WINDOWS" /D "BOOST_CONFIG_SUPPRESS_OUTDATED_MESSAGE=1" /D "CMAKE_INTDIR=\"Debug\"" /D "_MBCS" /errorReport:prompt /WX- /Zc:forScope /RTC1 /Gd /Oy- /MD /std:c++17 /FC /Fa"Debug/" /EHsc /nologo /Fo"proxy.dir\Debug\" /Fp"proxy.dir\Debug\proxy.pch" /diagnostics:classic

I got it to work with a following the example in the docs i.e. cl /c /Yc a.cpp

Justin Randall

@Justin-Randall

@bitsrules Have you tried with /Z7?

bitsrules

@bitsrules

/Z7 did also not work. The solution was to move the source code from my shared drive to the local disk

Ben Peck

@bpeck

@jsmouret Have you had a chance to look into using virtual drives with stash?

Justin Randall

@Justin-Randall

It does work with mklink

olesteban

@olesteban

Hi, I installed yesterday this product for evaluation. Coming from linux ccache (even distcc) I'm surprised to see it depends on the absolute paths (two working copies of the same code don't share cache results). I've read it's a restriction for distributed use, but is it also local?

olesteban

@olesteban

I've seen (playscale/stashed.io#6) it's also local :-( OTOH, I'd say ccache does not depend on any environment variable), but (not sure, as I read about it long time ago) on the command line and a hash of the source files involved.

Justin Randall

@Justin-Randall

Will have another look and add a test case for relative paths.

Adrian Cowan

@othrayte

@Justin-Randall, in your update to #6 you mention the tradeoffs for relative path support and asked if "everyone [would] be OK with undefined debug behavior if the strange outcomes were well documented?". I imagine this would be acceptable to me, but also without relative path support almost all of the advantage of stashed would be lost for me. Could you go into what the kinds of strange outcomes might be? i.e. what the documentation might say.

Where communities thrive

People

Repo info

Activity