Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • May 17 20:27
    mimaslanka closed #19
  • May 17 20:27
    mimaslanka opened #19
  • May 17 20:01
    mimaslanka closed #18
  • May 17 20:00
    mimaslanka opened #18
  • Jul 26 2021 05:37
    dependabot-preview[bot] closed #87
  • Jul 26 2021 05:37
    dependabot-preview[bot] opened #88
  • Jul 07 2021 05:38
    dependabot-preview[bot] closed #86
  • Jul 07 2021 05:38
    dependabot-preview[bot] opened #87
  • Jun 18 2021 05:41
    dependabot-preview[bot] closed #85
  • Jun 18 2021 05:41
    dependabot-preview[bot] opened #86
  • May 28 2021 05:40
    dependabot-preview[bot] closed #84
  • May 28 2021 05:40
    dependabot-preview[bot] opened #85
  • May 06 2021 05:37
    dependabot-preview[bot] closed #83
  • May 06 2021 05:37
    dependabot-preview[bot] opened #84
  • May 03 2021 05:37
    dependabot-preview[bot] closed #81
  • May 03 2021 05:37
    dependabot-preview[bot] opened #83
  • Apr 29 2021 20:21
    dependabot-preview[bot] labeled #13
  • Apr 29 2021 20:21
    dependabot-preview[bot] opened #13
  • Apr 28 2021 21:59
    dependabot-preview[bot] labeled #82
  • Apr 28 2021 21:59
    dependabot-preview[bot] labeled #17
Jamie Winsor
@reset
@sersut just pushed berkshelf 3.1.5 out - would be good to get a new chefdk release
Serdar Sutay
@sersut
on our todo list @reset... let us know if there is a significant issue that requires an immediate release..
Dan DeLeo
@danielsdeleo
The question I have is, what specific problem do you have in berks w/ no metadata "name" attribute? Is this problem restricted to shared cookbooks from supermarket and such?
My thinking is that we have not had in-code deprecation for not having the name attribute in your metadata, so it's a little harsh to suddenly require it with not much warning.
It could make sense to have a "strict mode" for metadata, where the name attribute is required, but by default we only activate the strict mode for uploads to supermarket
otherwise, you get a noisy warning.
then we make it required in 13.0, having given users plenty of advance notice
Dan DeLeo
@danielsdeleo
Would that solve the problems you've encountered, or am I missing something?
Jamie Winsor
@reset

@danielsdeleo no, it won’t solve the problems we’ve been having. The identifier for the cookbook is a name and a version. It should NEVER change. It changes depending on the directory it’s located in. Because Berkshelf stores things in it’s shelf as {cookbook_name}-{version} the name of the cookbook is wrong on upload.

As a side note, and this isn’t fingerpointing and has nothing to do with you or anyone on your team, it’s been two years since I filed that ticket and it’s pretty awful that it’s taken this long for it to be addressed.

There is a hard requirement instead of a warning or a deprecation because Chef was taking way too long to fix it themselves and Berkshelf gained enough traction to educate it’s users how to properly develop their cookbooks
You’d be hard pressed to find a Berkshelf user who doesn’t know that the name attribute is required. This should be part of the training going forward if it’s not immediately obvious
Dan DeLeo
@danielsdeleo
@reset I'm 100% with you on the philosophical side
As for the time it's taken to fix, it's a breaking change, and you filed the ticket against 11.10.4. Everyone's been in favor of the change, it's just not something we can do in a minor version. If you upgrade from 11.14 to 11.16 and you can't upload cookbooks that worked with 11.14, I think you have a right to be pissed off, even if everyone has good intentions and is correct
Dan DeLeo
@danielsdeleo
@reset anyway, let's focus on the technical problem. Are you saying that some berks users set their Chef::Config[:cookbook_path] to be the berks cache, store cookbooks with no metadata name attribute in there, and then use knife cookbook upload to upload them?
Dan DeLeo
@danielsdeleo
@reset I'm also looking into CHEF-4811 CHEF-4810 http://blog.vialstudios.com/the-importance-of-compiled-metadata/
I've done a little bit of testing so far, and even when I intentionally insert a bug in a metadata.rb file, chef-client never sees it
I have
raise "bug is here" unless `hostname`.chomp == "lorentz.local"
where lorentz.local is the hostname of my laptop, and I'm testing w/ chef-client on a VM (using default hostname "ubuntu" )
but chef-client never hits that line of code. I've verified the metadata.rb in my /var/chef/cache/cookbooks has the version of metadata.rb with this raise statement
So, I'm wondering if you have a bug report in berks/ridley/etc. with a stack trace I can look at to see how the metadata.rb is getting evaluated by chef-client
Dan DeLeo
@danielsdeleo
Or am I misunderstanding and the problem is more like, upload to chef-server, download to other machine, test with chef-solo (maybe in vagrant or TK or whatever), chef-solo blows up?
Jamie Winsor
@reset
@danielsdeleo yeah it’s understandable that you don’t want to break minor releases - it’s my opinion that we went too long without seeing a major release which is what forced tool developers to force their hand. Not useful to this conversation though, so I’ll get off my box ;)
@danielsdeleo no, Berkshelf stores cookbooks in ~/.berkshelf/cookbooks. It stores them in directories named {cookbook_name}-{version} or {cookbook_name}-{hash}. This is completely fine for cookbooks with a metadata.json present (because we favor it), but not for cookbooks whose metadata needs to be evaluated
Because the name of the cookbook when metadata.json and a name field in metadata.rb isn’t present is inferred from the directory name. This is just a awful oversight that has lasted way too long in the codebase.
Jamie Winsor
@reset
@danielsdeleo for your third question: no, the CookbookVersionLoader prefers the raw metadata: https://github.com/opscode/chef/blob/master/lib/chef/cookbook/cookbook_version_loader.rb#L68-L74
So in my testing the Chef-Client will load it’s cookbooks and prefer the raw metadata to that of the compiled JSON. You can repro it extremly easiliy with a cookbook who infers it’s version number by running some Ruby code to evaluate the contents of a file on disk that isn’t present on the Chef Server
Here’s a cookbook that exhibits that behaviour: https://github.com/berkshelf/berkshelf-api/blob/master/cookbook/metadata.rb
Dan DeLeo
@danielsdeleo
@reset do you have a stack trace for this happening on chef-client? I tried to repro with that code above which should have blown something up if chef was reading the metadata.rb
Jamie Winsor
@reset
@danielsdeleo no, I filed that ticket a while ago so I don’t have any of the test data I was using. It’s very easy to reproduce by uploading the Berkshelf-API cookbook to a Chef Server with a version of Knife which will compile the metadata but also upload the raw (.rb) and compiled (.json) metadata to the server. Add a recipe of the cookbook to the run list of a node and run Chef. You should see that it barfs attempting to resolve the version of the cookbook because it is loading it’s raw metadata.
Dan DeLeo
@danielsdeleo
@reset I think that behavior must have changed for some unrelated reason, chef-client is definitely not evaluating the metadata.rb, it's using the in-memory copy of the metadata it got from the server
The code path for chef-client doesn't hit CookbookVersionLoader now anywhere that I see
Jamie Winsor
@reset
That’s good news for the latest versions of Chef then
Pretty much ALL codepaths which attempt to evaluate metadata should be looking at compiled instead of raw. The CookbookVersionerLoader was the one that I had found a while back which is what made me open that ticket
Dan DeLeo
@danielsdeleo
Anyway, the point of all this is that the metadata handling is still insane, trying to get a handle on what the shape of "sane" looks like
Jamie Winsor
@reset
Ya the code base seems to be inconsistent in a lot of places where Ruby compiles into JSON. I think a general rule of thumb should be to write it all with the Ruby DSL and let tools compile and compare the JSON (or whatever the fuck format)
This will let tools developers generate non-Ruby tools to work with Chef artifacts
@danielsdeleo thanks so much for taking an interest in this piece of the picture. It’s extremely important for tools developers
Also, I linked you to an article that I want you to read on supermarket 701. It’s how Bundler’s new index works
This is also the route we decided to take in the Elixir language with our package manager
Dan DeLeo
@danielsdeleo
yeah, so a very long time ago, we always compiled to metadata.json on-disk then uploaded that
Jamie Winsor
@reset
Yeah I remember those days. It was confusing as all hell
Dan DeLeo
@danielsdeleo
that sucked for SVN users, since svn sucks at ignoring files
Jamie Winsor
@reset
I started using Chef in 0.8 and one of the biggest oddities was when to use the Ruby DSL and when to use JSON.
Dan DeLeo
@danielsdeleo
yeah, you could also just have a metadata.json and no .rb
Jamie Winsor
@reset
That’s what my very first chef-repo looked like
It wasn’t until things got more complex that I realized I didn’t want to be writing the JSON myself
Dan DeLeo
@danielsdeleo
@reset anyway, y'know, I'd read that article about the rubygems index format thing, totally spaced it. Anyway, how does the elixir repo publish updates within a cluster of index servers?
Jamie Winsor
@reset
Hex is the Elixir package manager and this is the approach which is being adopted. First steps are in on downloading the entire universe cache and then diffing will come second when necessary.