Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Mika Fonsén
    @mfonsen
    The error I mentioned on Friday was probably about something different. Takomo started working with mfa after I removed unused profiles from ~/.aws/config
    Mika Fonsén
    @mfonsen
    I had defined credential_process set in ~/.aws/config for unused profiles. These seemed to confuse Takomo (or aws-sdk). The error message ("Profile manager did not include credential process") would hint this was the case. Anyway now it works great.
    Henri Meltaus
    @hmeltaus
    ok, the way AWS SDK handles credentials is quite complex. I should probably do some more digging and at least document the most common use cases and problems
    Mika Fonsén
    @mfonsen
    That's true. Personally I wouldn't regard this as a problem in Takomo so I didn't verify this finding or try to replicate this with trivial aws-sdk example.
    Mika Fonsén
    @mfonsen

    I upgraded Takomo from 3.11.1 to 3.35.1 and found out that the dependency resolution no longer works like it used to. have c.yml stack configuration which has depends: - /e/{{ stackGroup.data.globalRegion }}/d.yml (no other dependencies). In the log output I can see that globalRegion is us-east-1. d.yml has no dependencies.

    When deploying I get: Deploy stacks in the following order (among other dependencies):

    Would you have suggestions what I could change or how could I debug this further?

    Henri Meltaus
    @hmeltaus
    hi, I need to check this. It certainly seems that something has changed there. It shouldn't take too long to fix the problem if I'm able to reproduce it with the configuration you gave.
    Henri Meltaus
    @hmeltaus

    I couldn't reproduce the issue with these three stacks which indicates there's something else that messes the order.

    I got this order:

    Anyway, thanks for the error report. I will go through the implementation to find where the problem lies.
    Henri Meltaus
    @hmeltaus
    In the meantime, you can print the stack dependency graph with this command and see if there are differences between Takomo versions
    Mika Fonsén
    @mfonsen
    Thank you. My guess is also that it's not that straight forward. Here's the new and old dependency output:
    Mika Fonsén
    @mfonsen
    Just to cut the long story short: they are the same
    Mika Fonsén
    @mfonsen
    the difference is in "load configuration" output
    Henri Meltaus
    @hmeltaus
    @mfonsen the bug affecting the stack deployment order has been fixed in v3.35.2 released this morning. Please, give it a try and let me know if the problem is gone
    There was a change in the logic that loaded existing stacks from the target accounts. It messed up the deployment order in cases when there was stacks with inter-region dependencies.
    Mika Fonsén
    @mfonsen
    Thank you very much! I noticed the release and I'm on it :). At least the deployment went through fine. The tests are still running but I would not expect any problems there.
    I noticed there's now a test case for "when super complex dependency graph is given" which is great for us. Thank you
    One small thing I noticed was a change in working directory of command hooks. For us it used to be the folder from which Takomo was executed but now it had changed to the path given with --dir parameter. This was easy to fix in our configuration.
    Henri Meltaus
    @hmeltaus
    I used your configuration to create that super complex test case =) , it helped a lot to spot where the problem was.
    Henri Meltaus
    @hmeltaus
    Glad to hear the unintentional breaking change in command hook working dir didn't cause you too many problems. The --dir option is used to specify the project directory from where Takomo looks for files with relative file path. Previously, the command hook didn't obey it.
    Tatu Tamminen
    @ttamminen

    Hi!

    We have been experimenting with running Takomo on Windows and so far there haven't been major issues and we should be able to fix the remaining problems.

    • running specific YAML file requires that the path starts with forward-slash which leads to an incorrect path on Windows. We might have to loosen up the regex a little bit
    • change the package.json os to include win32
    • possibly separate a few unit tests to os-specific, for example, tests that handle path/argument parsing
    The GitHub repository is now forked and we create PR to the forked repo. @hmeltaus is there anything we should know regarding PR's or things that would make life easier when (later) reviewing the code?
    Henri Meltaus
    @hmeltaus
    @ttamminen Hi, nice to hear that you've done some experimenting! Regarding "running specific YAML file", the path you give from the command line isn't intended to be actual file path but a "stack path", i.e. path used to identify a stack. That should remain as is because it's used in configuration files to refer to stacks. Instead, you should see how the stack path is used to locate files from file system. I think it should be enough to make sure the file paths translated from stack paths are handled correctly in Windows and Linux.
    Henri Meltaus
    @hmeltaus
    Then, I should document how to run integration tests using your own accounts. I have a pool of 50 AWS accounts that are used to run the integration tests, and you should configure a similar setup for yourself. Of course, you don't need to have that many accounts but it would be really nice if you could run integration tests, too. Alternatively, I can also run the integration tests if needed. Anyway, I try to create some documentation during the weekend.
    Henri Meltaus
    @hmeltaus
    hey, I felt Gitter isn't the best platform for this discussion so I created a Telegram group. It would be nice if you could post your messages there. https://t.me/takomo_io
    Tatu Tamminen
    @ttamminen

    Non-Windows related question.. this might be an actual bug, but maybe it's better to verify first.

    The backstory for the issue is that we were doing refactoring on YAML files (mainly style changes as we Prettier was enabled also on YAML files). We wanted to check that no unwanted functional changes were done between Master and PR branch.

    I noticed that when I run Takomo with detect-drift argument, the Takomo said on each stack that there are changes even though I had just deployed the latest version to my own environment.

    yml/us-east-1                   smith-us-east-1-devtatu                  UPDATE_COMPLETE           DRIFTED       7

    Then I tried to deploy, but I asked Takomo to show the differences per stack.

    The review showed the following

    Stack policy:
    
      Stack policy will be updated
    
        {
          "Statement": [
            {
      -       "Effect": "Deny",
      -       "Action": ["Update:Replace", "Update:Delete"],
      -       "Principal": "*",
      -       "Resource": "*",
      +       "Action": [
      +         "Update:Replace",
      +         "Update:Delete"
      +       ],
              "Condition": {
                "StringEquals": {
                  "ResourceType": [
                    "AWS::Cognito::UserPool",
                    "AWS::Cognito::UserPoolClient"
                  ]
                }
      -       }
      +       },
      +       "Effect": "Deny",
      +       "Principal": "*",
      +       "Resource": "*"
            },
            {
      +       "Action": "Update:*",
              "Effect": "Allow",
      -       "Action": "Update:*",
              "Principal": "*",
              "Resource": "*"
            }
          ]
        }

    There are no actual changes, only the order is different and it shouldn't matter.

    Our conclusion is that the diff is done incorrectly, as a text and not as an object where there is no object key order.

    @hmeltaus What are your thoughts on this? Is it a bug or a feature? :)

    Tatu Tamminen
    @ttamminen
    Example YAML with redacted information
    template: template.yml
    {{> shared.hbs }}
    parameters:
      Stage: ""
      Suffix: ""
      StageWithSuffix: ""
    stackPolicy: |
      {
        "Statement": [
          {
            "Effect": "Deny",
            "Action": ["Update:Replace", "Update:Delete"],
            "Principal": "*",
            "Resource": "*",
            "Condition": {
              "StringEquals": {
                "ResourceType": [
                  "AWS::Cognito::UserPool",
                  "AWS::Cognito::UserPoolClient"
                ]
              }
            }
          },
          {
            "Effect": "Allow",
            "Action": "Update:*",
            "Principal": "*",
            "Resource": "*"
          }
        ]
      }
    Henri Meltaus
    @hmeltaus
    it's a bug. I try to fix it tonight
    Henri Meltaus
    @hmeltaus
    You also posted output from drift detection command. It uses this, so your local Takomo configuration doesn't affect to the outcome. Could you check the drift status directly from AWS console and let me know if there's something unexpected.
    Henri Meltaus
    @hmeltaus
    @ttamminen stack policy bug is fixed in version 3.40.1
    Tatu Tamminen
    @ttamminen
    Excellent, thank you! We are going to test the version today
    Tatu Tamminen
    @ttamminen

    @hmeltaus We tested the fix, and the order of the properties didn't cause differences; the bug can be closed. Thank you for the bug fix. 👍

    But, there is another issue :) It is about YAML comments.

    After successfully deploying the latest source code to my environment, I reran the deployment on the latest Takomo, expecting no changes.

    To my surprise, all stacks 35 was going to update

    ~ /events/eu-west-1/x.yml/eu-west-1:             (stack will be updated)
    ...

    I chose "continue, but let me review changes to each stack" and "review changes in the stack template" for the first stack.

    AWSTemplateFormatVersion: "2010-09-09"
        Description: >
          Redacted
    
        Parameters:
          Stage:
            Type: String
          Suffix:
            Type: String
            Default: ""
          StageWithSuffix:
            Type: String
            Description: >
              Redacted   
        Resources:
          # Redacted
          MachineUserPool:
            Type: AWS::Cognito::UserPool
            Properties:
              UserPoolName: !Sub "<redacted>"
      +       # Redacted
      +       # Redacted
              AliasAttributes:
                - email
              Schema:
                - Name: name
                  AttributeDataType: String
                  Mutable: true
                  Required: true
                - Name: email
                  AttributeDataType: String
                  Mutable: true
                  Required: true
    ...

    See those two comment lines? Some of the comments were in the source code repository, but not on the AWS side and it seems that getting those synchronised isn't going to happen with deploy command.

    Our theory is that when adding a new template (with comments), the CloudFormation will apply the template and its comments, like in the example line below, the resources: was found.

    Later, the developer adds only comment and/or comment changes, and those doesn't affect stacks; the change is never applied.

    The Takomo will always say that there are differences as the source code will have the comments, but AWS will not have those until someone makes changes that affect the stack.

    Henri Meltaus
    @hmeltaus
    ok, first of all, when you review the stacks you are about to deploy, Takomo says "stack will be updated" for all stacks that currently exists in the target account. This doesn't mean that the stack will be actually updated as that can't be known at this point of the deployment. I know that this can be a bit misleading, maybe better phrasing would be something like "stack might be updated". What do you think?
    Henri Meltaus
    @hmeltaus
    Then, about the comments in template files. CloudFormation won't perform stack update if stack resources haven't changed. So, updating only comments or changing the order of resources won't trigger an update. Also, Takomo shouldn't let you review changes if only comments were added or modified, instead it should just skip the stack in question and move on to the next one. There's at least one corner case where you are able to review the changed template even if only the template's comments were changed, though. That is when you modify the stack's tags via stack configuration file and also modify the stack template file by adding or removing some comments. Because the stack tags will be added to the stack resources, the stack update will be performed even if there are no real changes in the stack template.
    Tatu Tamminen
    @ttamminen
    Sorry for the long delay. I was doing a major refactoring to the YAML files and gathered some insights regarding CloudFormation and Takomo. 👍
    One question where we didn't find an answer was a mismatch between takomo detect-drift and actual deployment with a review phase
    Tatu Tamminen
    @ttamminen

    After successful deployment to an environment, I ran the detect-drift. This is what I got (names redacted)

    ⇒  npx takomo stacks detect-drift --dir aws 
    [info ] - Load configuration
    comparing undefined 
    2022-01-16 16:43:58 +0200 [info ] - Detecting drift, this might take a few minutes...
    2022-01-16 16:43:58 +0200 [info ] - Load current stacks
    
    Path                                                  Name                               Status           Drift status  Drifted resources
    ----------------------------------------------------  ---------------------------------  ---------------  ------------  -----------------
    /l/eu-north-1/a.yml/eu-north-1                        a-eu-north-1-dev                    UPDATE_COMPLETE  IN_SYNC       0                
    /l/eu-north-1/b.yml/eu-north-1                        b-eu-north-1-dev                    UPDATE_COMPLETE  IN_SYNC       0                
    /l/eu-north-1/c.yml/eu-north-1                        c-eu-north-1-dev                    UPDATE_COMPLETE  IN_SYNC       0                
    /l/eu-north-1/d.yml/eu-north-1                        d-eu-north-1-dev                    UPDATE_COMPLETE  DRIFTED       7                
    ...   
    /l/eu-west-1/d.yml/eu-west-1                          d-eu-west-1-dev                     UPDATE_COMPLETE  DRIFTED       7                
    /l/us-east-1/j.yml/us-east-1                          j-us-east-1-dev                     UPDATE_COMPLETE  IN_SYNC       0                
    /l/us-east-1/dd.yml/us-east-1                         dd-us-east-1-dev                    UPDATE_COMPLETE  IN_SYNC       0                
    /l/us-east-1/ee.yml/us-east-1                         ee-us-east-1-dev                    UPDATE_COMPLETE  IN_SYNC       0                
    /l/us-east-1/d.yml/us-east-1                          d-us-east-1-dev                     UPDATE_COMPLETE  DRIFTED       7                
    /ö/eu-west-1/ff.yml/eu-west-1                         ff-eu-west-1-dev                    UPDATE_COMPLETE  DRIFTED       1                
    /ö/us-east-1/j.yml/us-east-1                          j-us-east-1-dev                     UPDATE_COMPLETE  IN_SYNC       0                
    /ö/us-east-1/dd.yml/us-east-1                         dd-us-east-1-dev                    UPDATE_COMPLETE  IN_SYNC       0                
    /ö/us-east-1/ee.yml/us-east-1                         ee-us-east-1-dev                    UPDATE_COMPLETE  IN_SYNC       0

    This is weird as it was just deployed and when I run deployment for a single stack that has drift number 7, there are now changes

    This is probably some kind of bug on the CloudFormation drift functionality. It would help a lot if the command would print what causes the drifting 🙂
    The second CloudFormation related problem was unicode on YAML files
    Tatu Tamminen
    @ttamminen
    image.png
    We have Cognito/SES related emails that contained copyright mark which is on the Unicode space instead of ANSI. The Takomo will read UTF-8 YAML files correctly, but unfortunately, the CloudFormation API always returns Unicode characters as a question mark. This causes unnecessary diffs
    Tatu Tamminen
    @ttamminen
    One really nice feature would be to get the final Handlebars/YAML result dumped into a directory. As we cannot rely on drift functionality, it would be nice to take source code from master and dump the compiled output, then take PR with YAML changes and finally make a diff that there are no accidental changes. We had a lot of changed files as we had to modify files that mixed both YAML and Handlebars to be compatible with Prettier and YAML Linting :)
    Anyways, as I said, these were more like CloudFormation related issues and probably there isn't much Takomo can do except the file output
    Henri Meltaus
    @hmeltaus
    Regarding drifts, you can check stack's drifted resources from AWS console. I'll see how improve detect drift command to include more information in the output.
    I'll also check how unicode behaves with YAML and see if there's something we can improve
    Henri Meltaus
    @hmeltaus
    Dumping compiled YAMLs to files sounds simple but there's one challenge. That is when there are dependencies between stacks. If stack A depends on stack B, we need to actually deploy stack B before we can compile template for stack A. This is because, stack A template might contain references to stack B's outputs or some other dynamically resolved variables. That's why we can't just simply compile all templates.
    If you want to make sure there are no unexpected changes after your refactorings, you can use --expect-no-changesoption when deploying the stacks. It runs deploy but stops immediately if some stacks are actually updated. See https://takomo.io/docs/command-line-usage/deploy-stacks
    Mika Fonsén
    @mfonsen
    I've tried to build an "after" hook that reads stack output. It seems that the stack inputs or outputs are not available as the input parameters. Are they available elsewhere? I'll work around by using aws cli as other hooks and then reading their output. Currently running 3.35.2.
    Henri Meltaus
    @hmeltaus
    Hi, stack input nor outputs are not available for hooks. You need to use AWS SDK or CLI as you said. It shouldn't be too hard to expose stack operation results to after hooks. I'll see how to implement that.
    Mika Fonsén
    @mfonsen
    Thank you. Yes, this is definitely not critical as there's an easy work around. Just wanted to confirm that I hadn't missed something. The use case was to initialize a database after creation using the generated IDs from the stack.