takomo-io/community

Takomo - Organize, parameterize and deploy CloudFormation stacks

Henri Meltaus

@hmeltaus

Currently, the template is processed before the parameters, and that's why parameters are not available in templates. It should be possible to change this ordering and expose resolved parameters to the template.

jonatcorus

@jonatcorus

Hi, cool project. Is there a larger example of something more complete then the examples in takomo-examples?

17 replies

Henri Meltaus

@hmeltaus

This message was deleted

jonatcorus

@jonatcorus

Any opinions on pulumi?

1 reply

Mika Fonsén

@mfonsen

I tried takomo@3.0.0-alpha.5. The first thing I bumped into was the following error:
2021-01-21 11:25:44 +0200 [debug] - Init resolver 'my-resolver' for stack: '/events/eu-north-1/a.yml', parameter: 'myParameter' with properties:

resolver: my-resolver
functionName: my-function
role: 'arn:aws:iam::00000:role/b-eu-north-1-c-role'
region: eu-north-1
immutable: false

ERROR

TypeError: Cannot read property 'keys' of undefined
at Object.schema (/project/aws/resolvers/upload.js:6:19)
at ResolverRegistry.initResolver (/packages/takomo/3.0.0-alpha.5/node_modules/@takomo/stacks-resolvers/dist/resolver-registry.js:33:41)
at initializeResolver (/packages/takomo/3.0.0-alpha.5/node_modules/@takomo/stacks-context/dist/config/parameters.js:17:45)
at Object.buildParameters (/packages/takomo/3.0.0-alpha.5/node_modules/@takomo/stacks-context/dist/config/parameters.js:28:36)
at async Object.buildStack (/packages/takomo/3.0.0-alpha.5/node_modules/@takomo/stacks-context/dist/config/build-stack.js:39:24)
at async Promise.all (index 2)
at async processStackGroupConfigNode (/packages/takomo/3.0.0-alpha.5/node_modules/@takomo/stacks-context/dist/config/process-config-tree.js:93:29)
at async Promise.all (index 0)
at async processStackGroupConfigNode (/packages/takomo/3.0.0-alpha.5/node_modules/@takomo/stacks-context/dist/config/process-config-tree.js:96:5)
at async Promise.all (index 0)

OTHER INFO

Your environment:
OS: darwin
Node version: v14.15.4
Takomo version: 3.0.0-alpha.5

Get support:
Docs: https://takomo.io
Bugs: https://github.com/takomo-io/takomo/issues

Related code:
module.exports = {
name: "my-resolver",
schema: (joi, schema) => {
return schema.keys({
functionName: joi.string().required(),
region: joi.string().required(),
role: joi.string().required(),
})
},
init: async props => {
...

No issues with takomo@2.12.0

7 replies

Mika Fonsén

@mfonsen

Bumped into a problem when reviewing stack changes which deletes an Elastic Container Registry. Happened on 3.0.1 and did not try with the latest version so I did not create a issue. Worked fine without reviewing the changes:

2021-03-17 08:42:51 +0200 [debug] - /app/eu-north-1/stack.yml/eu-north-1 - Begin step 'review-change-set'

Review deployment plan for a stack:
-----------------------------------
A stack deployment plan has been created and is shown below.

  stack path:                    /app/eu-north-1/stack.yml/eu-north-1
  stack name:                    stack
  stack region:                  eu-north-1
  operation:                     UPDATE

Parameters:

  + ParameterA:          (parameter will be created)
      current value:             <undefined>
      new value:                 valueA

  + ParameterB:      (parameter will be created)
      current value:             <undefined>
      new value:                 valueB

  - ParameterC:       (parameter will be removed)
      current value:             valueC
      new value:                 <undefined>
2021-03-17 08:42:51 +0200 [error] - /app/eu-north-1/stack.yml/eu-north-1 - An error occurred Error: Unsupported parameter operation: 'delete'
    at /Users/user/projects/xmevents/node_modules/@takomo/cli-io/dist/stacks/deploy-stacks/parameters.js:134:23
    at Array.map (<anonymous>)
    at Object.printParameters (/Users/user/projects/xmevents/node_modules/@takomo/cli-io/dist/stacks/deploy-stacks/parameters.js:125:10)
    at Object.confirmStackDeploy (/Users/user/projects/xmevents/node_modules/@takomo/cli-io/dist/stacks/deploy-stacks/deploy-stacks-io.js:160:48)
    at reviewChangeSet (/Users/user/projects/xmevents/node_modules/@takomo/stacks-commands/dist/stacks/deploy/steps/review-change-set.js:13:29)
    at /Users/user/projects/xmevents/node_modules/@takomo/stacks-commands/dist/stacks/deploy/transitions.js:60:22
    at executeStep (/Users/user/projects/xmevents/node_modules/@takomo/stacks-commands/dist/stacks/common/steps.js:30:22)
    at Object.executeSteps (/Users/user/projects/xmevents/node_modules/@takomo/stacks-commands/dist/stacks/common/steps.js:56:24)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Object.executeDeployContext (/Users/user/projects/xmevents/node_modules/@takomo/stacks-commands/dist/stacks/deploy/execute-deploy-context.js:90:27)

7 replies

Mika Fonsén

@mfonsen

Thank you for the latest releases. Logs are no more filled with throttling. Also the possibility to relay aws credentials to hooks is nice. So is the ability to refer to hook output from a resolver.

Some first impressions on these: It would be nice if AWS region would also be included with the credentials but that is easy to implement in a stack config. The output reading resolver was a bit surprising as it reads all output. My expectation was that it would only read last line of stdout, not all lines or stderr. But also this was easy to get around.

Henri Meltaus

@hmeltaus

Thanks for the feedback! I'll add a new option "exposeStackRegion" to control if stack's region should be exposed to command hook, would that work for you?

Mika Fonsén

@mfonsen

That would be great. But this is by no means critical. It is just a thing you learn when you try the feature for the first time.

Henri Meltaus

@hmeltaus

Yeah but it's easy to implement and useful feature so I'll do it like that.

You're right that getting all output from a command hook isn't always what is needed. One option would be to add an option to specify that only the last line should be collected. There could also be an option to specify to collect everything that is printed after some marker string until an optional end marker is found.

for example, collect everything after line "--- start ---" until output ends or end marker "--- end ---" is encountered

Mika Fonsén

@mfonsen

I was also wondering that does Cloudformation support multiline parametets but maybe there’s some other usecase that benefits from multiline output

My use case was to prepare a container image and return it’s ID in ECR. I had to forward output from docker build to a file to avoid it being included. A marker or last line would work well

Mika Fonsén

@mfonsen

Took latest Takomo into use. Used region expose and last line parameter features of the command resolver. They work very well. Thank you

Henri Meltaus

@hmeltaus

Excellent!

jonatcorus

@jonatcorus

Wow you've been quite productive in the last couple months. I might be blind, but I do not see the AWS organizations feature in the docs, was it removed?

Actually I see it in the code, so I guess the docs are missing is all...

In case you haven't seen this project https://github.com/org-formation/org-formation-cli

jonatcorus

@jonatcorus

How does takomo handle moving OUs and accounts around and can you rename accounts? I played around with using terraform and changing the name of an account makes TF want to delete/recreate the account...

Henri Meltaus

@hmeltaus

Hi, I haven't seen org-formation-cli. Looks interesting, I'll definitely have to check it out. Have you used it yourself?

I moved the docs to gitbook some time ago and haven't included org management features there yet. I should take the time to finalize the docs.

Henri Meltaus

@hmeltaus

Takomo lets you move OUs and accounts around. You can also rename accounts but that you have to do manually from AWS console. I believe there's no APIs for that.

Henri Meltaus

@hmeltaus

You can check org management docs from the old site https://takomo.io/docs/release/v3-4-0/docs/organizations/introduction

Mika Fonsén

@mfonsen

Does Takomo expose assumed roles or base credentials? With Takomo 3.11.1 I tried commandRole for the first time but the command resolver I have fails. By requesting caller identity from the command resolver it seems that I got the base credentials, not the temporary credentials from commandRole. Not 100% sure I did everything right, but I have checked the setup couple times already. The stacks that don't have such hooks/resolvers get deployed.

Mika Fonsén

@mfonsen

Oh, I got two things mixed up there. I checked the identity from a custom resolver that does:
const aws = require("aws-sdk")
const sts = new aws.STS()
...
const credentialManager =
await input.ctx.credentialManager.createCredentialManagerForRole(
props.role,
)
const credentials = await credentialManager.getCredentials()
console.log(
credentials,
await sts.getCallerIdentity().promise(),
)

command hook got the correct role

Maybe there's a better way of requesting credentials from a custom resolver?

Mika Fonsén

@mfonsen

My goal is to make Takomo run all parts of the deployment including build, data seeding and others through one command. This should speed up the deployment. Currently there are scripts commanding Takomo but that feels unnecessary

Mika Fonsén

@mfonsen

Maybe the iamRoles array the resolver can ask would be a clean solution?

Henri Meltaus

@hmeltaus

Hi, so are you implementing a custom resolver and wonder how to use credentials there?

Mika Fonsén

@mfonsen

Yes. I would like to use the assumed role the stack has (commandRole) to assume another role to push files to a bucket.

The custom resolver works without the commandRole if the base credentials given to Takomo are correct

Henri Meltaus

@hmeltaus

custom resolvers's resolve function gets one argument of type ResolverInput. The current stack is in stack property and from it you can get credential manager that has credentials that are bound to the stack (base credentials or credentials from commandRole).

So, in resolve function you can get stack credentials like so:

    resolve: async (input) => {

      const credentials = await input.stack.credentialManager.getCredentials()

      // ...rest of the resolve function
    }

Mika Fonsén

@mfonsen

Thank you! I will try that

Henri Meltaus

@hmeltaus

Built-in command resolver exposes the current stack's credentials if exposeStackCredentials is set to true. You can find the code here https://github.com/takomo-io/takomo/blob/ad91d05b3a3d5b9cb2fa86909d4ca501d1721ee9/packages/stacks-resolvers/src/cmd-resolver.ts#L53

Mika Fonsén

@mfonsen

Thank you. That worked great. I also got the resolver working that assumes subsequent roles:

        const credentialManager = await input.stack.credentialManager.createCredentialManagerForRole(
          props.role,
        )
        const credentials = await credentialManager.getCredentials()

Mika Fonsén

@mfonsen

Another challenge with the commandRole approach is MFA. I have a setup similar to the documentation (https://docs.takomo.io/configuration/aws-credentials#assuming-roles-that-require-mfa). Instead of starting Takomo for each role separately with --profile switch I would like to run all stacks in one go. This would require Takomo or some earlier tool to ask for the MFA once. Is there a way to get this use case working?

My first idea was to use profile (as account-a-admin in the docs) in stack group configuration as in Sceptre but that option does not seem to be supported. I could then let Takomo to ask for MFA once or write temporary credentials to ~/.aws/credentials using a tool such as awsume -o to avoid interactive prompts.
I then tried what happens if I give just the main account (manager in the docs) using the profile switch. This fails as Takomo will not ask MFA. Probably there's no way it could as there's no way of telling what mfa_serial should be used. Maybe there could be an option to specify the MFA serial. This seems to be supported in AWS SDK/STS.

Henri Meltaus

@hmeltaus

Good to hear that you were able to use credentials with resolvers.

Henri Meltaus

@hmeltaus

I'll check how to achieve your use case with MFA and get back to you later

Mika Fonsén

@mfonsen

Maybe if I'll give takomo a role that was already assumed and thus the MFA check had been completed. Using that role I Takomo would be able to assume other roles as they would not need to check for MFA anymore. There's one extra level role in the chain but that probably works.

Mika Fonsén

@mfonsen

Got it working with that one additional assume role layer.

This worked: awsume manager + takomo ... (no profile). Awsume takes care of the MFA. Takomo assumes roles as defined with multiple commandRoles. These do not require further MFAs.

This did not work: takomo ... --profile manager --> ProcessCredentialsProviderFailure: Profile manager did not include credential process. Takomo does ask MFA first before failing. Aws cli works when an target profile (account-a-admin) was defined.

The complexity of this solution goes into IAM and the developer experience is still good so I guess I'll go with this approach.

Henri Meltaus

@hmeltaus

hmm, I'm just about to do some testing myself. I'll see if there's something I can do to make it simpler.

Mika Fonsén

@mfonsen

The error I mentioned on Friday was probably about something different. Takomo started working with mfa after I removed unused profiles from ~/.aws/config

Mika Fonsén

@mfonsen

I had defined credential_process set in ~/.aws/config for unused profiles. These seemed to confuse Takomo (or aws-sdk). The error message ("Profile manager did not include credential process") would hint this was the case. Anyway now it works great.

Henri Meltaus

@hmeltaus

ok, the way AWS SDK handles credentials is quite complex. I should probably do some more digging and at least document the most common use cases and problems

Mika Fonsén

@mfonsen

That's true. Personally I wouldn't regard this as a problem in Takomo so I didn't verify this finding or try to replicate this with trivial aws-sdk example.

Where communities thrive

People

Repo info

Activity

ERROR

OTHER INFO