Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

Jorge O. Castro

@castrojo

If you have a burning issue/PR/problem and want to drop by you're more than welcome! We'll start at meet.google.com/mii-evqh-esh at the top of the hour!

Kapil Thangavelu

@kapilt

cloud custodian in the aws 2021 reinforce keynote.. from hbo max and how they use https://youtu.be/H3LTjVWSQ6g?t=2657

membra

@membra

Hi everyone, sorry, was trying to find an answer to this before but didn't manage so far,
How to prevent a policy to be executed on same resource again?
I was thinking of tagging a resource in the actions, and then including an additional condition in filters to skip resources with a certain value in that tag. But if same resource is being actioned by another policy, the tag gets overwritten (I mean I could use different tags but it complicates things in terms of later reporting).
That's current state of the policy for reference:

policies:
# Custodian CW log group retention period set
  - name: cloudwatch-set-log-group-retention
    description: |
      Cloud Custodian log group retention period setting
    comment: |
      Cloud Custodian log group retention period setting
    resource: log-group
    mode:
      type: periodic
      role: *execution-role
      schedule: *schedule-rate
    filters:
     - type: config-compliance
       eval_filters:
        - type: value
          key: ResultRecordedTime
          value_type: age
          value: 30
          op: less-than
       rules:
        - cfg-as2-g-loggroup-retention-period-check
       states:
        - NON_COMPLIANT
     - not:
       - type: value
         key: *custodian-tag-key-with-tag
         op: contains
         value: "Retention period set"
    actions:
     - type: retention
       days: 90
     - type: tag
       key: *custodian-tag-key
       value: "Retention period set to default 90."

Kristina Trump

@KristinaTrump_twitter

@kapilt , can you please guide me on how below policy works , even if I give values as true or false below is not returning any output,policies:

- name: account-check-config-services
  resource: account
  region: us-east-1
  filters:
    - type: check-config
      all-resources: false
      global-resources: false
      running: false

1 reply

bobrich

@bobrich

quick question on EKS support (i've been away from cloud custodian for a spell). the commit notes indicate that its capturing the same info that Config captures, which unfortunately excludes the access control settings for the api endpoint. is that the case with custodian then as well?

manvik4u

@manvik4u

Hello all,
Question on custodian mailer, can we include multiple-queues from different accounts in queue_url under mailer.yml?

2 replies

Ryan Ash

@ryanash999

Question...
AWS Macie has a finding that helps to identify the follow:

Policy:IAMUser/S3BucketReplicatedExternally
Data replication was enabled and configured to replicate objects from the bucket to an Amazon Web Services account that isn't part of your organization. An organization is a set of Macie accounts that are centrally managed as a group of related accounts through AWS Organizations or by Macie invitation.

Has anyone written, or know how to write, a similar custodian policy?

3 replies

membra

@membra

Hello everyone, I was looking into report command. What I was expecting of it is the information "which policy was executed, against which resource, when"
When I actually ran this

custodian report -s s3://custodian-reports-path C:\Users\custodian-pipeline\regional\cw-set-log-group-retention.yml --format csv --profile rs --field retentionInDays=retentionInDays

I got this

"logGroupName","creationTime","retentionInDays"
"TestLogGroup-LLG396XE-xfkMazGmyn3P","1630537609188","60"

Seems like it is running only against resources.json files, thus reporting which resources were actioned only (so no info on when and which policy was executed).
Am I missing something and this information can be got from report or is it something we need to do separately on top?

Thank you

3 replies

Marcin

@Baraszkiewicz_gitlab

Hi, I'm trying to write a policy to resize ECS services on schedule like described here.
But I want to customize schedule base on tag values like this: off=(M-F,19);on=(M-F,7).

The problem is that some of those characters are not allowed in ECS tags:
Some tags contain invalid characters. Valid characters: UTF-8 letters, spaces, numbers and _ . / = + - : @.

Any thoughts how can I solve this issue?

2 replies

Mark Kenny

@ynnekkram

Hi, I am writing an AWS tag compliance policy. Is it possible to declare * on all resources when running this policy or will I require a policy for each resource?

2 replies

Mahesh Bhatt

@B99Bhatt_twitter

Hi, is there a way we can collect aws eks cluster matrics using cloudcustidian for monitoring purpose?

3 replies

kryptik-one

@kryptik-one

Hello. Looking for a way to check for publicly available databases(rds, dynamodb) in aws and notify.

1 reply

Ihar Vauchok

@vauchok

Hi all,
is there any chance to get round creation of the policy for each tag? I don't want to overwrite the tag value if it exists. I just want to create the tag if it doesn't exist.

an example of the policy:

name: ec2-attach-default-tags
mode:
  type: periodic
  schedule: "rate(1 day)"
  role: role_arn
resource: ec2
comments: Attach default tags to EC2 instances.
filters:
  - or:
    - "tag:one": absent
    - "tag:two": absent
    - "tag:three": absent
  - not:
    - "State.Name": terminated
actions:
  - type: tag
    tags:
      one: "test1"
      two: "test2"
      three: "test3"

as I see, if any of the tags are absent, the policy will create/write each of them.

any ideas?

Thanks

2 replies

esque

@PankajMoolrajani

How can i get all s3 buckets that are not communicating over https. Found this example, but this is to set the policy.

policies:
  - name: force-s3-https
    resource: s3
    actions:
      - type: set-statements
        statements:
          - Sid: "DenyHttp"
            Effect: "Deny"
            Action: "s3:GetObject"
            Principal:
              AWS: "*"
            Resource: "arn:aws:s3:::{bucket_name}/*"
            Condition:
              Bool:
                "aws:SecureTransport": false

4 replies

Todd Stansell

@tjstansell

Where is one supposed to go to see the calendar of upcoming community meetings? I've looked on the community github repo but that only seems to have meeting notes from previous meetings and i don't see anything on cloudcustodian.io ... how do new folks find out about such things or access the next meeting's agenda?

6 replies

kryptik-one

@kryptik-one

I have a use case where i need to allow a one time exception to instances using unapproved ami. Normally we will terminate the the ec2 but want to make an exception this time.

1 reply

Mike

@mikejgray

I'm working on an Azure policy to catch unused load balancers. I'm pretty sure the metric filter works, but in the case of my test load balancer, it has had nothing pass through it at all so I'm not sure what the API returns. Has anyone been able to confirm if you need a second filter for that?

policies:
  - name: az-notify-inactive-loadbalancer
    resource: azure.loadbalancer
    filters:
      - type: metric
        metric: PacketCount
        op: le
        aggregation: total
        threshold: 0
        timeframe: 168 # 1 week

3 replies

Filip Chyla

@fchyla

Did anyone manage to add/create a GCP Cloud Run resource/module?

@liz-acosta

Hello!

I am currently writing an intro to c7n tutorial for beginners. While I have successfully executed a policy in cloudtrail mode before, I can't recall which permissions are required for the role configured in the policy file in order for the policy to work.

The policy in question:

policies:
  # List S3 buckets that are not encrypted
  - name: tutorial-policy
    resource: aws.s3
    description: |
      Lists all S3 Buckets instances that are not encrypted.
    mode:
      type: cloudtrail
      role: my-test-role
      events:
        - CreateBucket
    filters:
      - type: bucket-encryption
        state: False
    actions:
      - type: tag
        tags:
          "not-encrypted": " "

Therefore, the policies I need to attach to my-test-role would be:

Read/write access to S3 bucket (to tag)
Read/write access to CloudWatch logs (for the CreateBucket event)
And ... ?

There must be another one I am missing because the c7n policy is not yet tagging unencrypted buckets upon creation.

Thank you for your help!

2 replies

cleo2525

@cleo2525

Hello, I'm creating a policy that checks for Lambda Functions running deprecated runtimes. I was able to get my filter working by referencing a csv in an s3 bucket. However, I was wondering if there's a way to tag the Lambda Function with the "Phase 2 EOL" date, which is contained in the csv.

Ultimately I'm hoping to maintain the csv by running a Lambda function that queries AWS's site and pulls the list of deprecated runtimes and their dates. It would be nice to dynamically tag the Lambda Functions so I can include not only the Runtime version in my notification email, but also the EOL date.

policies:
  - name: lambda-deprecated
    resource: aws.lambda
    filters:
      - type: value
        key: "Runtime"
        op: in
        value_from:
          url: s3://my-bucket-name/lambda-deprecated-runtimes.csv
          format: csv2dict
          expr: Identifier
    action:
      - type: tag
        tag: "deprecated_lambda_runtime_date"
        value_from:
          url: s3://my-bucket-name/lambda-deprecated-runtimes.csv
          format: csv2dict
          expr: "End of support phase 2 start"

3 replies

Beau Witter

@beau-witter

Hi all, I have the following cloud c7n scheduled policy that has the undesired result of running against every region in AWS, even the ones that are otherwise completely unused. I am hoping it is simply a configuration issue on me part and was hoping for some feedback to solve that. It is my understanding that policies are global and so wouldn't need to be checked within every region so hopefully there is something I can change to support that way instead of running within every region.

policies:
  - name: cis-1-22-scheduled-remediate
    resource: iam-policy
    region: us-east-1
    mode:
      <<: *lambda-config
      type: periodic
      schedule: cron(0 * * * ? *)
    filters:
      # Exclude AdministratorAccess policy
      - not:
        - type: value
          key: PolicyName
          value: AdministratorAccess
      - type: has-allow-all
    actions:
      # ~~~~~~~~ Excluded from post as these should not be pertinent to the issue ~~~~~~~~~

13 replies

anergiti

@anergiti

Hello,
Im running into a super odd issue.
Im trying to provision a policy for all my account's regions, but unfortunately, its only provisioning on a single region (probably the default). why? how can I make it iterate through all regions? thanks!

this is my policy:

policies:
  - name: root-user-login-detected
    resource: account
    description: |
      Notifies Security and Cloud Admins teams on any AWS root user console logins
    mode:
       type: cloudtrail
       role: arn:aws:iam::12345:role/lambda_allow_for_custodian
       events:
          - ConsoleLogin
    filters:
       - type: event
         key: "detail.userIdentity.type"
         value_type: swap
         op: in
         value: Root
    actions:
        - type: notify
          violation_desc: blblabla
          slack_msg_color: danger
          to:
            - https://hooks.slack.com/services/ABC
          transport:
            type: sqs
            queue: https://sqs.us-east-1.amazonaws.com/12345/custodian-sqs

this is the command I'm using:

custodian run -s out --region all root_login.yml

2 replies

Espinoza

@Espinoz04005337_twitter

I am working on a security group policy. I am using the ingress filter to look for any IPs with a subnet mask of /24. Is there any way of doing this without having to use specific IPs addresses AND a subnet mask? I can only get it to work if I use specific IPs and "/24".

filters:
        - type: ingress
          Cidr:
                  value: "0.0.0.0/24"

3 replies

Beau Witter

@beau-witter

15 replies

Samarth Shivaramu

@s_samarth03_twitter

I'm working on a project where the requirement is to host Cloud Custodian on an EC2 instance. One of the requirements of the project is to not have the AWS credentials stored on the EC2 instance. Is it possible to execute Cloud Custodian policies when the AWS credentials don't exist in ~/.aws/credentials folder? Is it possible to store the access keys in secret manager in an AWS account and have Cloud Custodian retrieve them before execution or is there a better way to handle such a deployment?

3 replies

Jorge O. Castro

@castrojo

Hey everyone, if you're not on the google group, Governance as Cloud Day is coming along, we have our speakers and hope to have the schedule out by the end of this week: https://hopin.com/events/governance-as-code-day-with-cloud-custodian-hosted-by-stacklet

The event is virtual and is 100% free, no KubeCon ticket required!

sadik13

@sadik13

Hello, Team,
If i deploy and trigger the cloud custodian function email in the Dev environment it has showing the email content after triggering the email as below way:
example:-
devtestproject - us-east-1
violation: The CMK keys are deleted and disable please check it.

If i deploy and trigger the cloud custodian function email in the sandbox/test environment it has showing the email content after triggering the email as below way(sandboxtestproject- us-east-1 instead of this showing as - us-east-1):
example:-
us-east-1
violation: The CMK keys are deleted and disable please check it.

so please suggest how to fix this one in the AWS custodian.
Greatly appreciated!!!

2 replies

David Hodge

@davezen1

Is it possible to create one policy for all regions of s3. Or would you create a separate policy per region?

5 replies

David Hodge

@davezen1

I can create a policy and filter for aws.firewall resources. However, if I try to use the tag action I get the InvalidParameterException. Is this a known issue?

jdelee

@jdelee

I'm trying to group AMIs by a regex of the ImageLocation attribute, but I'm getting a validation error
Policy

policies:
  - name: mark-old-amis
    resource: ami
    filters:
      - type: image-age
        days: 50
      - unused
      - type: reduce
        group-by:
          key: "ImageLocation"
          value_type: "string"
          value_regex: "^[0-9]+\/(.+)[0-9]{10}$"
        discard: 1

Error

AttributeError: 'ReduceFilter' object has no attribute '_validate_value_regex'
    self._validate_value_regex(self.group_by['value_regex'])
  File "/src/c7n/filters/core.py", line 868, in validate
    f.validate()
 File "/src/c7n/policy.py", line 1083, in validate
    [p.validate() for p in collection]
  File "/src/c7n/policy.py", line 60, in <listcomp>
    [p.validate() for p in collection]
  File "/src/c7n/policy.py", line 60, in load
    collection = policy_load(options, fp, validate=validate, vars=vars)
  File "/src/c7n/commands.py", line 51, in _load_policies
    command(config)
  File "/src/c7n/cli.py", line 360, in main
    main()
  File "/usr/local/bin/custodian", line 5, in <module>
Traceback (most recent call last):
AttributeError: 'ReduceFilter' object has no attribute '_validate_value_regex'
    self._validate_value_regex(self.group_by['value_regex'])
  File "/src/c7n/filters/core.py", line 868, in validate
    f.validate()
  File "/src/c7n/policy.py", line 1083, in validate
    [p.validate() for p in collection]
  File "/src/c7n/policy.py", line 60, in <listcomp>
    [p.validate() for p in collection]
  File "/src/c7n/policy.py", line 60, in load
    collection = policy_load(options, fp, validate=validate, vars=vars)
  File "/src/c7n/commands.py", line 51, in _load_policies
    command(config)
  File "/src/c7n/cli.py", line 360, in main
    main()
  File "/usr/local/bin/custodian", line 5, in <module>
Traceback (most recent call last):

4 replies

aakshaik2

@aakifshaikh

policies:

- name: aws-ecr-repository-publicly-accessible
  resource: aws.ecr
  comment: |
    This policy identifies any publicly accessible ECR Repository.
  filters:
    - type: cross-account
      everyone_only: true

   This is not identifying the right public repo. In the debug I found that it only counts the private ones and don't count the public ones. Example it says ECR of 17. Filtered 17 of 0. And 17 is all private ECR. Not sure why custodian is not considering the ones in the public tab. please advise.

Goal is to identify the ECR repo exposed to the world.
@ajkerrigan

1 reply

Landon Dale

@landale4

So I’ve been deploying cloudformation stacks with c7n org run-scripts locally for a while now, but I need to centralize it for other team members. I’ve been trying to run it in AWS CodeBuild and continue to get this error
An error occurred (AlreadyExistsException) when calling the CreateChangeSet operation: ChangeSet awscli-cloudformation-package-deploy-1632100442 cannot be created due to a mismatch with existing attribute Description
Has anyone else seen this, and if so, have you found a solution?

Seshadhri

@Seshadhri

@jtroberts83 @kapilt @ajkerrigan

Trying to execute a cloud custodian policy for RDS DB instance that has public accessibility as "true".

Am trying to execute this in config mode. Below are the policies that am using.

1st policy only identifies the "non-complaint" RDS DB instance that has public accessibility as "true" and reports back to config rule:

policies:

name: RDSPublicExposurePolicy
resource: rds
mode:
type: config-rule
role: LambdaRole
filters:
- type: value
  key: DBInstanceStatus
  value: "available"
- PubliclyAccessible: true

2nd policy takes only the "non-compliant" resource(s) as input from the above first policy and performs the action as mentioned below:

policies:

name: enforce_RDSPublicExposurePolicy
resource: rds
mode:
type: config-rule
role: LambdaRole
filters:
- type: config-compliance
  rules: [custodian-RDSPublicExposurePolicy]
  states: [NON_COMPLIANT]
  actions:
- type: set-public-access
  state: false

Now the issue here is,

Both the 1st and 2nd policies are not working.
There seems to be sync issues between AWS Config and RDS in such a way that, when RDS goes for a modification it takes some time and config is unable to capture the exact change and I guess it is returning the older result itself. Even when config captures the configuration change correctly, it is giving incorrect results.
Waited for a long time to see the changes in config, tried executing this multiple times and still the results are in-correct.
In some cases when i see the public accessibility as "true" for a resource in config timeline, then also AWS config projects the resource as compliant.

Requesting your help on this issue. From cloud custodian policy perspective, i feel there are no errors.

Wanted to understand, can we introduce some delay when config goes and looks on RDS DB instance state, once the modification is done, so that it might give us the correct result.

Please provide your inputs as-well on this issue.

Thanks !!

2 replies

Michael Tracey

@michael-tracey

Trying to improve my templates (particularly the slack template) for guard-duty alerts. I understand that {{ event }} is available and I can get the full json block into the slack message, but I have been unable to extract particular values from it. I've tried the simple:
{{ event['title'] }}
parsing it out w/ jinja2:

              "title":"Guard-Duty Title",
              "value":"{{ event  |  selectattr('title', 'undefined') | map(attribute='title') | list  }}"
            },

and even assuming it's a string, and tojson first:

              "title":"Guard-Duty Title",
              "value":"{{ event | tojson |  selectattr('title', 'undefined') | map(attribute='title') | list  }}"
            },

but nothing seems to work. Does anyone have any guidance on this, or a good slack template for guardduty I could learn from?

4 replies

Alex R.

@areifert

Hello all. Is there any way to either combine policies, or take the output of one policy and feed it into another policy? Use case: my org is trying to identify EC2 instances that are using older gp2 EBS volumes, and obtain the EC2 instance name (denoted by the "Name" tag). We are currently accomplishing this using the AWS CLI, by calling aws ec2 describe-volumes, taking the getting the instance IDs from the Attachments field, and then calling aws ec2 describe-instances with the instance IDs.

2 replies

aakshaik2

@aakifshaikh

@ajkerrigan - I am writing a cis policy for v1.4.0 - 2.1.2 . Is there a way I can straight check for conditional value from the bucket policy- SecureTransport: false? I am thinking of 2 elements here- 1. look for s3 bucket that does not have any bucket policy and 2. where bucket policy is present look for securetransport:false. If either is True- we can say https is not enforced. Please help me with this.

- name: cis-s3-does-not-enforce-https
  resource: aws.s3
  comment: |
    CIS Amazon Web Services Foundations v1.4.0 (2.1.2). Identify s3 where https is not enforced via bucket policy.
    This policy looks for two elements- 1) Does bucket have a bucket policy? 2) Have Bucket Policy and bucket policy does not include the condition of where "aws:SecureTransport"
    is false. If either is true, bucket is not enforcing https.
  filters:
    - or:
      - type: value
        key: Policy
        value: empty
      not-in:
        - type: has-statement
          statements:
            - Effect: Deny
              Action: s3:GetObject
              Principal: '*'
              Condition:
                Bool:
                  "aws:SecureTransport": false

1 reply

Espinoza

@Espinoz04005337_twitter

Trying to exclude an account number or id from my findings. Basically, I'm pulling the account from the accounts.yml file. The first filter is for a security group that works as expected. The policy doesnt work properly together though. Am I missing something?

resource: security-group
filters:
  - type: ingress
    Ports: [5439, 5432]
    Cidr:
      op: glob
      value: "30*/10"
  - type: value
    key: "account_id"
    op: ne
    value: "XXXXXX"

1 reply

Espinoza

@Espinoz04005337_twitter

resource: security-group
filters:
  - type: ingress
    Ports: [1433, 5432]
    Cidr:
      op: glob
      value: "30*/10"
  - type: value
    key: "{account_id}"
    op: ne
    value: "XXXXXXXXX"

5 replies

mchidambaram1990

@mchidambaram1990

how can we find out which custodian version we installed?

1 reply

manvik4u

@manvik4u

Hello!,

We are using C7n mailer to send notifications to our cloud admin group on policy violations. We have a requirement to add account owners and few other users in those notifications based on account number. Is there a way I can achieve this OOTB, basically I want to change my 'to' list to include additional email addresses based on the account number?

Maybe by fetching value from by CSV stored in s3 or entry from dynamo db table.

@jtroberts83 | @kapilt | @ajkerrigan

9 replies

aakshaik2

@aakifshaikh

# 5.1 Ensure no Network ACLs allow ingress from 0.0.0.0/0 to remote server administration ports (Automated)
- name: cis-networkacl-restrict-admin-ingress
  resource: aws.network-acl
  comment: |
    CIS Amazon Web Services Foundations v1.4.0 (5.1). 
  filters:
  - type: event
    key: cidrBlock:
    value: 0.0.0.0/0
  - type: event
    key: PortRange
    value: [22,3389]

1 reply

network acl does not support ingress and egress...does this policy looks OK for 5.1 CIS benchmark 1.4.0 @ajkerrigan

anergiti

@anergiti

Hello,
If a policy is not installed as Lambda, what is the best approach to run it periodically?
And can I set a "notify" action type (slack) for a non lambda policy?

3 replies

jfdoube

@jfdoube:matrix.org

[m]

Hi, question here. Is it possible to group notification, e.g. I want to notify the owner of 20 instances with 1 email containing the results ?

3 replies

anergiti

@anergiti

where can I find documentation for how to set an argument on a violation_desc? e.g - {{user}}

khapp

@khapp

Is there a way to do a cross-account for iam-role to check for specific cognito-identities? We currently have a policy that is checking for iam-role cross account and it is flagging a trust policy that includes a cognito-identity that is within its own account. We are trying to find a way to filter this out.

1 reply

Mahesh Bhatt

@B99Bhatt_twitter

Hi, is there a way to get an aws inventory for all the account under an org using custodian?

9 replies

KVInventoR

@KVInventoR

Is there any way to deploy CC jobs over terraform to aws account?
I found: https://pypi.org/project/c7n-terraform/ but be honest, didn't find good examples which will give me possibility to rewrite my CC jobs to be deployed over terraform

Where communities thrive

People

Repo info

Activity