Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

Ravindra babu

@Ravindrababu99_twitter

@kapilt can we delete unused rds paramere-groups with custodian, dont see any direct filter for the same.

1 reply

nph007

@nph007

Hi all, I'm new in Cloud Custodian and I have plan to use with Google Cloud Platform. Currently I want to auto assign labels with sytax "creator:email_user" when they creating Compute Engine instance. However, I don't know how to filter a user's email value on audit logs.

policies:
  - name: compute-engine-no-labels
    resource: gcp.instance
    mode:
      type: gcp-audit
      methods:
        - beta.compute.instances.insert
    actions:
      - type: set-labels
        labels:
          "creator": authenticationInfo.principalEmail

!(https://ibb.co/m4TRkY4)

protoPayload: {
@type: "type.googleapis.com/google.cloud.audit.AuditLog"
authenticationInfo: {
principalEmail: "p******@*****.com"
}

This is logs structure on Google Cloud audit logs

nph007

@nph007

Am I misunderstanding or not looking for the correct keyword for this intent?

sl805

@sl805

Hi folks. Does anybody know where I can get cloud-custodian messgae template example ? I'm trying to send slack notifications directly without mailer usage.

aakshaik2

@aakifshaikh

https://github.com/jtroberts83/Cloud-Custodian/blob/3d0a45630b7e445989b0c3c937c5e9c6e7df5668/Policies/CleanupC7nDeployedPolicies/DeleteAllLambdaCWERules.ps1 - @jtroberts83 - You have not defined the function to delete the log group created by custodian? Or I am overlooking this.

5 replies

aakshaik2

@aakifshaikh

can we deploy cloud custodian using the terraform?

2 replies

codehead1997

@codehead1997

I wanted to deploy a policy as a config rule which filters all s3 buckets w/o bucket-logging, but when i am running the policy w/o specifying any mode it is running as expected but when i specify mode as config-rule it is not working as expected.
This is my policy-
policies:

      - name: s3-server-access-logging-check
        resource: s3
        mode:
          type: config-rule
          role: arn:aws:iam::xxxxxxxxx:role/s3-server-access-logging-check-role
        description: |
          filter s3 bucket for which server access logging is disabled.
        filters:
          - type: bucket-logging
            op: disabled

6 replies

codehead1997

@codehead1997

@/all How to filter all the s3 bucket for which object level logging is not enabled?

1 reply

pendyalal

@pendyalal

Hi everyone, Is there a way to find all amazonLinux1 EC2 instances with custodian policy, without ssm?

5 replies

srinath4u33

@srinath4u33

Hello Team, I m checking if there is any CC policy that will tag the S3 bucket which is created temporarily when uploading the CF template?

6 replies

jvoeller

@jvoeller

This may be a stupid questions but are there advantages of running c7n in the cloud instead of a local machine? Say for a project where only one person needs access to the environment

2 replies

kingoftheants

@kingoftheants3_twitter

Hey everyone,
I just started with CC and tried to do the Getting Started Guide for AWS. Sadly it fails with the ec2-require-non-public-and-encrypted-volumes policy. The lambda function doesn't trigger and it also says that the log group does not exist (

The specific log group: /aws/lambda/custodian-ec2-require-non-public-and-encrypted-volumes does not exist in this account or region.

)
I passed the region flag for my region and also tried it in the us-east-1 but with the same results. I used the example policy in the tutorial to allow the creation of log groups and log streams and created a role with lambda.amazonaws.com as a trusted entity. I'm not sure what else I could try. Anyone got an idea?

5 replies

codehead1997

@codehead1997

@/all I wanted to filter all the ec2 instances in which restricted ports open. Any port which is not specified in the policy will be restricted port.
policies:

  - name: ec2-predefine-port-check
    resource: ec2
    filters:
    - type: security-group
      key: "IpPermissions[].FromPort"
      op: not-in
      value: [80,8000,22,443]

This is what I came up with but it doesn't work. I guess it's because not-in operator doesn't works for two list operands. All I wanted to know is how can we check if all the elements of one list are in other list.

4 replies

Sonia Gurdian

@PendragonDay

I'm getting the following error/warning in running with c7n in AWS: 2021-05-04 12:02:21,016: custodian.iamaccess:WARNING no handler:handle_sts_externalid op:StringEquals key:sts:externalid values:('exampleexternalid',)

This is my policy:

 - name: iam_role_with_unauthorised_principals
  resource: iam-role
  region: us-east-1
  filters:
    - type: cross-account
      whitelist_from:
        url: "s3://myc7nbucket/exceptions/accounts.json"
        format: json
        expr: 'All[*]'
    - type: value
      key: "RoleName"
      op: not-in
      value_from:
        url: "s3://myc7nbucket/exceptions/aws-iam-role-/exceptions.csv"
        format: csv

accounts.json contains accounts whitelisted in the following format:

{
    "All": [
        "012345678911",
        "098765432100"
    ],
}

exceptions.csv contains RoleName list in the following format:

RoleName,
myRoleName1,
myRoleName2,

The policy works like a charm. The only issue is the WARNING. IS this a known issue? BTW this WARNING only comes up when the cross-account role policy includes an external id. I have tons of roles without external id and the warning does not come up for those.

3 replies

Andrey Romanov

@ansromanov

Hi, I have a short question. How can I filter out S3 buckets by name prefix, for example find all buckets starting with "personal-*" pattern?

2 replies

wallabyies

@wallabyies

Is there any documentation on usage for the lambda action "trim-versions"? I see this action mentioned in a few issues, but I'm not seeing documentation.

4 replies

kapilt

@kapilt:matrix.org

[m]

Yeah the docs thing is wierd

I think our doc builds might be hosed, that one is still on drone

jfricioni

@jfricioni

Does any documentation exist for running a validate and deploy for files in github? I know there is the deploy which walks through validating a single file but I'm looking to validate the entire stack of policies and also at the same time then deploy them if they pass. I'm using github actions to do it but any CD tool like Azure pipelines works if there is documentation for it

aakshaik2

@aakifshaikh

Is there a command or approach to find all resources that supports action: auto-tag-user instead of going through each resources schema or documentation.

2 replies

cloudymatt

@cloudymatt

Hi, I'm struggling with the ECR schema. I want to expire untagged images after 7 days. To identify those images, I'm using the example from the docs but I wanted to double check if my action looks right?

policies:
  - name: ecr-lifecycle-untagged-expire
    resource: aws.ecr
    description: |
      Expire untagged images
    filters:
      - type: lifecycle-rule
        state: False
        match:
          - selection.tagStatus: untagged
          - action.type: expire
          - type: value
            key: selection.countNumber
            value: 7
            op: less-than
    actions:
      - type: set-lifecycle
        state: True
        rules: 
          - selection.tagStatus: untagged
          - action.type: expire
          - description: Set by Cloud Custodian - expire untagged images
          - rulePriority: 0
          - type: value
            key: selection.countNumber
            value: 7
            op: greater-than
          - selection.countType: sinceImagePushed
          - selection.countUnit: days

1 reply

Michael Davis

@MichaelDavisTSN

Just added this comment to an old issue that is probably not the place to put it... should this be a new issue?
I have a use case for either a copy-related-tag or just a check of a security group's attachment to an elb. There are certain CIDRs that I want to allow for elb-attached sgs and not allow for all other sgs. It appears that attachments are not an attribute for security groups in the describe function. But I know that there is a security-group filter in the elb resource for cloud custodian, and an unused filter for security groups. So I'm hoping somewhere in there is the ability to look at a SG and determine whether it has an elb attached.

Michael Davis

@MichaelDavisTSN

why does this not work (no error, just returns no results when there is at least one sg with this value):

filters:
    - type: value
      key: SecurityGroups[].IpPermissions[].FromPort
      op: eq
      value: -1

3 replies

RajaKoppuravuri

@RajaKoppuravuri

Hello Team,
I wanted to terminate ec2 instnces if they have a public IP assgined explicitly and in-order to verify that, I am applying filters on the cloudtrail RunInstances event. But the filters are not working as expected. Here is the snippet of the policy

  ---

policies:
  - name: ec2-public-ip
    resource: aws.ec2
    description: >
      Event: RunInstances|
      Compliance:No Public Ip Address|
      Remediation: Delete|
    mode:
      type: cloudtrail
      events:
        - source: ec2.amazonaws.com
          event: RunInstances
          ids: responseElements.instancesSet.items[].instanceId
      timeout: 200
      delay: 40
      role: "custodian_role"
    filters:
    - or:
      - type: value
        key: detail.requestParameters.networkInterfaceSet.items[0].associatePublicIpAddress
        value: true
      - type: value
        key: detail.requestParameters.instanceType
        value: t2.micro
      - type: value
        key: region
        value: ap-south-1
    actions:
      - type: terminate

1 reply

deepthimm

@deepthimm

I'm using EC2 off hours policy to stop and start the machines automatically. There is a machine in our environment that remains stopped until someone manually starts it. We have tagged the machine (maid_offhours off=(M-F,17);tz=cet), so if someone forgets to stop it after using, it should automatically stop at 5PM. It works fine but it somehow starts the machine automatically after a few hours. How can I avoid it?

13 replies

jvoeller

@jvoeller

Hey everyone,
I got a question concerning the event filter for the cloudtrail mode.
For example when I wanted to filter instances based on the region I checked the event record for a RunInstances event in CloudTrail. There I found a key named awsRegion. When I tried to filter against that key no instance was selected. Looking at the documentation I found out the key I wanted was just region. Why is that and how do I find the right key in AWS?

15 replies

Jon Gilmore

@JonGilmore_gitlab

hey all! I'm looking for a way to limit the percentage of resources that a policy takes action on (to limit my blast radius). I did find max-resources-percent, but it's not clear to me if that will actually run the actions on the resources under that percentage or if it just stops it from running the action entirely. Any thoughts?

7 replies

Ravindra babu

@Ravindrababu99_twitter

@jtroberts83 is @kapilt Is there any way to filter out the security-groups having maximum number of rules (60 Egress)

3 replies

Shawn L

@slaphitter

Hello, I'm trying to write a tag-checker policy for EMR clusters and I've noticed that there seems to be… some kind of caching for lack of a better term. So for example if I am checking the value of Environment and I start with a valid value "staging" the cluster properly does not get picked up. If I change it to "stooging" then the policy trips properly. But when I change it back to "staging" and run the policy (this is all running from my local machine on the command line BTW) the policy still trips, and the resource.json file is showing the old/bad tag value. But if I describe-instance the value is correct. Has anyone else seen this?

10 replies

Rahul Vinod Sharma

@rahulsharma0810_twitter

   "settings": {
      "databaseFlags": [
        {
          "name": "default trace enabled",
          "value": "off"
        },
        {
          "name": "1222",
          "value": "on"
        }
      ],
}

Hello, I am trying to check the condition if default trace enabled is present and its value is on, does anyone have clue how to achieve it?

4 replies

Todd Stansell

@tjstansell

I know it's Friday afternoon, but noticed #6606 from a month ago to prepare for the 0.9.12 release. Any idea when the next release will come out?

Varun Chandak

@kintuparantu_twitter

Hello. I am trying to create a policy that checks CMK KMS policies for specific Actions and it's corresponding Principals, which I can define in filters. However, when I run the below policy, I get the resources.json mentioned below. Now Policy key in the resources.json contains the escaped JSON, which I can't figure out how to parse via JMESPATH.

Please suggest how do I parse that escaped JSON (or any other alternative).

policies:
  - name: only-root-user-principal-can-delete-cmk
    description: only account root user principal can delete CMK KMS
    resource: kms-key
    mode:
      role: arn:aws:iam::123456789012:role/c7n-role
      type: periodic
      schedule: "rate(1 day)"
    filters:
        - or:
          - type: cross-account
          - Policy: present

the output of resources.json is:

[
  {
    "KeyId": "91b948c7-00a8-48f3-a95a-051c6a9f8869",
    "KeyArn": "arn:aws:kms:us-east-1:123456789012:key/91b948c7-00a8-48f3-a95a-051c6a9f8869",
    "AWSAccountId": "123456789012",
    "Arn": "arn:aws:kms:us-east-1:123456789012:key/91b948c7-00a8-48f3-a95a-051c6a9f8869",
    "CreationDate": "2021-05-08T19:13:04.305000+05:30",
    "Enabled": true,
    "Description": "test",
    "KeyUsage": "ENCRYPT_DECRYPT",
    "KeyState": "Enabled",
    "Origin": "AWS_KMS",
    "KeyManager": "CUSTOMER",
    "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
    "EncryptionAlgorithms": [
      "SYMMETRIC_DEFAULT"
    ],
    "AliasNames": [
      "alias/test"
    ],
    "Tags": [],
    "c7n:MatchedFilters": [
      "AliasNames[0]",
      "[Policy|[0]]"
    ],
    "Policy": "{\n  \"Version\" : \"2012-10-17\",\n  \"Id\" : \"key-consolepolicy-3\",\n  \"Statement\" : [ {\n    \"Sid\" : \"Enable IAM User Permissions\",\n    \"Effect\" : \"Allow\",\n    \"Principal\" : {\n      \"AWS\" : \"arn:aws:iam::123456789012:root\"\n    },\n    \"Action\" : \"kms:*\",\n    \"Resource\" : \"*\"\n  }, {\n    \"Sid\" : \"Allow access for Key Administrators\",\n    \"Effect\" : \"Allow\",\n    \"Principal\" : {\n      \"AWS\" : \"arn:aws:iam::123456789012:role/some-role-name-here\"\n    },\n    \"Action\" : [ \"kms:Create*\", \"kms:Describe*\", \"kms:Enable*\", \"kms:List*\", \"kms:Put*\", \"kms:Update*\", \"kms:Revoke*\", \"kms:Disable*\", \"kms:Get*\", \"kms:Delete*\", \"kms:TagResource\", \"kms:UntagResource\", \"kms:ScheduleKeyDeletion\", \"kms:CancelKeyDeletion\" ],\n    \"Resource\" : \"*\"\n  }, {\n    \"Sid\" : \"Allow use of the key\",\n    \"Effect\" : \"Allow\",\n    \"Principal\" : {\n      \"AWS\" : \"arn:aws:iam::123456789012:role/some-role-name-here\"\n    },\n    \"Action\" : [ \"kms:Encrypt\", \"kms:Decrypt\", \"kms:ReEncrypt*\", \"kms:GenerateDataKey*\", \"kms:DescribeKey\" ],\n    \"Resource\" : \"*\"\n  }, {\n    \"Sid\" : \"Allow attachment of persistent resources\",\n    \"Effect\" : \"Allow\",\n    \"Principal\" : {\n      \"AWS\" : \"arn:aws:iam::123456789012:role/some-role-name-here\"\n    },\n    \"Action\" : [ \"kms:CreateGrant\", \"kms:ListGrants\", \"kms:RevokeGrant\" ],\n    \"Resource\" : \"*\",\n    \"Condition\" : {\n      \"Bool\" : {\n        \"kms:GrantIsForAWSResource\" : \"true\"\n      }\n    }\n  } ]\n}"
  }
]

6 replies

Rahul Vinod Sharma

@rahulsharma0810_twitter

Does anybody have working example of GCP-periodic Pubsub pull policy

properties:
  environment:
    type: object
  execution-options:
    type: object
  labels:
    type: object
  max-instances:
    type: integer
  memory-size:
    type: integer
  network:
    type: string
  schedule:
    type: string
  service-account:
    type: string
  timeout:
    type: string
  trigger-type:
    enum:
    - http
    - pubsub.  -<
  type:
    enum:
    - gcp-periodic
  tz:
    type: string
required:
- schedule
- type

Currently scheduler not able to trigger cloud function (Permission Denied) because of OIDC .

deepthimm

@deepthimm

Varun Chandak

@kintuparantu_twitter

@kapilt:matrix.org do we have support for KMS Custom Key Store? I am trying to write a policy and I cannot see any documents regarding it.

policies:
  - name: cloud-hsm
    description: cloud-hsm
    resource: cloudhsm-cluster  
    filters:
      - and:
        - type: value
          key: CustomKeyStores[].ConnectionState
          value: DISCONNECTED
          op: equal

3 replies

pendyalal

@pendyalal

This message was deleted

4 replies

Pradeep Reddy

@prareed_twitter

Hello, I am trying to configure mailer and seeing below error:
(custodian) [creddy@ip-10-180-1-102 cloud-custodian]$ c7n-mailer --config mailer.yml --run
2021-05-11 09:46:02,101 - custodian-mailer - INFO - Downloading messages from the SQS queue.
Traceback (most recent call last):
File "/home/creddy/cloud-custodian/custodian/bin/c7n-mailer", line 11, in <module>
load_entry_point('c7n-mailer', 'console_scripts', 'c7n-mailer')()
File "/home/creddy/cloud-custodian/cloud-custodian/tools/c7n_mailer/c7n_mailer/cli.py", line 271, in main
processor.run()
File "/home/creddy/cloud-custodian/cloud-custodian/tools/c7n_mailer/c7n_mailer/sqs_queue_processor.py", line 105, in run
for sqs_message in sqs_messages:
File "/home/creddy/cloud-custodian/cloud-custodian/tools/c7n_mailer/c7n_mailer/sqs_queue_processor.py", line 46, in next
AttributeNames=['SentTimestamp']
File "/home/creddy/cloud-custodian/custodian/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/creddy/cloud-custodian/custodian/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ReceiveMessage operation: Access to the resource https://queue.amazonaws.com/ is denied.

I have granted SQS full access to the role that I am using.

Below is my mailer.yml:
queue_url: https://sqs.us-east-1.amazonaws.com/************/custodian
role: arn:aws:iam::*:role/custodian
from_address: creddy@*.com

1 reply

Seshadhri

@Seshadhri

re public subnet, you would do a subnet filter on the ec2 resource, depends in part on how you determine a public subnet, you could just query out on public ip associated to the ec2 instance.

policies:

name: SecCCEC2PublicRule
resource: ec2
mode:
type: config-rule
role: LambdaAdminTEST
filters:
- PublicIpAddress: present
  actions:
- type: notify
  transport:
  type: sns
  topic: arn:aws:sns:eu-central-1:123456789102:test-enforcement
  region: eu-central-1
  template: default
  priority_header: 1
  subject: " SecCCEC2PublicRule: Created EC2 Violation"
  violation_desc: |
  " EC2 instance created which has a public IP address."
  action_desc: |
  "Action Taken: Notification Only: The violating EC2 instance needs to be terminated as it violates Zuora security policy."

@kapilt:matrix.org for the above policy even though the policy is executing am not the getting the notification to the mentioned SNS topic. Also I have subscribed my email address to this SNS topic. Am not getting email notifications as-well. plz help in fixing this issue..

4 replies

Varun Chandak

@kintuparantu_twitter

For resource: firehose, it has minimal details about the Kinesis data streams such as ARN, etc. However, in resource: kinesis, we have the details of kinesis data streams, such as shards, KMS keys, status, etc. My question is how do I fetch the details of resource: kinesis and use it in filters of resource: firehose in a single policy ? Similar use case like this would be really helpful.

crickyyy1

@crickyyy1

Hi All, I have a question regarding the "state-age" filter when used against EC2 resources. I'm testing a policy that will mark and subsequently terminate an EC2 if it's remained in a "stopped" state for 14+ days, but it does not appear that the filter is picking up all applicable instances. The only obvious difference I can see in the resources.json file is that the EC2s being picked up by the filter have a "StateTransitionReason" with a value of something like "User Initiated <date/time>", while instances with a "StateTransitionReason" of "Instance Initiated" do not have a date/timestamp. Additionally, stopping an EC2 with a simple Custodian policy gives the "StateTransitionReason" key an empty pair of quotes in resources.json.

Any ideas on what accounts for the inconsistencies in the values?

3 replies

Andrey Romanov

@ansromanov

Hello, I'm trying to implement temporary exclusions for managing of EC2 instances. The idea is to add users possibility to add custom tag "custodian_postpone: true" to bypass any custodian operations on instance. And I want to automatically remove this tag after some period of time, for example, after 48 hours. And I stucked with this part, maybe you could propose some tips how to implement tag management. Or maybe propose better solution...

1 reply

shaikhsajid18

@shaikhsajid18

I opened an issue for question 1: #2964

@sshvetsov I was also looking for policy to enable termination protection for CFN. Can you please tell me if you were able to test and its working for you ?
I am using below policy:

policies:
  - name: cfn-enable-termination-protection
    resource: cfn
    mode:
      packages: [boto3, botocore, urllib3]
      type: periodic
      schedule: "cron(40 15 * * ? *)"
    description: |
      Policy to enable TerminationProtection for CFN
    filters:
      - type: value
        key: StackStatus
        op: in
        value:
          - CREATE_COMPLETE
          - UPDATE_COMPLETE
      - type: value
        key: EnableTerminationProtection
        op: not-equal
        value: true
      - 'tag:cfn-termination-protect': absent
    actions:
      - type: tag
        tags:
          cfn-termination-protect: enabled
      - type: set-protection
        state: True
      - <<: *notify-var
        subject: "Enabled termination protection for CloudFormation Stack"
        violation_desc: "Enabled termination protection for CloudFormation Stack"

Issue 1: I get an error if I don't add tag.

[ERROR] ClientError: An error occurred (ValidationError) when calling the UpdateStack operation: No updates are to be performed.
Traceback (most recent call last):
File "/var/task/custodian_policy.py", line 4, in run
return handler.dispatch_event(event, context)
File "/var/task/c7n/handler.py", line 165, in dispatch_event
p.push(event, context)
File "/var/task/c7n/policy.py", line 1143, in push
return mode.run(event, lambda_ctx)
File "/var/task/c7n/policy.py", line 526, in run
return PullMode.run(self)
File "/var/task/c7n/policy.py", line 317, in run
results = a.process(resources)
File "/var/task/c7n/tags.py", line 411, in process
_common_tag_processer(
File "/var/task/c7n/tags.py", line 120, in _common_tag_processer
raise error
File "/var/lang/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "/var/task/c7n/resources/cfn.py", line 165, in process_resource_set
_tag_stack(client, s, add=tags)
File "/var/task/c7n/resources/cfn.py", line 189, in _tag_stack
client.update_stack(
File "/var/task/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/task/botocore/client.py", line 676, in _make_api_call
raise error_class(parsed_response, operation_name)

Issue 2: If I add tag, problem is it gets pushed to all resources of CFN.

Can anyone suggest how to get it working without adding extra tags to CFN Stack. Thanks.

Jason Reeves

@jare19

Hello, I was trying to get a regex filter on a list of alias for the resource kms-key. I have been banging my head on it for a while, the best I could come up with is this:

resource: aws.kms-key
filters:
  - type: value
    key: AliasNames
    op: in
    value_type: swap
    value_regex: ^((?!alias\/(aws\/|[A-Z]{2,10}[-][a-zA-Z]{3,63}-cmk)).)*$

This filter only works when I use a regular value type -value: alias/aws/ssm and NOT the regex value type value_regex, when I use the latter, it errors out and says c7n.exceptions.PolicyValidationError: Missing 'value' in value filter
any thoughts on how I could use a regex value here for the list type of aliases in kms-key?

7 replies

aakshaik2

@aakifshaikh

Is there a way to know the health check on Cloud Custodian- meaning what policy has stopped working?

1 reply

Zeke Marffy

@zmarffy

Is it possible to use notin with glob or regex when filtering? AKA “doesn’t match any of these globs”.

Where communities thrive

People

Repo info

Activity