Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

Michael Davis

@MichaelDavisTSN

or a way to use value filter to match a missing attribute (users)

4 replies

Mitushi Pradeep

@mpradeep23

Hi @kapilt , I have a scenario where I have to compare if rds tag- "application" is equal to its rds subnet group's application tag. Could I get an example of how to compare those two tags?

Nagarjuna

@Avuthu_gitlab

Hi Team, How do I add a tag to a instance as part of tag compliance check with no Value assigned. Want end user to add value to for the instance to be compliant, but I would like to pre-populate the tag key's.

policies:
- name: ec2-tag-compliance-mark
  resource: ec2
  mode:
      packages: [boto3, botocore, urllib3]
      type: cloudtrail
      events:
        - RunInstances
  description: |
    Find all EC2(non-ASG) instances that are not conformant
    to tagging policies, and tag them for stoppage in 1 hour.
  comments: |
    Your EC2 instance does not have all the required TAGS on it and will be stopped after 1 hour.
  filters:
    - "tag:aws:autoscaling:groupName": absent
    - "tag:custodian-tag-compliant": absent
    - or: *tag-compliance-filters-mark
  actions:
    - type: tag
      key: custodian-tag-compliant
      value: "no"
#    - type: tag
#      key: AppName
    - type: tag
      key: custodian-stop-notification-sent
      value: "yes"
    - type: mark-for-op
      op: stop
      hours: 1

5 replies

Mike

@mikejgray

I'm looking through some of the aws.account docs and it looks like an iam-user reference slipped in: https://cloudcustodian.io/docs/aws/resources/account.html#credential

Steve G

@thed000d

Hi - I'm trying to Copy AMIs and receiving the error:

File "/home/kwelch/mypython3/lib/python3.8/site-packages/c7n/resources/ami.py", line 217, in process
    Description=self.data.get('description', image['Description']),
KeyError: 'Description'

Is this because the AMI does not have a description in AWS?

2 replies

Jon Gilmore

@JonGilmore_gitlab

@kapilt seems like you've stated it's possible to use copy-related-tagsto copy a tag value from one key to another on itself (also saw this in a github issue). Is there any chance you can further describe how this is possible? Once I get a working example of it, I'd be happy to contribute back to documentation examples because it seems this question has been asked quite a few times

you can use copy-related-tag to copy from an instance to itself

Geraldo Magella

@gmagella-ca

<vent> I hate how GCP is behind on everything, rendering wonderful things like CloudCustodian so limited </vent>

1 reply

tomarv2

@tomarv2

question on GCP, I just started to deploy it, the deployment in on functions, does this mean each function needs to be deployed in all the projects or can a function be deployed in one project and it can check all projects?

1 reply

Itamar Dori

@doriitamar

Hey! A question about the value filter, is something like that achievable? Doesn't seem to be filtering right though no errors are thrown

          - type: value
            key: '"c7n:AccessKeys"[?Status == ''Inactive''] | length(@)'
            value: '"c7n:AccessKeys" | length(@)'
            value_type: expr

(the match-filter: and for the access-key filter, didn't filter all users that all their access keys are inactive, so I was trying to work around this issue)

rachgupt

@rachgupt

Hi, i am trying to write a policy to check on diagnostic setting on Azure , where logs are sent to storage account . Looks like this feature is not present in azure.cosmosdb type?

'''custodian1:~/policies$ cat az-find-cosmosdb-with-logs-enabled.yml
policies:

  - name: az-find-cosmosdb-with-logs-enabled
    resource: azure.cosmosdb
    filters:
     - type: diagnostic-settings
       key: logs[?category == 'DataPlaneRequests'][].enabled
    value: True
    op: in
    value_type: swap'''

custodian1:~/policies$ custodian validate az-find-cosmosdb-with-logs-enabled.yml
2020-09-14 11:12:59,389: custodian.commands:ERROR Configuration invalid: az-find-cosmosdb-with-logs-enabled.yml
2020-09-14 11:12:59,389: custodian.commands:ERROR policy:az-find-cosmosdb-with-logs-enabled has unknown keys: value,op,value_type

Misja Pronk

@bollebeer

Hello, i am trying to filter Azure Data Factory based on its LinkedServices has anyone tried this before?

Vishnu-Lakkimsetty-E3640

@Vishnu-Lakkimsetty-E3640

Hello team, Would like to know if there is an ability to pass/interpolate dates (typically the date after 4 days from today dynamically) into notify subject header?

1 reply

satvan23

@satvan23

@kapilt I'm trying to run a report but see this error. So what am I doing wrong ?

(custodian) [ccustodian@usohc7n901 policies]$ custodian report -s s3://dev-entsvc-cloud-custodian-logs/policies/ -p security-groups-unused -v > terminated.csv
2020-09-14 18:50:27,877: c7n.policies:WARNING Policy pattern "security-groups-unused" did not match any policies.
2020-09-14 18:50:27,877: custodian.commands:WARNING Warning: no policies matched the filters provided.
2020-09-14 18:50:27,878: custodian.commands:WARNING Filters:
2020-09-14 18:50:27,878: custodian.commands:WARNING Policy name filter (-p): security-groups-unused
2020-09-14 18:50:27,878: custodian.commands:WARNING Available policies:
2020-09-14 18:50:27,878: custodian.commands:WARNING (none)
2020-09-14 18:50:27,880: custodian.commands:ERROR Error: must supply at least one policy

(custodian) [ccustodian@usohc7n901 policies]$ custodian report -s s3://dev-entsvc-cloud-custodian-logs/policies/ -p unused-sg.yml -v > terminated.csv
2020-09-14 18:50:56,581: c7n.policies:WARNING Policy pattern "unused-sg.yml" did not match any policies.
2020-09-14 18:50:56,581: custodian.commands:WARNING Warning: no policies matched the filters provided.
2020-09-14 18:50:56,581: custodian.commands:WARNING Filters:
2020-09-14 18:50:56,581: custodian.commands:WARNING Policy name filter (-p): unused-sg.yml
2020-09-14 18:50:56,581: custodian.commands:WARNING Available policies:
2020-09-14 18:50:56,581: custodian.commands:WARNING (none)
2020-09-14 18:50:56,583: custodian.commands:ERROR Error: must supply at least one policy

(custodian) [ccustodian@usohc7n901 policies]$ head -5 unused-sg.yml
policies:

name: security-groups-unused
resource: security-group
filters:
- unused
  (custodian) [ccustodian@usohc7n901 policies]$

5 replies

DigeratiDad

@digeratidad

@jtroberts83 Hey, is it possible to use CC to notify on AWS SSM agent removal?

Kapil Thangavelu

@kapilt

yes, see ssm filters on ec2

your looking for ping times over a threshold

though that could have other origins (instance stopped), etc depending on threshold

shirleymorillo

@shirleymorillo

Hello There, Can you share how are you deploying cloud custodian policies to different environments using a ci tool?

francoislagier

@francoislagier

Hello, I've been trying to debug a weird (and I'm sure naive) issue for the last few hours so I decide to ask here before I throw my computer against the wall :D

I'm trying to write a simple policy to detect ALB which no requests in the last 14 days.

I wrote the following policy (form the metadata.json):

{
  "description": "Detect any ALB with a 0 connection count",
  "filters": [
    {
      "and": [
        {
          "days": 14,
          "missing-value": 0,
          "name": "RequestCount",
          "op": "lte",
          "period": 86400,
          "statistics": "Sum",
          "type": "metrics",
          "value": 0
        }
      ]
    }
  ],
  "name": "alb-unused",
  "resource": "app-elb"
}

but I'm still getting the following ALB in my resources.json:

{
    "LoadBalancerArn": "arn:aws:elasticloadbalancing:{REMOVED}",
    "DNSName": "{REMOVED}.elb.amazonaws.com",
    "CanonicalHostedZoneId": "{REMOVED}",
    "CreatedTime": "2019-12-19T06:38:39.410000+00:00",
    "LoadBalancerName": "{REMOVED}",
    "Scheme": "internet-facing",
    "VpcId": "{REMOVED}",
    "State": {
      "Code": "active"
    },
    "Type": "application",
    "AvailabilityZones": [
      {
        "ZoneName": "{REMOVED}",
        "SubnetId": "{REMOVED}",
        "LoadBalancerAddresses": []
      },
      {
        "ZoneName": "{REMOVED}",
        "SubnetId": "{REMOVED}",
        "LoadBalancerAddresses": []
      },
      {
        "ZoneName": "{REMOVED}",
        "SubnetId": "{REMOVED}",
        "LoadBalancerAddresses": []
      }
    ],
    "SecurityGroups": [
      "{REMOVED}"
    ],
    "IpAddressType": "ipv4",
    "c7n.metrics": {
      "AWS/ApplicationELB.RequestCount.Sum": [
        {
          "Timestamp": "2020-09-01T00:10:00+00:00",
          "Sum": 0.0,
          "Unit": "Count"
        },
        {
          "Timestamp": "2020-09-14T00:10:00+00:00",
          "Sum": 115306.0,
          "Unit": "Count"
        },
        {
          "Timestamp": "2020-09-13T00:10:00+00:00",
          "Sum": 0.0,
          "Unit": "Count"
        },
        ...
      ]
    }
  }

It worked once I removed the period and use eq with the value 0 (better design overall) but I would still like to understand the way the metrics filter is working. Here is my question, If there is a period, should ALL the metrics match the filter to be true or just of the metrics?

Any ideas or thoughts on this? Thanks in advance

myoung34

@myoung34

i want to look for s3 buckets created without default encryption
It doesnt look like i can do this with cloud-trail (correct me if im wrong) because the encryption is a separate call (PutBucketEncryption) from the CreateBucket cloudtrail event

Is there a way to filter s3 buckets by creation date instead? If i can filter by ones created in the last x minutes I can achieve this with that + bucket-encryption filter

7 replies

deepthimm

@deepthimm

Is there a way to generate one email from several policies written in a yml file? for example, I would like to check the tag compliance of ec2 instances, s3 buckets, rds etc. I do not want to receive 3 different emails but one consolidated email of all the services.

1 reply

David Barranco

@dbarranco

@kapilt if you have a chance, ptal at:

https://github.com/cloud-custodian/cloud-custodian/pull/6074#discussion_r487785723

I left some Q's for you :sweat_smile:

17 replies

tomarv2

@tomarv2

GCP/c7n experts, double-checking again.. I noticed we have over 200GCP projects, some with just one or two resources, so when I am deploying functions.. it needs to be deployed in each and every project. nothing like maybe at folder level... anyone who has done the deployment can you share your deployment method or ideas..

10 replies

myoung34

@myoung34

anyone know what the magic to GetBucketEncryption is? I have allow GetBucketEncryption to * and get Message: An error occurred (AccessDenied) when calling the GetBucketEncryption operation: Access Denied Bucket: on CreateBucket event with bucket-encryption filter and it makes no sense

i even tried:

        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketEncryption"
            ],
            "Resource": [
                "arn:aws:s3:::*",
                "arn:aws:s3:::*/*"
            ]
        },

no dice

Geraldo Magella

@gmagella-ca

Unfortunately, I don't have a single policy working on GCP

6 replies

vkuchi

@vinaykuchibhotla

is it possible to encrypt all the ebs volumes including the ones in use with encrypt-instance-volumes action? I am able to encrypt the volumes attached to an instance but I need to encrypt 100+ volumes in an account and ending up with list index out of range error running this policy -

  - name: encrypt-unencrypted-ebs
    resource: ebs
    filters:
     # - type: instance
     #   key: tag:Name
     #   value: some-instance-name
      - Encrypted: false
    actions:
      - type: encrypt-instance-volumes
        key: alias/aws/ebs

2 replies

Christian Yarros

@CYarros10

how do you guys divide up your IAM permissions for your cloud custodian policies? do you have all relevant IAM permissions in a single role, do you divide it up by service so that the s3 custodian policies use the s3 role? etc.

4 replies

pendyalal

@pendyalal

Hi all, So I've a S3 metrics policy with filters for all storage types. I'm getting only metrics for StorageType: StandardStorage , but not for all. Can anyone has any idea on this?

2 replies

theotherothermatt

@theotherothermatt

Hey all! working on a policy to auto-remediate kms keys that do not have rotation enabled. The lambda is failing because it runs into a key that is set to deletion.

KMSInvalidStateException: An error occurred (KMSInvalidStateException) when calling the EnableKeyRotation operation: arn:aws:kms:us-east-1::key/99 is pending deletion.

Is there a filter we can use to disregard deletion pending KMS keys?

tdiener01

@tdiener01

Hey everyone, creating a policy using PHD as the mode in order to send AWS Health events to an SQS queue (somewhat similar to what @myoung34 shows here ) I’d like to grab issue and scheduledChange events that are open or upcoming. I’d think the following policy would work, though it fails to actually create, despite passing the dry-run. Am I missing some required property under mode?

policies:
  - name: health-event-notify
    resource: account
    description: |
      Health event notifications to SQS.
    mode:
      type: phd
      categories: ['issue','scheduledChange']
      statuses: ['open','upcoming']
      role: arn:aws:iam::{account_id}:role/cloud-custodian-phd
    actions:
    - type: notify
      template: default
      priority_header: '2'
      subject: testing the c7n mailer
      to:
        - name@domain.com
      transport:
        type: sqs
        queue: <sqs queue>

7 replies

srikanthcs

@srikanthcs

Hi Guys, hi @kapilt ,

Can we process the events from target accounts in a central account and trigger the custodian lambda in the central account which then sts into target account and performs the respective actions ?
Having the custodian lambdas in all the target accounts is a lot of infrastructure. we have 1k+ accounts and increasing . That will be a lot of maintenance
Is this centralized design feasible ? Or is there already a way to run custodian in a centralized way ?

6 replies

Monica

@seemonicago

What is the right way to send emails from a child account SNS topic to a parent account SQS queue? I'm receiving the base64 encoded string as the payload on the parent SQS. Do I have to setup mailer in each child account and mail directly from their instead of centralizing?

7 replies

saurabh hirani

@saurabh-hirani

Hi everyone - I just started with custodian and I am amazed by all it can do. To start out I am trying to dump instance ids with a specific tag and sending it to a webhook. However, I cannot get the instance_id in the query string. Does query-params in webhook block not allow jmespath queries?

policies:
  - name: my-first-policy
    resource: aws.ec2
    filters:
      - "tag:target_tag": present
    actions:
      - type: webhook
        url: http://127.0.0.1:8080
        query-params:
          account_id: account_id
          region: region
          instance_id: "[].InstanceId"
          policy_name: policy.name

The webhook url has None for instance_id http://127.0.0.1:8080?account_id=12345678&region=us-east-1&instance_id=None&policy_name=my-first-policy

even though there are 31 matches

Got it - had use resource.InstanceId

David Lin

@davidclin

Is it possible for c7n to detect ec2 instance Status Checks?

nitro

@nitrocode

few questions about automatically attaching a managed policy to new roles.

lookingat this policy: https://cloudcustodian.io/docs/aws/examples/iamsetrolepolicy.html

but after running this policy, i see a lot of managed aws iam roles that come up which i want it to ignore. so anything with a path containing /aws-service-role.

i also wonder, could this be done using a cloudtrail event on new iam roles ? cloudtrail seems to be the best way to go but it's nebulous to me how to locally test out policies.

what do you all recommend ?

19 replies

tomarv2

@tomarv2

can any share if they have a repo for GCP which is public, trying to see some autotagging stuff..

jfcottrell

@jfcottrell

I'm running a metrics filter on CloudWatch and my understanding is that the last datapoint logged is what will be returned. What I'm wondering is, is there any way to query the last N data points in CW using Cloud Custodian? Specifically I'm trying to see if an EMR cluster is idle. I can see if it's idle right now but, ideally, I'd like that it's been idle for several days. My current policy is:

            - name: emr-idle
              resource: emr
              filters:
                  - type: metrics
                    name: IsIdle
                    value: 1
                    op: equal

tomarv2

@tomarv2

@jfcottrell this is what I use for my kinesis:

    filters:
      - type: metrics
        name: GetRecords.Bytes
        statistics: Sum
        days: 7
        op: lte
        value: 0

mblonsky

@mblonsky

hey there, i'm having trouble getting slack integration to work. Email notifications through SNS are working just fine. I'm trying to figure out what I am doing wrong. I don't currently have a token defined in my mailer.yml. I played around with that but for a simple webhook it was my understanding a token was not needed. I know I am doing something wrong so any assistance would be appreciated. Thanks

policies:

name: tag-policy
resource: aws.ec2
filters:
- "tag:Policy": absent
  actions:
- type: notify
  slack_template: slack
  slack_msg_color: danger
  to:
  - https://hooks.slack.com/services/*****************************
     subject: "EC2 Resource missing tag <Policy> - not tagged properly
    transport:
    type: sqs
    queue: https://sqs.us-west-2.amazonaws.com/123456789/cloud-custodian-mailer

mblonsky

@mblonsky

thanks in advance for any assistance

Farrukh Sadykov

@farrukh90

Hello everyone,

I am new to custodian. I was able to use custodian with Run option. Is there anyway I can offboard it? I mean remove from lambda? or should I delete manually?

1 reply

Where communities thrive

People

Repo info

Activity