Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

Samarth Shivaramu

@s_samarth03_twitter

Hi,

I've deployed a Cloud Custodian policy across all the AWS environments to assess CIS benchmark compliance. All actions in the CC policy are set to notifying the event owner and the InfoSec team. Is there a way to use the email address verified in SES in a separate AWS account to be defined in the mailer.yml for all of the AWS environments?

Here's an example of the mailer.yml file:

queue_url: <SQS URL>
role: <IAM role ARN>
from_address: security@company.com
region: us-east-1
ses_region: us-west-2

The aforementioned "from_address" in the mailer.yml file has already been verified in the root account of an AWS organization. I'd like to use the same email address to send emails in all of the AWS environments. I setup a sending authorization policy allowing the other AWS root accounts to have access to the identity "security@company.com". But I still see the error:

error: An error occurred (MessageRejected) when calling the SendRawEmail operation: Email address is not verified. The following identities failed the check in region US-EAST-1:

Should the same "from_address" be individually verified in all the AWS accounts where c7n-mailer is deployed?

1 reply

Brandon Lashmet

@blashmet

Hi, for policies that detect missing tags, does CloudCustodian report only those resources that have had a tag in the past, or will it return resources that are missing tags and have never had one in the past?

3 replies

I ask because this API call will return resources missing tags, but only if they have been tagged before.

https://docs.aws.amazon.com/powershell/latest/reference/items/Get-RGTResource.html

Jorge O. Castro

@castrojo

Hi everyone! Just reposting what I said on the list here if you're not subscribed, we had our first community meeting yesterday! Here are the notes: https://github.com/cloud-custodian/community/discussions/15

1 reply

We also now have a nice shiny YouTube channel, we'll be posting the meeting videos there as well as new content so feel free to subscribe and tell a friend! https://www.youtube.com/channel/UCdeXCdFLluylWnFfS0-jbDA

aakshaik2

@aakifshaikh

Question regarding auto-tag-user: Custodian can take action to do 2 things- tag with owner, and principalId. Can I also add userName?

aakshaik2

@aakifshaikh

Adding the userName to the tag- makes it easy for us to map the principalid to the role. Otherwise we have dig that into the cloudtrail as what role does that principalid belongs to. We deploy things using the pipeline 99% of the time- and it does not have owner - because different roles are deploying the stuff in pipeline. So if we see the name in the tag- instead of principalid- its easy for us to understand it was deployed using the pipeline. And this avoid us the mapping from principalid to role...

2 replies

jvoeller

@jvoeller

Hey, is there any infographic of some sort that displays the service coverage that can be achieved with c7n? The docs include a reference for all services but is there some sort of summary that can be referenced?

2 replies

aakshaik2

@aakifshaikh

what is the schema to look for updated AWS common filters and AWS common actions. I am not refering to the one like- custodian schema aws.s3.actions.auto-tag-user.....in general

Jamison Roberts

@jtroberts83

https://cloudcustodian.io/docs/filters.html would show the generic filters for you

Not a way to do that within the CLI that I am aware of

Kamerabuilt

@Kamerabuilt

Hi all, I am trying to install CloudCustodian (on ubuntu) via : python3 -m venv custodian (works fine), but then the following fails: source custodian/bin/activate

5 replies

giving me no such file or directory

Roman Panov

@romanpanov993

Hello,
Is it possible to add resource attribute to tag automatically?
For example, I need to add Glue job name to tag, but I can't find the way to do it by custodian.

I tried something like this:

  - type: tag
    key: GlueJobName
    value: requestParameters.jobName

Kamerabuilt

@Kamerabuilt

Can Cloud Custodian integrate with aws-vault, or Hashicorp Vault, or similar secret managers to avoid leaving the aws credentials in the .aws/credentials file or in the command line history?

3 replies

udomsak

@udomsak

Is custodian current (0.9.12, also branch 0.8) support for RDS offhour ? I try to reach out with issues:

after i try looking in rds.py code it just run register method that it differences from ec2.py that have decorator with function to handling the action, Is it correct? (i am new, for cloud custodian and does not understand code as well, sorry if i wrong)

https://github.com/cloud-custodian/cloud-custodian/blob/b94c0760d500c8f91087264100c8f758a3862d89/c7n/resources/rds.py#L247

Can't make filter offhour work for RDS event try to add tag (regard above issues). my policy that i try

policies:
  - name: rds-offhour
    resource: aws.rds-cluster
    actions:
      - stop
    filters:
      - type: offhour
        default_tz: utc
        weekends: true
        offhour: 15
        opt-out: true
      - "tag:custodian_control": present

3 replies

Kamerabuilt

@Kamerabuilt

Finally got the policy to enable-s3-encryption-on-new-buckets, however, and after hours of troubleshooting, I confirmed that it wont work as long as a new Trail (under CloudTrail) is created. Any reason why the default CloudTrail events (under History) are not enough when we set type: cloudtrail (and I did find the S3 bucket creation under the default Cloudtrail logs)?

8 replies

esque

@PankajMoolrajani

How can i run cloud custodian from python scripts instead of running from command line, any examples ?

3 replies

kapilt

@kapilt:matrix.org

[m]

Cloudtrail on is pretty foundational, in any well managed setup. You can set one up on a n org level, or have a custodian policy do it for you on an account basis.

2 replies

CurtisAndersenSysdig

@CurtisAndersenSysdig

I am trying to make a auto tag system

    mode:
      type: cloudtrail
      role: arn:aws:iam::{account_id}:role/custodian-lambda-role
      events:
        -  CreateCluster
    filters:
      - tag:AutoTag_Creator: absent
    actions:
      - type: auto-tag-user
        tag: AutoTag_Creator

Is there something that I am doing wrong because If i create an ecs it does not auto tag the user

4 replies

Kim Wai

@Thida_00001_gitlab

Hello, I am new to Cloud Custodian. I am trying to set true to disableAPiTermination attribute for a AWS ec2 instance from Cloud Custodian' action. I can't find any such action to modify the attribute. Is this even possible to do so with CC?

1 reply

Liz Acosta

@liz-acosta

eee! hello frens!!!

took me a minute to muster up the courage to start talking in here, but here we go!

sooo i am trying to write a policy that filters on s3 buckets with a specific tag and then performs an action on them. have confirmed bucket is indeed tagged, however, upon execution of the policy, no resources are returned. can you help?

i'm a total n00b so i hella appreciate y'all's super smart and experienced brains!

6 replies

Yair Fried

@yfried

I’m trying to create an auto-tag for all aws resources in all regions in a single account.
Do I have to duplicate the following for all resource types and all regions?

policies:
  - name: tag-compliance
    resource: ec2
    mode:
      type: cloudtrail
      events:
        - RunInstances
      tags:
        CreatorName: custodian
        Project: Custodian
      execution-options:
        region: us-east-1
    filters:
      - tag:CreatorName: absent
    actions:
      - type: auto-tag-user
        tag: CreatorName

2 replies

Yair Fried

@yfried

Is there a doc about the roles and policies required to use the “cloudtrail” mode?
I’m getting:

botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda.

14 replies

Yair Fried

@yfried

How do I purge the lambda resources created by custodian?

16 replies

Jorge O. Castro

@castrojo

~30 minutes until the next community meeting! We'll be here: meet.google.com/mii-evqh-esh and notes will be here: https://hackmd.io/g5DfTqN2RRqxbwT_NonczA -- which we'll push to the github community repo afterwards, hope to see you there! All skill levels welcome!

pentagonal-proboscis

@pentagonal-proboscis

Hi there,
I am currently running a policy to disable unrotated keys and see the following output:

Which implies that the policy has run successfully. However, the key is not disabled - is this a known issue?

Policy is as follows:

policies:
 - name: disable-unrotated-iam-keys
   resource: iam-user
   filters:
   - type: credential
     key: access_keys.active
     value: true 
   - type: credential
     key: access_keys.last_rotated
     value_type: age
     value: 90
     op: gt
   actions:
    - type: remove-keys
      matched: true
      disable: true

The policy is being run with administrator permissions.

17 replies

manvik4u

@manvik4u

Hey Guys! I have question on c7n-org, where do I create config file? the documentation says something about automating it through subcommand, but I don't see any instructions, neither that link is active anymore( cloud-custodian/cloud-custodian#2420 ) .. I am looking for AWS only

7 replies

Liz Acosta

@liz-acosta

hello everyone!

i am trying to create a policy in event mode. i would like to use the CreateBucket event to trigger my policy. this policy filters for unencrypted s3 buckets and takes the notify and tag actions

upon executing my policy, cloud custodian successfully provisioned the appropriate lambda function. i am using terraform to provision my test infrastructure and i have verified terraform apply was successful

however, the function does not seem to be triggering upon CreateBucket. i have reviewed the cloudtrail logs for s3 resources and have verified the event was created. however, the lambda logs reveal no invocation. i am assuming this is because GetLambdaFunctionRecommendations is returning a OptInRequiredException

my policy is here

policies:
  - name: list-all-not-encrypted-s3-notify
    resource: aws.s3
    description: |
      Lists all S3 Buckets that are not encrypted. Notifies resource owner of violation and required remediation.
    comments: |
      Lists all S3 Buckets that are not encrypted, notifies resource owner, tags for Deletion when `list-all-unecrypted-s3s-delete` is pulled.
      mode:
    mode:
      type: cloudtrail
      role: arn:aws:iam::<account>:role/service-role/lacosta-s3-lambda-role-5b9982tb
      events:
        - CreateBucket
    filters:
      - type: bucket-encryption
        state: False
    actions:
      - type: tag
        tags:
          "tag:toDelete": "okay"
      - type: notify
        template: default
        priority_header: 1
        subject: "S3 Bucket - Unencrypted - [custodian {{ account }} - {{ region }}]"
        violation_desc: |
          Your S3 Bucket is not encrypted. You need to encrypt your S3 Bucket.
        action_desc: |
          "Notification"
        to:
          - liz@stacklet.io
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/<account>/lacosta-c7n-mailer-practice
          region: us-east-1

21 replies

Kamerabuilt

@Kamerabuilt

Regarding the "actions, type: encrypt-keys" (s3 service), the documentation mentions that it "Will scan all keys in the bucket for unencrypted keys and by default remediate them such that they are encrypted", do we mean by that: "scanning all keys in the bucket(s) for unencrypted S3 objects"?

3 replies

shrabonidey

@shrabonidey

Hello, I am having one requirement for my aws account, I need to send out an email to the account owners with the details of all the instances count project wise(tag is present for each project in the instances). Help me with the policy. Example : Project :XYZ is having 6 instances out of which 2 are stopped and 4 are running. Project ABC is having 7 instances out of which 2 are stopped and 5 are running. Note - The details should be in the body of the email in a tabular format/column wise. Thanks!!

11 replies

shrabonidey

@shrabonidey

6 replies

jay.r123

@jay.r123:matrix.org

[m]

is it possible to trigger a cloudcustodian policy from a lambda policy and send along 2-3 parameters?

3 replies

jay.r123

@jay.r123:matrix.org

[m]

Basically I'm using a policy to pass some info to a lambda through invoke-lambda action in the c7n policy. I want to use that info to create tags for certain ec2 instances, so I want to know if I can create those tags using a c7n policy and pass the info from the lambda to that policy so the tags are created with the right info on the right ec2 instances

1 reply

Naidu Kandulapati

@naiduklr936

@all - I'm working on the following ec2 tag compliances policy. I'm looking for two tags, Billing and CostCenter, either of one tag have to exist, CostCenter tag exists with anyvalue it can skip the action, if the CostCenter not exist and Billing tag exists, it needs to be look for the specific value from the given values. It is working fine if my policy has CostCenter tag, but when I add the Billing tag it is able to create the lambda but when I trigger the lamda it is not filtering as expected. Any help on this is much appreciated.

policies:
- name: tag-compliance-check
   resource: ec2
   description: Tag compliance check for EC2
   filters:
   - "tag:CostCenter": absent
   - type: value
      key: "tag:Billing"
      op: not-in
       value:
        - 11111
        - 22222
  actions:
   - stop

4 replies

Naidu Kandulapati

@naiduklr936

Did anyone created the regular expression for the onhour and offhour values? If not following is the regex I created for offhour that might used for finding the valid tag values are added to the offhour tag values

off hour tag value

(off=\[\([M|T|W|H|F|S|U](-[M|T|W|H|F|S|U])?,([1]?[0-9]|2[0-3])\)(,\([M|T|W|H|F|S|U](-[M|T|W|H|F|S|U])?,([1]?[0-9]|2[0-3])\))*?\];tz=(cst|est|pst)$)|(^off=\[\([M|T|W|H|F|S|U](-[M|T|W|H|F|S|U])?,([1]?[0-9]|2[0-3])\)(,\([M|T|W|H|F|S|U](-[M|T|W|H|F|S|U])?,([1]?[0-9]|2[0-3])\))*?\]$)|(^off=\[\([M|T|W|H|F|S|U](-[M|T|W|H|F|S|U])?,([1]?[0-9]|2[0-3])\)\]$)|(^off=\([M|T|W|H|F|S|U](-[M|T|W|H|F|S|U])?,([1]?[0-9]|2[0-3])\)$)|(off=(\([M|T|W|H|F|S|U](-[M|T|W|H|F|S|U])?,([1]?[0-9]|2[0-3])\);tz=(cst|est|pst)$))

3 replies

Rahul Vinod Sharma

@rahulsharma0810_twitter

  {
    "kind": "sql#user",
    "etag": "b94aa31d9797874a723fd7f8eacafe79566dd129cf72d4c32403d7b903b3da24",
    "name": "root",
    "host": "%",
    "instance": "mysql-testest",
    "project": "cc-devsecops",
    "c7n:sql-instance": {
      "kind": "sql#instance",
      "state": "RUNNABLE",
      "databaseVersion": "MYSQL_5_7",
      "settings": {
        "authorizedGaeApplications": [],
        "tier": "db-f1-micro",
        "kind": "sql#settings",
        "availabilityType": "ZONAL",
        "pricingPlan": "PER_USE",
        "replicationType": "SYNCHRONOUS",
        "activationPolicy": "ALWAYS",
        "ipConfiguration": {
          "authorizedNetworks": [],
          "ipv4Enabled": true
        },
        "locationPreference": {
          "zone": "us-central1-f",
          "kind": "sql#locationPreference"
        },
        "databaseFlags": [
          {
            "name": "max_user_connections",
            "value": "500"
          }
        ],
        "dataDiskType": "PD_SSD",
        "maintenanceWindow": {
          "kind": "sql#maintenanceWindow",
          "hour": 0,
          "day": 0
        },
        "backupConfiguration": {
          "startTime": "04:00",
          "kind": "sql#backupConfiguration",
          "location": "us",
          "backupRetentionSettings": {
            "retentionUnit": "COUNT",
            "retainedBackups": 7
          },
          "enabled": true,
          "binaryLogEnabled": false,
          "transactionLogRetentionDays": 7
        },
        "settingsVersion": "40",
        "storageAutoResizeLimit": "0",
        "storageAutoResize": true,
        "dataDiskSizeGb": "10"
      },
      "etag": "1e9800e8af981c51405a251f4269d2a4d17c390fd449cef67b20b56d99ff9100",
      "ipAddresses": [
        {
          "type": "PRIMARY",
          "ipAddress": "34.72.17.151"
        }
      ],
      "serverCaCert": {
        "kind": "sql#sslCert",
        "certSerialNumber": "0",
        "commonName": "C=US,O=Google\\, Inc,CN=Google Cloud SQL Server CA,dnQualifier=3f361884-7aa9-40fd-998f-d05116c63bac",
        "sha1Fingerprint": "fe9330cb4f76341769f375d5bd6c47fff745b27e",
        "instance": "mysql-testest",
        "createTime": "2021-05-12T10:24:02.093Z",
        "expirationTime": "2031-05-10T10:25:02.093Z"
      },
      "instanceType": "CLOUD_SQL_INSTANCE",
      "project": "cc-devsecops",
      "serviceAccountEmailAddress": "p532446579504-xddgls@gcp-sa-cloud-sql.iam.gserviceaccount.com",
      "backendType": "SECOND_GEN",
      "selfLink": "https://sqladmin.googleapis.com/sql/v1beta4/projects/cc-devsecops/instances/mysql-testest",
      "connectionName": "cc-devsecops:us-central1:mysql-testest",
      "name": "mysql-testest",
      "region": "us-central1",
      "gceZone": "us-central1-f"
    }
  }]

How can I access resource.selfLink From the Json in webhook ?

Will resource.c7n:sql-instance.selfLink work ?

jay.r123

@jay.r123:matrix.org

[m]

@liz-acosta: sorry for not replying earlier, but yes, that's the intended behavior. Do you know if this is possible to do with cloud custodian?

Naidu Kandulapati

@naiduklr936

Hello @all - Can cloud custodian untag the EC2 instance after certain time using mark-for-op ? Something like this

actions:
  - type: mark-for-op
    op: untag
    tags: ['UnTagKey']
    hours: 1

3 replies

Ryan Ryke

@rryke

hi everyone ... new to custodian. running into an issue where items are in the docs but say they are not allowed in custodian

for example

  - name: delete-iam-role
    resource: aws.iam-role
    filters:
      - type: marked-for-op
        tag: custodian_status
        op: delete
    actions:
      - delete
        force: true

does not run

its cranky at the force portion

4 replies

Ryan Ryke

@rryke

but

custodian schema aws.iam-role.actions.delete

shows that its allowed

Liz Acosta

@liz-acosta

@jay.r123:matrix.org i don't think that is possible, but i am not certain

@jtroberts83 or @ajkerrigan do you know?

1 reply

pentagonal-proboscis

@pentagonal-proboscis

hi all - is there a policy out there which allows me to query any AWS global accelerators I have configured?

1 reply

vishal parmar

@vishal_parmar_gitlab

I am new to cloudcustodian and wanted to see an example of retrieveing a tag value from aws acount. and want to use that value and apply it to other resources for a tag

1 reply

lkolchin

@lkolchin

Hello everybody,
I can see that you can filter by ssm status with something like this:

policies:
  - name: ec2-ssm-check
    resource: ec2
    filters:
      - type: ssm
        key: PingStatus
        value: Online

My question is - can you get a list of instances that aren't registered with ssm (for whatever reason - either a lack of ssm agent installed or missigng iam role)?

2 replies

Where communities thrive

People

Repo info

Activity