Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

DigeratiDad

@digeratidad

^ @jtroberts83 - any thoughts on how that filter would look?

18 replies

Chris Ramsey

@cramseyio

Hi! I am curious if aws.cfn resource supports mark-for-op action?

I notice when i run custodian schema aws.cfn it is not listed in the actions list:

aws.cfn:
  actions:
  - auto-tag-user
  - copy-related-tag
  - delete
  - invoke-lambda
  - invoke-sfn
  - notify
  - post-finding
  - post-item
  - put-metric
  - remove-tag
  - set-protection
  - tag
  - webhook
  filters:
  - config-compliance
  - event
  - finding
  - ops-item
  - reduce
  - value

The use-case here is for me to be able to report the finding to Security Hub before Custodian deletes the stack. The only other way I can think to do this is to utilize tag and remove-tag actions with delay, along with CloudTrail action for UpdateStack while filtering on that tag. Is there a better way to achieve this?

3 replies

udomsak

@udomsak

May be is an ages question, why this policy does not work. custodian version 0.9.15

policies:
  - name: offhour-ec2
    description: |
      Shutdown EC2 Image.
    resource: ec2
    filters:
      - type: offhour
        tag: maid_offhours
        default_tz: 'Asia/Bangkok'
        offhour: 10
        weekends: true
    actions:
      - stop

  - name: onhour-ec2
    resource: ec2
    filters:
      - type: onhour
        tag: maid_offhours
        default_tz: 'Asia/Bangkok'
        onhour: 9
    actions:
      - start

2022-05-07 09:46:24,117: custodian.commands:DEBUG Loaded file ec2.offhour.yaml. Contains 2 policies
2022-05-07 09:46:24,123: custodian.aws:DEBUG using default region:ap-southeast-1 from boto
2022-05-07 09:46:25,483: custodian.output:DEBUG Storing output with <LogFile file://./offhour-ec2/custodian-run.log>
2022-05-07 09:46:25,489: custodian.policy:DEBUG Running policy:offhour-ec2 resource:ec2 region:ap-southeast-1 c7n:0.9.15
2022-05-07 09:46:25,492: custodian.cache:DEBUG Using cache file /Users/udomsak/.cache/cloud-custodian.cache
2022-05-07 09:46:25,492: custodian.resources.ec2:DEBUG Using cached c7n.resources.ec2.EC2: 62
2022-05-07 09:46:25,492: custodian.filters:WARNING offhour implicitly filtered 61 of 62 resources key:State.Name on running
2022-05-07 09:46:25,492: custodian.resources.ec2:DEBUG Filtered from 62 to 0 ec2
2022-05-07 09:46:25,493: custodian.policy:INFO policy:offhour-ec2 resource:ec2 region:ap-southeast-1 count:0 time:0.00
2022-05-07 09:46:25,493: custodian.output:DEBUG metric:ResourceCount Count:0 policy:offhour-ec2 restype:ec2 scope:policy
2022-05-07 09:46:25,493: custodian.output:DEBUG metric:ApiCalls Count:0 policy:offhour-ec2 restype:ec2
2022-05-07 09:46:25,494: custodian.output:DEBUG Storing output with <LogFile file://./onhour-ec2/custodian-run.log>
2022-05-07 09:46:25,494: custodian.policy:DEBUG Running policy:onhour-ec2 resource:ec2 region:ap-southeast-1 c7n:0.9.15
2022-05-07 09:46:25,495: custodian.cache:DEBUG Using cache file /Users/udomsak/.cache/cloud-custodian.cache
2022-05-07 09:46:25,495: custodian.resources.ec2:DEBUG Using cached c7n.resources.ec2.EC2: 62
2022-05-07 09:46:25,495: custodian.filters:WARNING onhour implicitly filtered 1 of 62 resources key:State.Name on stopped
2022-05-07 09:46:25,495: custodian.resources.ec2:DEBUG Filtered from 62 to 0 ec2
2022-05-07 09:46:25,495: custodian.policy:INFO policy:onhour-ec2 resource:ec2 region:ap-southeast-1 count:0 time:0.00
2022-05-07 09:46:25,496: custodian.output:DEBUG metric:ResourceCount Count:0 policy:onhour-ec2 restype:ec2 scope:policy
2022-05-07 09:46:25,496: custodian.output:DEBUG metric:ApiCalls Count:0 policy:onhour-ec2 restype:ec2

30 replies

aakshaik2

@aakifshaikh

Blog on how to do a quick policy health checks - https://ismsguy.medium.com/cloud-custodian-policy-health-checks-fa843e06fd7b (one way of doing it using SIEM solution).

2 replies

Oluadun

@Oluadun

Hello all, please I need some help. Has anyone written a policy to"notify on overprovisioned EC2 instances". Please I need some contributions. Thank you

6 replies

Jorge Castro

@jcastro:matrix.org

[m]

Just a reminder if you're going to KubeCon let us know so we can hang out! https://www.surveymonkey.com/r/ZQ8NXWK

Jorge Castro

@jcastro:matrix.org

[m]

In-progress notes for tomorrow's community meeting if you want to add anything to the agenda! https://hackmd.io/lxIIbW6eSoSYmWawNbqmPg?edit=

manvik4u

@manvik4u

Hey guys: Is there a way for me to remove any security group created using the default name 'launch -wizard'. I don't see any sg actions matching my need. Is action 'mark-for-op =terminate' work on EC2 or SG? I am looking at deleting the security group, not just removing permissions.
@ajkerrigan | @jtroberts83

6 replies

faan

@fdeswardt:matrix.org

[m]

Hitting UnrecognizedClientException when running following policy

- name: aws-dynamo-db-query-is-encrypted-with-aws-key
  resource: aws.dynamodb-table
  description: |
    Finds all DynamoDB tables where KMS key is AWS managed.
  filters:
    - type: kms-key
      key: KeyManager
      value: AWS

with the command

c7n-org run --cache-period 60 --cache-path /output/.c7n-cache \
  -s /output/test_policies/aws-dynamo-db-query-is-encrypted-with-aws-key \
  -c /config/accounts.yaml \
  -u /policies/aws_storage_query_impact_analysis.yaml \
  -p aws-dynamo-db-query-is-encrypted-with-aws-key \
  -r all

Found cloud-custodian/cloud-custodian#4863 indicating that should not hit this when using c7n-org

Here is the output from custodian version --debug

Custodian:   0.9.14
Python:      3.8.10 (default, Mar 15 2022, 12:22:08)
             [GCC 9.4.0]
Platform:    posix.uname_result(sysname='Linux', nodename='ip-10-229-146-15', release='5.11.0-1022-aws', version='#23~20.04.1-Ubuntu SMP Mon Nov 15 14:03:19 UTC 2021', machine='x86_64')
Using venv:  True
Docker: False
Installed:

argcomplete==1.12.3
attrs==21.2.0
boto3==1.19.12
botocore==1.22.12
docutils==0.17.1
importlib-metadata==4.8.1
jmespath==0.10.0
jsonschema==3.2.0
pyrsistent==0.18.0
python-dateutil==2.8.2
pyyaml==5.4.1
s3transfer==0.5.0
setuptools==44.0.0
six==1.16.0
tabulate==0.8.9
typing-extensions==3.10.0.2
urllib3==1.26.7
zipp==3.6.0

1 reply

faan

@fdeswardt:matrix.org

[m]

Yip, that works, but different accounts in AWS Org have different opt-in regions enabled so static list will either miss a region, or error out like with -r all

Was expecting c7n-org to query the list of active regions for each account before running the policy for all those active regions when c7n-org is invoked with the -r all parameter

1 reply

Jorge Castro

@jcastro:matrix.org

[m]

Hi everyone, I've started kicking off drafting a governance.md file, which will be our process for how people can become maintainers in c7n. As a CNCF project we're free to carve out what we want it to look like, but we do need to have one and stick to it so people can have the proper expectations when donating their time. Please take a look and open to any sorts of ideas. cc @thisisshi @darrendao : cloud-custodian/cloud-custodian#7149

We did kind of set a goal for ourselves for this calendar year that we'd move to having more lead maintainers and a more collaborative structure and less of a benevolent-dictatorship so interested in feedback!

Also this makes it clear that we should probably finish what we want an enhancement process to look like, last we checked Kapil had ideas on starting with a github issue template for proposals. (We probably don't need something highly structured like PEPs/KEPs but we probably don't want wild west either.) So if you have ideas around that or know of a project that has an interesting enhancement process for us to take a look at please let me know!

Sonny

@thisisshi

Hello everyone! 0.9.16.0 Has been released, release notes here: https://github.com/cloud-custodian/cloud-custodian/releases/tag/0.9.16.0

2 replies

DigeratiDad

@digeratidad

Hello everyone, has anyone had any issues running CC policies in AWS China?

15 replies

svujasin

@svujasin

I’m new to cloud custodian first off. Was able to stand it up in cloud run via docker image in gcp. I’ve recently discovered the policy mode functionally specifically gcp-audit where a cloud function will get spun up for you. The issue I’m running into is that when cloud build attempts to create the function in the background, it’s using gcr (Google container registry) which we have disabled in favor of artifact registry. This being said, the function won’t build. If you were to build a cloud function via gcloud, the way to force artifact registry usage is to use —docket-repository= and the path to your artifact registry location. Does cloud custodian support artifact registry cloud function creation, and if so how do you toggle to using that?

3 replies

Markus Geiger

@blurayne

hiho, is there a way to filter aws resource tags by name / regex?

2 replies

Gerald Cetrone

@gcetrone3

New to CloudCustodian
Im am using Terraform and a bash script to call cloud custodian to process my policies and get the message below.

botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetFunction operation: The security token included in the request is invalid.

Can anyone give me some pointers on how to resolve the issue?

My script is below

#! /bin/sh
echo "*** Script run_policies.sh: Running Script run_policies.sh ***"

set -x

pip3 install c7n

for policy in policies/*
do
  echo "*** Script run_policies.sh: custodian run for $policy ***"
  custodian run -s out -c $policy
done

6 replies

Ulisses Oliveira

@usoliveira

Hi everyone, I'm looking for a way to create a condition to check if an "aws.iam-user" has a "permission boundary" already setted, and If not, use "set-boundary" to automatically apply it.

I noticied that using "aws.iam-user" - check-permissions will bring me if it has boundary applied, but I can't make a filter for ONLY selecting those who do not have it already applied.

Any ideias? Thanks a lot.

4 replies

Ananth Balasubramanian

@linuxananth1976

Hello,
I have a query regarding kms-key disable action in c7n. I couldn't see disable keyword in the documentation. Actually I'm trying to disable the untagged keys.
Whether anything I'm missing or it might be differ in kms? please let me know and suggestions welcome.

1 reply

Sergio Cuellar

@herrsergio

Hello, I've been using the verbose option to execute custodian. But for my logging purposes, I find that the verbose information is not enough. For example, I would like to see in the logs, the buckets' names, that the c7n policy is working on. I only see info like this:

2022-05-16 11:33:45,056 - custodian.resources.s3 - DEBUG - Filtered from 59 to 3 s3
 2022-05-16 11:33:45,058 - custodian.policy - INFO - policy:s3-set-bucket-encryption resource:s3 region:us-east-1 count:3 time:54.41
 2022-05-16 11:33:45,062 - custodian.output - DEBUG - metric:ResourceCount Count:3 policy:s3-set-bucket-encryption restype:s3 scope:policy
 2022-05-16 11:33:45,062 - custodian.output - DEBUG - metric:ApiCalls Count:650 policy:s3-set-bucket-encryption restype:s3

Is it possible to increase verbosity with -vv, for example

6 replies

aakshaik2

@aakifshaikh

https://ismsguy.medium.com/upgrade-your-c7n-mailer-and-other-components-84e06320a4f3 - BLOG on upgrading the versions.

2 replies

vijay23vikram

@vijay23vikram

Hi Team,

I have a query about S3 bucket encryption, while applying terraform apply command, the cloud custodian function is triggered and sends a notification email to the resource owner and intimating as enable the s3 bucket encryption, but while finishing the terraform apply command execution our s3 bucket is encrypted with the sse_kms encryption as per the below sample code logic. here we are using terraform latest resource code for enabling the s3 server-side encryption by using KMS.

Here is the sample Terraform code we are using,
resource "aws_kms_key" "mykey" {
description = "This key is used to encrypt bucket objects"
deletion_window_in_days = 10
}
resource "aws_s3_bucket" "mybucket" {
bucket = "mybucket"
}
resource "aws_s3_bucket_versioning" "versioning_example" {
bucket = aws_s3_bucket.example.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "example" {
bucket = aws_s3_bucket.mybucket.bucket rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.mykey.arn
sse_algorithm = "aws:kms"
}
}

}

Here is the sample custodian filters we are using to check and trigger the notification email to resource owner.

filters:

  - "tag:non_encrypt_bucket": absent
  - type: bucket-encryption
    state: False
actions:
  - type: mark-for-op
    tag: non_encrypt_bucket
    op: set-bucket-encryption

So please suggest if the custodian filters are not supported to the latest terraform code.

Thanks in advance,

1 reply

BinduHK

@BinduHK

This message was deleted

4 replies

codehead1997

@codehead1997

Hi @/all ,
Is there any command to delete all the resource created while deploying custodian policies i.e config rule, lambda functions, triggers,etc

2 replies

aakshaik2

@aakifshaikh

I know we have a separate command to get a report for a specific policy- example: custodian report --output-dir=. --format simple policy.yml
Is there a way I can declare the custodian output format. It produces gz files. Can i declare to produce simple txt file as it runs per schedule. I am asking this for azure cloud. We have a challenge - extra hoops to convert that gz before ingesting into SIEM.
something like this custodian run --output-dir=. --format simple policy.yml

Lloyd O'Brien

@githublloyd

This message was deleted

4 replies

Lloyd O'Brien

@githublloyd

hey all - is there a way for CC to return a value if no policy actions are met? i.e. Post to Slack with a message similar to "No violations today in account xxx"

2 replies

Markus Geiger

@blurayne

Are there currently plans to include AWS storage lens?

2 replies

BinduHK

@BinduHK

This message was deleted

8 replies

aakshaik2

@aakifshaikh

I am still having issues with Azure - Upon deploying the policy - the debug messages shows all good. But don't see any FUNCTION within the Function App. It is empty. Same bug reported the last #7160. This was closed. I will create another ticket to track the resolution. So once this is fixed we must try again in our environment.

aakshaik2

@aakifshaikh

cloud-custodian/cloud-custodian#7271 - Created to track issues with AZURE

KVInventoR

@KVInventoR

Is there any way to find s3 buckets where Name is not equal to tag:Name?

    filters:
      - type: value
        key: Name
        op: not-equal
        value_type: normalize
        value: tag:Name

this filter doesn't work for me and just returned all buckets which I have in account

Sonia Gurdian

@PendragonDay

Hi there
I just created a new issue in the c7n repo. But wanted to also post here in case somebody has figured out a way to solve this: cloud-custodian/cloud-custodian#7272

But basically, our environment is a large organization (using AWS Orgs). We are constrained in the use of SES.

SES send-email API calls must include a source-arn. The source-arn is not in the same AWS account as the AWS account where c7n_mailer is deployed.

By default c7n_mailer is trying to use the default local account source-arn, which would look like this:

arn:aws:ses:us-east-1:{this_AccountId}:identity/mydomain.com

However, in our environment all accounts must use the SES source-arn that is in the Master AWS account:

arn:aws:ses:us-east-1:{Master_AccountId}:identity/mydomain.com

Is it possible to add the source-arn as a property in the config schema so it can be passed to the c7n_mailer Lambda?
https://github.com/cloud-custodian/cloud-custodian/blob/b611e5addd5c91f10897b23e7917e37ed8299c05/tools/c7n_mailer/c7n_mailer/cli.py#L34

3 replies

Leigh Hayward

@leigh507

Hi All - Just starting out looking at C7N here. This is likely a question you get a lot.... so apologies in advance.
We primarily interact with AWS through terraform and have created some organization config rules that cascade through our AWS Organization. I am struggling to see what the differences between C7N and managing AWS Config rules in code. Is the principal advantage that this tool can be used across multiple clouds? Or is it that C7N is supplementary to AWS Config and there is more i can do with c7n but I'm (very likely) missing the point

3 replies

numerotres

@numerotres:matrix.org

[m]

Hi, question: How are most folks handling IAM role deployment to multiple accounts? I'm just curious if there was a trend or framework folks were following.

Alexander Hrechenko

@Liqudity2provider

Hi, Custodian Team, my question is - could Fork of Cloud-Custodian use arn:aws:securityhub:eu-central-1::product/cloud-custodian/cloud-custodian this arn to push it to another accounts?
Or could I have a contact of a person with who I can discuss all details about?
Thank you for the reply!

faan

@fdeswardt:matrix.org

[m]

numerotres: Recommend taking a look at StackSets that make deploying a role for c7n to all accounts in AWS Org simple and automatic i.e. every time new account is created or invited into org this role will be created.
Also recommend implementing a SCP that protects this role from any manipulation in the individual accounts, especially non-prod accounts where it os common to allow engineers full admin access to build and test new projects.

Let me know if you need sample StackSet template or SCP?

I find https://asecure.cloud/l/scp/ invaluable to generate SCPs and then have the accompanying TF and CF code to automate the deployment.

1 reply

Jorge Castro

@jcastro:matrix.org

[m]

Hi everyone, we're working on the agenda for tomorrow's community meeting: https://hackmd.io/lxIIbW6eSoSYmWawNbqmPg?edit=

if you have an issue or PR that you'd like to get more eyeballs on and discuss please add a :boom: emoji next to it and we'll discuss it tomorrow. Also the agenda is open if anyone wants to add something to discuss!

KVInventoR

@KVInventoR

Hi there,
Is there any example to copy ec2 id pr s3 bucket name to tags?
Unfortunately, this code doesn't work for me:

    filters:
      - type: value
        key: "Name"
        op: eq
        value: "prod-users"

    actions:
      - type: tag
        tag: 'SecondName'
        value: Name

I need to set current instance id or bucket name as additional tag.

DuckieHo

@DuckieHo

If policylambda.py is used to build a deployment artifact, are the the built-in c7n resources still functional?

I inherited an implementation and its CI/CD pipeline. The built-in c7n resource don't appear working within the Lambdas deployed under AWS Config Rules . The description of the tool contains " different tooling instead of custodian builtin capabilities" leading me to think not.

1 reply

adrieng1977

@adrieng1977

Hi. I did some research and I cannot seem to find what is the current process to delete the infrastructure created for a specific policy in GCP. Is there a command line arg or script similar to mugc for aws?

Gerald Cetrone

@gcetrone3

I have a CC Policy that sends an email when a user's Password or Keys is about to expire. Is there a method to have the same policy include the number of days before the Password or Keys expire to be included in the email?

2 replies

Akram321

@Akram321

Can you please help me with documentation for upgrading the Nginx from 1.80.1 to 1.20.1 or higher anyone through linux machine(not custodian)

Gerald Cetrone

@gcetrone3

Is there a way to send tags into CC so the resources create by CC are tagged?

8 replies

Jorge Castro

@jcastro:matrix.org

[m]

Hi everyone, we talked about enabling github discussions in the cloud-custodian repo yesterday so that we could have a more discoverable and searchable repo of user questions and answers, I am in the process of setting that up now but would like to point out some questions we have, if any of you have time to give these a once over and see if you can help out it'd be appreciated, want to make sure we don't leave anyone behind: https://github.com/cloud-custodian/cloud-custodian/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fquestion

misumme-pg

@misumme-pg

Hi everyone. I'm using c7n-org as i'm running the following commend " c7n-org run -c custodian-multi-aws-config.yml -s output -u test-custodian-policy.yml" I keep running into following error i'm unable to figure out the issue -- "running policy:cloud-custodian-auto-tag account:test-account region:us-west-2 error:An error occurred (AccessDeniedException) when calling the CreateFunction operation: Cross-account pass role is not allowed." Can you help?

Steve

@stepkirk

Hello all, trying to come up with a policy to identify SQS queues that have had no new messages in one year. This is what I have come up with:

- name: sqs-queue-report-unused
    resource: sqs
    filters:
      - and:
        - type: metrics
          name: NumberOfMessagesSent
          days: 365
          period: 86400
          op: equal
          value: 0
          missing-value: 0
          statistics: Sum
        - type: reduce
          sort-by: QueueArn

I checked resources.json after doing a dryrun and I see several queues that show Sum Counts > 0. I thought those would be filtered out. Did I miss something in the policy definition? Any troubleshooting pointers?

37 replies

Where communities thrive

People

Repo info

Activity

}