Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources
kapilt on master
gcp - sourcerepo - fix typo in … (compare)
I've deployed a cloud custodian policy that monitors for non-compliant security groups and uses c7n-mailer to inform the necessary teams of such violations. We recently encountered a situation where a user in one of the AWS accounts was trying to create multiple SGs using a template and Cloud Custodian remediated these SGs and sent 50 to 60 emails within 10 minutes.
It was good that we received all these emails, but is there a way we can consolidate multiple emails generated by Cloud Custodian into a single email if the specified violation belongs to the same AWS account. For e.g., if I see 10 SGs created in AWS Account "222222222222", I want to see a single email like an email digest that displays all 10 violations for the AWS account 222222222222. Is this possible to configure via Cloud Custodian?
tox -e docs
is faling because it needs a awscc module, I figured fixing the docs build docs would be a good first step for me.
This message is replying to a Matrix event but we were unable to find associated bridged Gitter message to put it in the appropriate threaded conversation.
Looks like I just volunteered to do the prereq docs for those of us who don't know the ins and outs of python, heh
Hi All - I am trying to pull stats around S3 bucket sizes using a policy similar to the following:
```policies:
However, in some of the results I am seeing "Fill value for missing data" (see screen grab).
Any ideas as to why these metrics aren't being returned? Historically, I am sure I was seeing something.
Hello, anyone know how to filter out shared vpc?
"policies": [
{
"name": "vpc-flowlog-creator",
"resource": "vpc",
"mode": {
"type": "periodic",
"schedule": "rate(1 hour)”,
},
"filters": [
{
"type": "flow-logs",
"enabled": false
}
],
The error is:
[ERROR] ClientError: An error occurred (UnauthorizedOperation) when calling the CreateFlowLogs operation: You are not authorized to perform this operation. A subnet in this vpc is shared but the provided object is not owned by you
This makes sense, but how do I filter out a VPC not owned by my account?
Hi! I am curious if aws.cfn resource supports mark-for-op action?
I notice when i run custodian schema aws.cfn
it is not listed in the actions list:
aws.cfn:
actions:
- auto-tag-user
- copy-related-tag
- delete
- invoke-lambda
- invoke-sfn
- notify
- post-finding
- post-item
- put-metric
- remove-tag
- set-protection
- tag
- webhook
filters:
- config-compliance
- event
- finding
- ops-item
- reduce
- value
The use-case here is for me to be able to report the finding to Security Hub before Custodian deletes the stack. The only other way I can think to do this is to utilize tag and remove-tag actions with delay, along with CloudTrail action for UpdateStack while filtering on that tag. Is there a better way to achieve this?
policies:
- name: offhour-ec2
description: |
Shutdown EC2 Image.
resource: ec2
filters:
- type: offhour
tag: maid_offhours
default_tz: 'Asia/Bangkok'
offhour: 10
weekends: true
actions:
- stop
- name: onhour-ec2
resource: ec2
filters:
- type: onhour
tag: maid_offhours
default_tz: 'Asia/Bangkok'
onhour: 9
actions:
- start
2022-05-07 09:46:24,117: custodian.commands:DEBUG Loaded file ec2.offhour.yaml. Contains 2 policies
2022-05-07 09:46:24,123: custodian.aws:DEBUG using default region:ap-southeast-1 from boto
2022-05-07 09:46:25,483: custodian.output:DEBUG Storing output with <LogFile file://./offhour-ec2/custodian-run.log>
2022-05-07 09:46:25,489: custodian.policy:DEBUG Running policy:offhour-ec2 resource:ec2 region:ap-southeast-1 c7n:0.9.15
2022-05-07 09:46:25,492: custodian.cache:DEBUG Using cache file /Users/udomsak/.cache/cloud-custodian.cache
2022-05-07 09:46:25,492: custodian.resources.ec2:DEBUG Using cached c7n.resources.ec2.EC2: 62
2022-05-07 09:46:25,492: custodian.filters:WARNING offhour implicitly filtered 61 of 62 resources key:State.Name on running
2022-05-07 09:46:25,492: custodian.resources.ec2:DEBUG Filtered from 62 to 0 ec2
2022-05-07 09:46:25,493: custodian.policy:INFO policy:offhour-ec2 resource:ec2 region:ap-southeast-1 count:0 time:0.00
2022-05-07 09:46:25,493: custodian.output:DEBUG metric:ResourceCount Count:0 policy:offhour-ec2 restype:ec2 scope:policy
2022-05-07 09:46:25,493: custodian.output:DEBUG metric:ApiCalls Count:0 policy:offhour-ec2 restype:ec2
2022-05-07 09:46:25,494: custodian.output:DEBUG Storing output with <LogFile file://./onhour-ec2/custodian-run.log>
2022-05-07 09:46:25,494: custodian.policy:DEBUG Running policy:onhour-ec2 resource:ec2 region:ap-southeast-1 c7n:0.9.15
2022-05-07 09:46:25,495: custodian.cache:DEBUG Using cache file /Users/udomsak/.cache/cloud-custodian.cache
2022-05-07 09:46:25,495: custodian.resources.ec2:DEBUG Using cached c7n.resources.ec2.EC2: 62
2022-05-07 09:46:25,495: custodian.filters:WARNING onhour implicitly filtered 1 of 62 resources key:State.Name on stopped
2022-05-07 09:46:25,495: custodian.resources.ec2:DEBUG Filtered from 62 to 0 ec2
2022-05-07 09:46:25,495: custodian.policy:INFO policy:onhour-ec2 resource:ec2 region:ap-southeast-1 count:0 time:0.00
2022-05-07 09:46:25,496: custodian.output:DEBUG metric:ResourceCount Count:0 policy:onhour-ec2 restype:ec2 scope:policy
2022-05-07 09:46:25,496: custodian.output:DEBUG metric:ApiCalls Count:0 policy:onhour-ec2 restype:ec2
Hitting UnrecognizedClientException
when running following policy
- name: aws-dynamo-db-query-is-encrypted-with-aws-key
resource: aws.dynamodb-table
description: |
Finds all DynamoDB tables where KMS key is AWS managed.
filters:
- type: kms-key
key: KeyManager
value: AWS
with the command
c7n-org run --cache-period 60 --cache-path /output/.c7n-cache \
-s /output/test_policies/aws-dynamo-db-query-is-encrypted-with-aws-key \
-c /config/accounts.yaml \
-u /policies/aws_storage_query_impact_analysis.yaml \
-p aws-dynamo-db-query-is-encrypted-with-aws-key \
-r all
Found cloud-custodian/cloud-custodian#4863 indicating that should not hit this when using c7n-org
Here is the output from custodian version --debug
Custodian: 0.9.14
Python: 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0]
Platform: posix.uname_result(sysname='Linux', nodename='ip-10-229-146-15', release='5.11.0-1022-aws', version='#23~20.04.1-Ubuntu SMP Mon Nov 15 14:03:19 UTC 2021', machine='x86_64')
Using venv: True
Docker: False
Installed:
argcomplete==1.12.3
attrs==21.2.0
boto3==1.19.12
botocore==1.22.12
docutils==0.17.1
importlib-metadata==4.8.1
jmespath==0.10.0
jsonschema==3.2.0
pyrsistent==0.18.0
python-dateutil==2.8.2
pyyaml==5.4.1
s3transfer==0.5.0
setuptools==44.0.0
six==1.16.0
tabulate==0.8.9
typing-extensions==3.10.0.2
urllib3==1.26.7
zipp==3.6.0
-r all
c7n-org
to query the list of active regions for each account before running the policy for all those active regions when c7n-org
is invoked with the -r all
parameter
New to CloudCustodian
Im am using Terraform and a bash script to call cloud custodian to process my policies and get the message below.
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the GetFunction operation: The security token included in the request is invalid.
Can anyone give me some pointers on how to resolve the issue?
My script is below
#! /bin/sh
echo "*** Script run_policies.sh: Running Script run_policies.sh ***"
set -x
pip3 install c7n
for policy in policies/*
do
echo "*** Script run_policies.sh: custodian run for $policy ***"
custodian run -s out -c $policy
done
Hi everyone, I'm looking for a way to create a condition to check if an "aws.iam-user" has a "permission boundary" already setted, and If not, use "set-boundary" to automatically apply it.
I noticied that using "aws.iam-user" - check-permissions will bring me if it has boundary applied, but I can't make a filter for ONLY selecting those who do not have it already applied.
Any ideias? Thanks a lot.
Hello, I've been using the verbose option to execute custodian. But for my logging purposes, I find that the verbose information is not enough. For example, I would like to see in the logs, the buckets' names, that the c7n policy is working on. I only see info like this:
2022-05-16 11:33:45,056 - custodian.resources.s3 - DEBUG - Filtered from 59 to 3 s3
2022-05-16 11:33:45,058 - custodian.policy - INFO - policy:s3-set-bucket-encryption resource:s3 region:us-east-1 count:3 time:54.41
2022-05-16 11:33:45,062 - custodian.output - DEBUG - metric:ResourceCount Count:3 policy:s3-set-bucket-encryption restype:s3 scope:policy
2022-05-16 11:33:45,062 - custodian.output - DEBUG - metric:ApiCalls Count:650 policy:s3-set-bucket-encryption restype:s3
Is it possible to increase verbosity with -vv, for example
Hi Team,
I have a query about S3 bucket encryption, while applying terraform apply command, the cloud custodian function is triggered and sends a notification email to the resource owner and intimating as enable the s3 bucket encryption, but while finishing the terraform apply command execution our s3 bucket is encrypted with the sse_kms encryption as per the below sample code logic. here we are using terraform latest resource code for enabling the s3 server-side encryption by using KMS.
Here is the sample Terraform code we are using,
resource "aws_kms_key" "mykey" {
description = "This key is used to encrypt bucket objects"
deletion_window_in_days = 10
}
resource "aws_s3_bucket" "mybucket" {
bucket = "mybucket"
}
resource "aws_s3_bucket_versioning" "versioning_example" {
bucket = aws_s3_bucket.example.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "example" {
bucket = aws_s3_bucket.mybucket.bucket rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.mykey.arn
sse_algorithm = "aws:kms"
}
}
Here is the sample custodian filters we are using to check and trigger the notification email to resource owner.
filters:
- "tag:non_encrypt_bucket": absent
- type: bucket-encryption
state: False
actions:
- type: mark-for-op
tag: non_encrypt_bucket
op: set-bucket-encryption
So please suggest if the custodian filters are not supported to the latest terraform code.
Thanks in advance,
custodian report --output-dir=. --format simple policy.yml
custodian run --output-dir=. --format simple policy.yml
Hi there
I just created a new issue in the c7n repo. But wanted to also post here in case somebody has figured out a way to solve this: cloud-custodian/cloud-custodian#7272
But basically, our environment is a large organization (using AWS Orgs). We are constrained in the use of SES.
SES send-email API calls must include a source-arn. The source-arn is not in the same AWS account as the AWS account where c7n_mailer is deployed.
By default c7n_mailer is trying to use the default local account source-arn, which would look like this:
arn:aws:ses:us-east-1:{this_AccountId}:identity/mydomain.com
However, in our environment all accounts must use the SES source-arn that is in the Master AWS account:
arn:aws:ses:us-east-1:{Master_AccountId}:identity/mydomain.com
Is it possible to add the source-arn as a property in the config schema so it can be passed to the c7n_mailer Lambda?
https://github.com/cloud-custodian/cloud-custodian/blob/b611e5addd5c91f10897b23e7917e37ed8299c05/tools/c7n_mailer/c7n_mailer/cli.py#L34
numerotres: Recommend taking a look at StackSets that make deploying a role for c7n to all accounts in AWS Org simple and automatic i.e. every time new account is created or invited into org this role will be created.
Also recommend implementing a SCP that protects this role from any manipulation in the individual accounts, especially non-prod accounts where it os common to allow engineers full admin access to build and test new projects.
Let me know if you need sample StackSet template or SCP?
I find https://asecure.cloud/l/scp/ invaluable to generate SCPs and then have the accompanying TF and CF code to automate the deployment.
:boom:
emoji next to it and we'll discuss it tomorrow. Also the agenda is open if anyone wants to add something to discuss!
Hi there,
Is there any example to copy ec2 id pr s3 bucket name to tags?
Unfortunately, this code doesn't work for me:
filters:
- type: value
key: "Name"
op: eq
value: "prod-users"
actions:
- type: tag
tag: 'SecondName'
value: Name
I need to set current instance id or bucket name as additional tag.