Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources
logachev on azure_sdk_update
[Azure SDK integration branch P… (compare)
logachev on azure_sdk_update
kapilt on master
aws - iam-role add ability to f… (compare)
Schema Diff for c7n 0.9.1.0 - 0.9.2.0 Has been generated here - https://github.com/jtroberts83/Cloud-Custodian/blob/master/schema_expander/AWS-0.9.1.0-0.9.2.0-Changes.md
A new version, awesome!
{now} UTC
value but it is tagging the date that the policy was created, not when the lambda CreateFunction was triggered. I was told there is an issue open on this but didn't find it. Any plans to get that fixed as it will solve some big problems for us with enforcing on resources which don't have a CreationDate in their metadata
error:An error occurred (ValidationException) when calling the DescribeElasticsearchDomains operation: Please provide a maximum of 5 Elasticsearch domain names to describe.
set-attributes
it would be nice to have the bucket prefix be something like this: Prefix: '{resource.DomainName}'
so that all the logs aren't dumped into the same directory and instead it pulls the DomainName value from the resource metadata and puts it as the bucket logging prefix'{resource.LastModifedTime}'
as exact creation time would be nice as well, I know the {now} option currently doesn't work for lambda run policies so this would be one solution. We have to have the creation date for enforcement as we do monthly release cycles and our policies apply to all gear created AFTER that monthly release cycle date so we required to verify a resource was created after 'X' date, otherwise we are not allowed to enforce on it at this time.Hi all,
Guys, as you know a couple weeks ago EC2 key pairs started supporting tags.
Our issue that we had ton of keys for a long time and I got an idea to clean up them, as example:
every 6 hours run CC job to get over all EC2 instances and if EC2 has a key, find this key and update tags for key: LastUsedTime.
Keys, which doesn't have a tag: LastUsedTime or LastUsedTime more than 30 days, should be removed.
Is it possible to write all this steps with CC?
https://awsapichanges.info/archive/changes/20d670-ec2.html
hey guys, howdy?
I'm trying to run a policy over all my regions using the keyword all but it's not working:
% docker run -it \
-v $(pwd)/output:/home/custodian/output \
-v $(pwd)/ec2.yml:/home/custodian/ec2.yml \
cloudcustodian/c7n run -v -s /home/custodian/output ec2.yml --region=all
2020-05-25 17:22:44,902: custodian.commands:DEBUG Loaded file ec2.yml. Contains 2 policies
Traceback (most recent call last):
File "/usr/local/bin/custodian", line 11, in <module>
load_entry_point('c7n', 'console_scripts', 'custodian')()
File "/src/c7n/cli.py", line 359, in main
command(config)
File "/src/c7n/commands.py", line 107, in _load_policies
policies += provider.initialize_policies(
File "/src/c7n/resources/aws.py", line 608, in initialize_policies
get_profile_session(options).client('ec2').describe_regions(
File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258, in client
return self._session.create_client(
File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 831, in create_client
client = client_creator.create_client(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 83, in create_client
client_args = self._get_client_args(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 285, in _get_client_args
return args_creator.get_client_args(
File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 71, in get_client_args
final_args = self.compute_client_args(
File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 147, in compute_client_args
endpoint_config = self._compute_endpoint_config(
File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 218, in _compute_endpoint_config
return self._resolve_endpoint(**resolve_endpoint_kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 300, in _resolve_endpoint
return endpoint_bridge.resolve(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 360, in resolve
resolved = self.endpoint_resolver.construct_endpoint(
File "/usr/local/lib/python3.8/site-packages/botocore/regions.py", line 121, in construct_endpoint
result = self._endpoint_for_partition(
File "/usr/local/lib/python3.8/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.
running the latest img version
any hint on what am I missing here?
- name: iam-user-remediate-non-compliant
resource: aws.iam-user
filters:
- type: config-compliance
rules: [securityhub-iam-user-unused-credentials-check-nv6rb]
states: [NON_COMPLIANT]
- name: disable-rds-deletion-protection
resource: rds
mode:
role: arn:aws:iam::{account_id}:role/ABCD
type: periodic
schedule: "rate(1 day)"
execution-options:
output_dir: s3://ABCD
filters:
- type: marked-for-op
op: modify-db
actions:
- type: modify-db
update:
- property: 'DeletionProtection'
value: false
- property: 'PubliclyAccessible'
value: false
policies:
- name: no_ssh_from_internet
resource: gcp.firewall
mode:
type: gcp-audit
methods:
- v1.compute.firewalls.insert
- v1.compute.firewalls.patch
filters:
- type: value
key: allowed[?IPProtocol=='tcp'].ports[]
value: "22"
op: contains
- or:
- type: value
key: "sourceRanges | {range: join(', ', @)}.range"
value: 0.0.0.0
op: contains
- type: value
key: "sourceRanges | {range: join(', ', @)}.range"
value: /0
op: contains
actions:
- type: notify
subject: no_ssh_from_internet
to:
- test@test.com
format: json
transport:
type: pubsub
topic: projects/root-cortex-265110/topics/custodian
policies:
- name: Security Hub Non compliant resources
resource: aws.iam-user
filters:
- type: finding
query:
ComplianceStatus:
- Comparison: EQUALS
Value: FAILED
Title:
- Comparison: EQUALS
Value: "1.4 Ensure access keys are rotated every 90 days or less"
RecordState:
- Comparison: EQUALS
Value: ACTIVE
Is anyone having this error for vpc flow logs using s3 buckets? 2020-05-27 18:24:02,273: custodian.output:ERROR Error while executing policy
14:24:02 Traceback (most recent call last):
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 291, in run
14:24:02 resources = self.policy.resource_manager.resources()
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 466, in resources
14:24:02 resources = self.filter_resources(resources)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/manager.py", line 109, in filter_resources
14:24:02 resources = f.process(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 299, in process
14:24:02 return self.process_set(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 318, in process_set
14:24:02 resources = f.process(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/resources/vpc.py", line 149, in process
14:24:02 fl['LogDestination'], destination)
14:24:02 KeyError: 'LogDestination'
```
policies:
- not:
- type: flow-logs
enabled: true
set-op: or
op: equal
traffic-type: all
status: active
destination: vpcflow-govcloud
Hello, it seems I'm having trouble with this variable {account_id}
. I'm running it on docker with version 0.8.46.0. Here is my policy, mailer.yml and output. I did a dryrun where it runs custodian and the mailer. Any ideas why I'm getting this error? botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://queue.amazonaws.com/ is not valid for this endpoint.
policies:
- name: ec2-tag-compliance-notify-only
resource: ec2
description: |
Scan EC2 resources that do not meet tag compliance policies.
filters:
- type: value
key: "State.Name"
op: ni
value: ['terminated']
- or: *tag-compliance-filters
actions:
- type: notify
template: general_template.html
priority_header: 2
subject: "Enterprise Tagging - {{ policy['resource'] }} Compliance Report! [{{ account }} - {{ region }}] ***TEST EMAIL***"
violation_desc: "is not tag compliant. Please see Cheat Sheet Link below and fix your tags!"
action_desc: |
resource out of compliance. Informational only. No marking, stopping or deleting of resources.
to:
- test@example.com
transport:
type: sqs
queue: https://sqs.us-east-1.amazonaws.com/{account_id}/cloud-custodian-mailer
mailer.yml
queue_url: https://sqs.us-east-1.amazonaws.com/{account_id}/cloud-custodian-mailer
role: arn:aws:iam::{account_id}:role/cloud-custodian
from_address: test@example.com
Output:
2020-05-27 22:47:17,807: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:ap-northeast-1 count:349 time:4.05
2020-05-27 22:47:19,742: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:ap-southeast-2 count:31 time:1.84
2020-05-27 22:47:24,625: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:eu-central-1 count:64 time:4.86
2020-05-27 22:47:30,105: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:us-east-1 count:167 time:5.45
2020-05-27 22:47:34,732: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:us-east-2 count:187 time:4.57
2020-05-27 22:47:35,449 - custodian-mailer - INFO - Downloading messages from the SQS queue.
Traceback (most recent call last):
File "/usr/local/bin/c7n-mailer", line 11, in <module>
load_entry_point('c7n-mailer==0.5.7', 'console_scripts', 'c7n-mailer')()
File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/cli.py", line 253, in main
processor.run()
File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/sqs_queue_processor.py", line 117, in run
for sqs_message in sqs_messages:
File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/sqs_queue_processor.py", line 54, in __next__
response = self.aws_sqs.receive_message(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 635, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://queue.amazonaws.com/ is not valid for this endpoint.
make: *** [cust-dryrun-report-prod] Error 1
When I hardcode the account number it works.
Hi all,
Does anyone used CC for select all s3 buckets which have turned on Versioning and don't have policy to delete old versions
I tried to write some filter like:
filters:
- type: value
key: Versioning.Status
value: Enabled
- type: value
key: Lifecycle.Rules[].NoncurrentVersionExpiration.NoncurrentDays
value: absent
op: not-in
but my filter looks wrong,
how it possible to write a correct filter?
When I run this policy
policies:
- name: auto-clean-security-groups-egress
resource: security-group
description: |
Remove security group internet egress at creation
mode:
type: cloudtrail
role: arn:aws:iam::{account_id}:role/Vid-CloudCustodianClientRole
events:
- source: ec2.amazonaws.com
event: CreateSecurityGroup
ids: "responseElements.groupId"
tags:
Name: "auto-clean-security-groups-remediate-egress"
supportgroup: "CLOUD"
service: "CloudCustodian"
costcenter: "{varcostcenter}"
projectcode: "{varprojectcode}"
actions:
- type: set-permissions
remove-egress:
- IpProtocol: "-1"
Cidr: "0.0.0.0/0"
I get the following error:
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in IpPermissions[0]: "Cidr", must be one of: FromPort, IpProtocol, IpRanges, Ipv6Ranges, PrefixListIds, ToPort, UserIdGroupPairs