Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources
Hi all,
Guys, as you know a couple weeks ago EC2 key pairs started supporting tags.
Our issue that we had ton of keys for a long time and I got an idea to clean up them, as example:
every 6 hours run CC job to get over all EC2 instances and if EC2 has a key, find this key and update tags for key: LastUsedTime.
Keys, which doesn't have a tag: LastUsedTime or LastUsedTime more than 30 days, should be removed.
Is it possible to write all this steps with CC?
https://awsapichanges.info/archive/changes/20d670-ec2.html
hey guys, howdy?
I'm trying to run a policy over all my regions using the keyword all but it's not working:
% docker run -it \
-v $(pwd)/output:/home/custodian/output \
-v $(pwd)/ec2.yml:/home/custodian/ec2.yml \
cloudcustodian/c7n run -v -s /home/custodian/output ec2.yml --region=all
2020-05-25 17:22:44,902: custodian.commands:DEBUG Loaded file ec2.yml. Contains 2 policies
Traceback (most recent call last):
File "/usr/local/bin/custodian", line 11, in <module>
load_entry_point('c7n', 'console_scripts', 'custodian')()
File "/src/c7n/cli.py", line 359, in main
command(config)
File "/src/c7n/commands.py", line 107, in _load_policies
policies += provider.initialize_policies(
File "/src/c7n/resources/aws.py", line 608, in initialize_policies
get_profile_session(options).client('ec2').describe_regions(
File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258, in client
return self._session.create_client(
File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 831, in create_client
client = client_creator.create_client(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 83, in create_client
client_args = self._get_client_args(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 285, in _get_client_args
return args_creator.get_client_args(
File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 71, in get_client_args
final_args = self.compute_client_args(
File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 147, in compute_client_args
endpoint_config = self._compute_endpoint_config(
File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 218, in _compute_endpoint_config
return self._resolve_endpoint(**resolve_endpoint_kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 300, in _resolve_endpoint
return endpoint_bridge.resolve(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 360, in resolve
resolved = self.endpoint_resolver.construct_endpoint(
File "/usr/local/lib/python3.8/site-packages/botocore/regions.py", line 121, in construct_endpoint
result = self._endpoint_for_partition(
File "/usr/local/lib/python3.8/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.
running the latest img version
any hint on what am I missing here?
- name: iam-user-remediate-non-compliant
resource: aws.iam-user
filters:
- type: config-compliance
rules: [securityhub-iam-user-unused-credentials-check-nv6rb]
states: [NON_COMPLIANT]
- name: disable-rds-deletion-protection
resource: rds
mode:
role: arn:aws:iam::{account_id}:role/ABCD
type: periodic
schedule: "rate(1 day)"
execution-options:
output_dir: s3://ABCD
filters:
- type: marked-for-op
op: modify-db
actions:
- type: modify-db
update:
- property: 'DeletionProtection'
value: false
- property: 'PubliclyAccessible'
value: false
policies:
- name: no_ssh_from_internet
resource: gcp.firewall
mode:
type: gcp-audit
methods:
- v1.compute.firewalls.insert
- v1.compute.firewalls.patch
filters:
- type: value
key: allowed[?IPProtocol=='tcp'].ports[]
value: "22"
op: contains
- or:
- type: value
key: "sourceRanges | {range: join(', ', @)}.range"
value: 0.0.0.0
op: contains
- type: value
key: "sourceRanges | {range: join(', ', @)}.range"
value: /0
op: contains
actions:
- type: notify
subject: no_ssh_from_internet
to:
- test@test.com
format: json
transport:
type: pubsub
topic: projects/root-cortex-265110/topics/custodian
policies:
- name: Security Hub Non compliant resources
resource: aws.iam-user
filters:
- type: finding
query:
ComplianceStatus:
- Comparison: EQUALS
Value: FAILED
Title:
- Comparison: EQUALS
Value: "1.4 Ensure access keys are rotated every 90 days or less"
RecordState:
- Comparison: EQUALS
Value: ACTIVE
Is anyone having this error for vpc flow logs using s3 buckets? 2020-05-27 18:24:02,273: custodian.output:ERROR Error while executing policy
14:24:02 Traceback (most recent call last):
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 291, in run
14:24:02 resources = self.policy.resource_manager.resources()
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 466, in resources
14:24:02 resources = self.filter_resources(resources)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/manager.py", line 109, in filter_resources
14:24:02 resources = f.process(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 299, in process
14:24:02 return self.process_set(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 318, in process_set
14:24:02 resources = f.process(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/resources/vpc.py", line 149, in process
14:24:02 fl['LogDestination'], destination)
14:24:02 KeyError: 'LogDestination'
```
policies:
- not:
- type: flow-logs
enabled: true
set-op: or
op: equal
traffic-type: all
status: active
destination: vpcflow-govcloud
Hello, it seems I'm having trouble with this variable {account_id}
. I'm running it on docker with version 0.8.46.0. Here is my policy, mailer.yml and output. I did a dryrun where it runs custodian and the mailer. Any ideas why I'm getting this error? botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://queue.amazonaws.com/ is not valid for this endpoint.
policies:
- name: ec2-tag-compliance-notify-only
resource: ec2
description: |
Scan EC2 resources that do not meet tag compliance policies.
filters:
- type: value
key: "State.Name"
op: ni
value: ['terminated']
- or: *tag-compliance-filters
actions:
- type: notify
template: general_template.html
priority_header: 2
subject: "Enterprise Tagging - {{ policy['resource'] }} Compliance Report! [{{ account }} - {{ region }}] ***TEST EMAIL***"
violation_desc: "is not tag compliant. Please see Cheat Sheet Link below and fix your tags!"
action_desc: |
resource out of compliance. Informational only. No marking, stopping or deleting of resources.
to:
- test@example.com
transport:
type: sqs
queue: https://sqs.us-east-1.amazonaws.com/{account_id}/cloud-custodian-mailer
mailer.yml
queue_url: https://sqs.us-east-1.amazonaws.com/{account_id}/cloud-custodian-mailer
role: arn:aws:iam::{account_id}:role/cloud-custodian
from_address: test@example.com
Output:
2020-05-27 22:47:17,807: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:ap-northeast-1 count:349 time:4.05
2020-05-27 22:47:19,742: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:ap-southeast-2 count:31 time:1.84
2020-05-27 22:47:24,625: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:eu-central-1 count:64 time:4.86
2020-05-27 22:47:30,105: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:us-east-1 count:167 time:5.45
2020-05-27 22:47:34,732: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:us-east-2 count:187 time:4.57
2020-05-27 22:47:35,449 - custodian-mailer - INFO - Downloading messages from the SQS queue.
Traceback (most recent call last):
File "/usr/local/bin/c7n-mailer", line 11, in <module>
load_entry_point('c7n-mailer==0.5.7', 'console_scripts', 'c7n-mailer')()
File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/cli.py", line 253, in main
processor.run()
File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/sqs_queue_processor.py", line 117, in run
for sqs_message in sqs_messages:
File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/sqs_queue_processor.py", line 54, in __next__
response = self.aws_sqs.receive_message(
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 635, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://queue.amazonaws.com/ is not valid for this endpoint.
make: *** [cust-dryrun-report-prod] Error 1
When I hardcode the account number it works.
Hi all,
Does anyone used CC for select all s3 buckets which have turned on Versioning and don't have policy to delete old versions
I tried to write some filter like:
filters:
- type: value
key: Versioning.Status
value: Enabled
- type: value
key: Lifecycle.Rules[].NoncurrentVersionExpiration.NoncurrentDays
value: absent
op: not-in
but my filter looks wrong,
how it possible to write a correct filter?
When I run this policy
policies:
- name: auto-clean-security-groups-egress
resource: security-group
description: |
Remove security group internet egress at creation
mode:
type: cloudtrail
role: arn:aws:iam::{account_id}:role/Vid-CloudCustodianClientRole
events:
- source: ec2.amazonaws.com
event: CreateSecurityGroup
ids: "responseElements.groupId"
tags:
Name: "auto-clean-security-groups-remediate-egress"
supportgroup: "CLOUD"
service: "CloudCustodian"
costcenter: "{varcostcenter}"
projectcode: "{varprojectcode}"
actions:
- type: set-permissions
remove-egress:
- IpProtocol: "-1"
Cidr: "0.0.0.0/0"
I get the following error:
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in IpPermissions[0]: "Cidr", must be one of: FromPort, IpProtocol, IpRanges, Ipv6Ranges, PrefixListIds, ToPort, UserIdGroupPairs