Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

Jason Roth

@jsnrth

Hi, I'm working on setting up cloud custodian against our test account in AWS. I'd like to have it clean up EC2 instances after some period of time, but I'd like that timing to be somewhat granular.

Ideally, I would like to tag EC2 instances with tag:cloud-custodian and values like 30m, 1h, 6h, 3d, etc., and use that as a policy to terminate the EC2 instances after 30 minutes, 1 hour, 6 hours, 3 days, etc.

Is that a supported use-case for this tool?

Jamison Roberts

@jtroberts83

@jsnrth Yes you can do that with the mark-for-op action and marked-for-op filters

Jason Roth

@jsnrth

@jro

@jtroberts83 great, thanks! i'll take a look at those docs

Jon Gilmore

@JonGilmore_gitlab

think I'm not quite understanding the conditions correctly. i'm trying to get a policy to only execute with accounts defined with c7n-org with tag type:prod

policies:
  - name: elb-no-instances
    resource: elb
    conditions:
      - type: value
        key: type
        value: "prod"
        op: eq
    filters:
      - Instances: []

and here's a snip from my org yml

  - account_id: "123123123"
    name: prod
    tags:
      - type:prod
    regions:
      - us-east-1
      - us-east-2
      - eu-west-1
      - ap-northeast-1
      - ap-northeast-2
    role: arn:aws:iam::123123123:role/c7n-role

27 replies

when I then run c7n-org, it comes back with zeros instantly. if I remove the condition, elbs return, so I know that its not just returning zero

Edward Moon

@edwardmoon

is there a way to detect vpc network acls which allow wide open access (eg like the default vpc rules shown here: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html)?

looked at the vpc network acl docs but didn't see anything useful there.

1 reply

Charles Roberts

@charles545587

@kapilt

I am working on a project where we are using AWS Config Managed rules for compliance visibility and because they cover a lot of the use cases we want.

Because there is no native integration between config and Security Hub and so we can build in some automated responses we are using c7n policies.

At the moment, there is no functionality within c7n to enable the managed config rules. This is easy to achieve in cfn or terraform. Would the ability to enable a named Config managed rule from within a c7n policy be a valid use case.

It would allow users to define everything within their policy and if there was no managed config rule they can define a custom config rule using c7n.

What would be your view?

3 replies

Jamison Roberts

@jtroberts83

@all - I got the my script posted which parses your current custodian schema to extract all resource types, then runs generic scan policies against all those resource types it discovers and then generates csv and json reports for all resources as well as an expanded schema and uploads all the report files to your S3 bucket. These JSON reports are EXTREMELY useful for writing policies as you can see the keys you can filter on and what data is available.

https://github.com/jtroberts83/Cloud-Custodian/blob/master/schema_expander/ScanAndReportALLResourceTypes.sh

To run this you would copy the script to your custodian environment, run a chmod +x sciptname.sh to make it executable, edit the S3 bucket variable in the script and then run it. Assumes you are using c7n-org. If you are not using c7n-org you can modify the c7n-org commands to use custodian run with your parameters. Let me know if you have any questions and enjoy! Cheers! :beers:

3 replies

manitmalik

@manitmalik

Hi, Running into an issue regarding IAM users with 2 access Keys
Filter :

- and:
    - type: credential
      key: access_keys.last_used_date
      value_type: age
      value: 80
      op: greater-than
    - type: credential
      key: access_keys.active
      value: true

The resource.json is bringing result as

"access_keys": [
        {
          "active": true,
          "last_rotated": "2019-03-07T15:28:09+00:00",
          "last_used_date": "2020-04-23T02:53:00+00:00",
          "last_used_region": "us-west-2",
          "last_used_service": "s3",
          "c7n:match-type": "credential"
        },
        {
          "active": false,
          "last_rotated": "2019-09-10T09:41:02+00:00",
          "last_used_date": "2019-10-03T15:00:00+00:00",
          "last_used_region": "us-west-2",
          "last_used_service": "s3",
          "c7n:match-type": "credential"
        }

Query is that why i am getting this resource as none of the access key satisfy both conditions. Is it working as "or" condition for both of the access keys
@jtroberts83 @kapilt Please share your thoughts

38 replies

Clayton

@hammer2j2

Are there any plans to support an un-deploy feature in CC? Sometimes we have cases where we may have more than a single lambda to enforce say MFA on a console user, and one lambda tags then after 7 days of the tag age another lambda removes access. We want to downgrade that policy to only keep the notification/tagging lambda but not have the removal lambda in some cases where it's been decided not to do that part.

14 replies

crickyyy1

@crickyyy1

Hello, I'm having a bit of trouble figuring out how to change the number of async retries when creating/running a lambda with periodic mode in AWS, does anyone have any experience changing it from the default of 2 to something else?

15 replies

Todd Stansell

@tjstansell

just posted a starting PR for refactoring how actions are handled ... essentially moving from a policy-level set of actions to resource-level ... where we actually track individual resource success/fail and proceed with only ones that succeed. this also allows us to have some actions like notify that can look at all resources, even previously failed ones, so we can report on failures. The PR is going to be pretty large, as I'm refactoring every action in every resource ... using common code for many things, etc. if you're interested, please review ... comments welcome as i continue to walk through all the resources and update their actions. :) cloud-custodian/cloud-custodian#5790

2 replies

Kapil Thangavelu

@kapilt

custodian 0.9.2 released - release notes ~ https://github.com/cloud-custodian/cloud-custodian/releases/tag/0.9.2.0

1 reply

Dan Parsons

@danparsons

Is there a way to get c7n (via c7n-org) to NOT attempt to write to output/ ?

20 replies

Dan Parsons

@danparsons

LOL this is what happens when you test c7n too many times in one day, from aws sqs

error: An error occurred (Throttling) when calling the SendRawEmail operation (reached max retries: 4): Daily message quota exceeded.

4 replies

Jamison Roberts

@jtroberts83

Schema Diff for c7n 0.9.1.0 - 0.9.2.0 Has been generated here - https://github.com/jtroberts83/Cloud-Custodian/blob/master/schema_expander/AWS-0.9.1.0-0.9.2.0-Changes.md

wallabyies

@wallabyies

Schema Diff for c7n 0.9.1.0 - 0.9.2.0 Has been generated here - https://github.com/jtroberts83/Cloud-Custodian/blob/master/schema_expander/AWS-0.9.1.0-0.9.2.0-Changes.md

A new version, awesome!

Jamison Roberts

@jtroberts83

@kapil Having an issue where I am trying to tag the Creation Time of a lambda via the CreatFunction cloudtrail event and tagging it with the {now} UTC value but it is tagging the date that the policy was created, not when the lambda CreateFunction was triggered. I was told there is an issue open on this but didn't find it. Any plans to get that fixed as it will solve some big problems for us with enforcing on resources which don't have a CreationDate in their metadata

2 replies

Is there a way to extract the LastModified time from the cloudtrail event and put it's value as the CreatedDate tag value?

Jamison Roberts

@jtroberts83

@kapilt - Also found and filed an issue with elasticsearch resource as it is now erroring out on accounts that have more than 5 elasticsearch domains with this error:

error:An error occurred (ValidationException) when calling the DescribeElasticsearchDomains operation: Please provide a maximum of 5 Elasticsearch domain names to describe.

cloud-custodian/cloud-custodian#5793

Jamison Roberts

@jtroberts83

This message was deleted

4 replies

I'm betting customized face masks is going to be the new hottest swag item at conferences now!

wallabyies

@wallabyies

Looks good!

Jamison Roberts

@jtroberts83

haha

Velmurugan Velayutham

@velmuruganvelayutham

Is there a way to terminate beanstalk environment based on the load balancer traffic?. I would like to terminate the environment if there is no request to load balancer/c name for last one week.

4 replies

Jamison Roberts

@jtroberts83

Is it possible to create a feature which would allow you to pull values from a resources metadata and apply it to an action? A couple usecases:

aws.distribution - when enabling logging via the set-attributes it would be nice to have the bucket prefix be something like this: Prefix: '{resource.DomainName}' so that all the logs aren't dumped into the same directory and instead it pulls the DomainName value from the resource metadata and puts it as the bucket logging prefix

aws.distribution/aws.lambda and a few others - being that several resource types don't have the resource created date in their metadata, being able to tag the '{resource.LastModifedTime}' as exact creation time would be nice as well, I know the {now} option currently doesn't work for lambda run policies so this would be one solution. We have to have the creation date for enforcement as we do monthly release cycles and our policies apply to all gear created AFTER that monthly release cycle date so we required to verify a resource was created after 'X' date, otherwise we are not allowed to enforce on it at this time.

Jamison Roberts

@jtroberts83

Hope everyone in the US has a great memorial day weekend!

KVInventoR

@KVInventoR

Hi all,

Guys, as you know a couple weeks ago EC2 key pairs started supporting tags.
Our issue that we had ton of keys for a long time and I got an idea to clean up them, as example:
every 6 hours run CC job to get over all EC2 instances and if EC2 has a key, find this key and update tags for key: LastUsedTime.
Keys, which doesn't have a tag: LastUsedTime or LastUsedTime more than 30 days, should be removed.

Is it possible to write all this steps with CC?
https://awsapichanges.info/archive/changes/20d670-ec2.html

2 replies

davidkshepherd

@davidkshepherd

@jtroberts83 There is an issue related to your question about using a distribution name here: cloud-custodian/cloud-custodian#5671

brunoeustaquio

@brunoeustaquio

hey guys, howdy?
I'm trying to run a policy over all my regions using the keyword all but it's not working:

% docker run -it \
  -v $(pwd)/output:/home/custodian/output \
  -v $(pwd)/ec2.yml:/home/custodian/ec2.yml \
  cloudcustodian/c7n run -v -s /home/custodian/output  ec2.yml --region=all  
2020-05-25 17:22:44,902: custodian.commands:DEBUG Loaded file ec2.yml. Contains 2 policies
Traceback (most recent call last):
  File "/usr/local/bin/custodian", line 11, in <module>
    load_entry_point('c7n', 'console_scripts', 'custodian')()
  File "/src/c7n/cli.py", line 359, in main
    command(config)
  File "/src/c7n/commands.py", line 107, in _load_policies
    policies += provider.initialize_policies(
  File "/src/c7n/resources/aws.py", line 608, in initialize_policies
    get_profile_session(options).client('ec2').describe_regions(
  File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258, in client
    return self._session.create_client(
  File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 831, in create_client
    client = client_creator.create_client(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 83, in create_client
    client_args = self._get_client_args(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 285, in _get_client_args
    return args_creator.get_client_args(
  File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 71, in get_client_args
    final_args = self.compute_client_args(
  File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 147, in compute_client_args
    endpoint_config = self._compute_endpoint_config(
  File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 218, in _compute_endpoint_config
    return self._resolve_endpoint(**resolve_endpoint_kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 300, in _resolve_endpoint
    return endpoint_bridge.resolve(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 360, in resolve
    resolved = self.endpoint_resolver.construct_endpoint(
  File "/usr/local/lib/python3.8/site-packages/botocore/regions.py", line 121, in construct_endpoint
    result = self._endpoint_for_partition(
  File "/usr/local/lib/python3.8/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

running the latest img version
any hint on what am I missing here?

7 replies

Maki2020

@Maki2020

Hi, I was wondering if it was possible to integrate cloud custodian with HashiCorp Vault somehow. We use vault for handling short-lived credentials and the cloud custodian tool implementation is on our radar for Q3. Thank you so much.

10 replies

manitmalik

@manitmalik

Ran into another issue with "config-compliance". Hence kindly share inputs.
Use Case : Get the non compliant resources via config compliance by giving the security hub config rule name rather than using our own filters.
Issue : Resource count of non compliant resources is coming as 0 rather than matching with the security hub

- name: iam-user-remediate-non-compliant
  resource: aws.iam-user
  filters:
  - type: config-compliance
    rules: [securityhub-iam-user-unused-credentials-check-nv6rb]
    states: [NON_COMPLIANT]

11 replies

manitmalik

@manitmalik

Alicia Steen

@aliciasteen

Has anyone managed to filter SAML Identity Providers? I've found this issue cloud-custodian/cloud-custodian#4871 but was wondering if anyone had a working solution?

wjamka

@wjamka

Hi All, I have simple question regarding mark-for-op, can I create my op name? For example here:

- name: disable-rds-deletion-protection
  resource: rds
  mode:
    role: arn:aws:iam::{account_id}:role/ABCD
    type: periodic
    schedule: "rate(1 day)"
    execution-options:
        output_dir: s3://ABCD
  filters:
    - type: marked-for-op
      op: modify-db
  actions:
    - type: modify-db
      update:
        - property: 'DeletionProtection'
          value: false
        - property: 'PubliclyAccessible'
          value: false

1 reply

Srinivas Krishna

@srini_krishna_twitter

hello, I want to create the custodian policy to get the rds properties and see if the 'EnabledCloudwatchLogsExports': [
'string',
], is enabled or not. anybody have already done this , any help in the policy please?

Kapil Thangavelu

@kapilt

@srini_krishna_twitter the value filter allows for rich comparisions with any resource attribute

3 replies

fakiestyle

@fakiestyle

Hello, I have a couple of questions:
Does c7n-org module support gcp.audit mode policies? I want to try some gcp.audit polices in such way that Custodian's Cloud Function could watch events in other projects from same organization. Is it possible?

Kapil Thangavelu

@kapilt

@fakiestyle you don't need c7n-org for that, you just run c7n in the centralized project, c7n-org is for parallel fan out execution

11 replies

fakiestyle

@fakiestyle

policies:
  - name: no_ssh_from_internet
    resource: gcp.firewall
    mode:
      type: gcp-audit
      methods:
        - v1.compute.firewalls.insert
        - v1.compute.firewalls.patch
    filters:
      - type: value
        key: allowed[?IPProtocol=='tcp'].ports[]
        value: "22"
        op: contains
      - or:
        - type: value
          key: "sourceRanges | {range: join(', ', @)}.range"
          value: 0.0.0.0
          op: contains
        - type: value
          key: "sourceRanges | {range: join(', ', @)}.range"
          value: /0
          op: contains
    actions:
      - type: notify
        subject: no_ssh_from_internet
        to:
          - test@test.com
        format: json
        transport:
          type: pubsub
          topic: projects/root-cortex-265110/topics/custodian

how to set multiple projects logs subscriber?

Edward Moon

@edwardmoon

Is there a way in Cloud Custodian to detect public EBS snapshots?

I created a test EBS snapshot and set it to public. When I ran a custodian script to generate EBS schema, I didn't find anything in the schema that the snapshot was public.

Kapil Thangavelu

@kapilt

@edwardmoon cross-account filter

manitmalik

@manitmalik

This message was deleted

1 reply

manitmalik

@manitmalik

thought to share in case someone is also trying to get the results of the security hub findings compliance failures

policies:
  - name: Security Hub Non compliant resources
    resource: aws.iam-user
    filters:
        - type: finding
          query:
            ComplianceStatus:
                - Comparison: EQUALS
                  Value: FAILED
            Title:
                - Comparison: EQUALS
                  Value: "1.4 Ensure access keys are rotated every 90 days or less"
            RecordState:
                - Comparison: EQUALS
                  Value: ACTIVE

Edward Moon

@edwardmoon

@kapilt Can you give more details on "cross-account filter"? the only reference to account_id i found was in the advanced usage section which i don't think applies

6 replies

Jorge Bianquetti

@jbianquetti-nami

Hello team! We're heavy users of CloudCustodian in several cloud providers and I'm in love with it: great job! We want to consolidate all our garbage collection actions using it, but we lack Kubernetes support which is marked as Not Ready yet. Do we have an ETA for that? How can I help with that since there's little information on this?

8 replies

Gautami Kher

@gautamikhervista_gitlab

Hello Team, I am using cloud custodian for sending email using sqs as transport. I dont want to use the default template for email but to utilize my own template. However, while running command "c7n-mailer --run -c mailer.yml" I am getting following error;
error: An error occurred (InvalidParameterValue) when calling the SendRawEmail operation: Missing '"'
Can you please help me with same?

4 replies

Where communities thrive

People

Repo info

Activity