by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
Jamison Roberts
@jtroberts83
wallabyies
@wallabyies

Schema Diff for c7n 0.9.1.0 - 0.9.2.0 Has been generated here - https://github.com/jtroberts83/Cloud-Custodian/blob/master/schema_expander/AWS-0.9.1.0-0.9.2.0-Changes.md

A new version, awesome!

Jamison Roberts
@jtroberts83
@kapil Having an issue where I am trying to tag the Creation Time of a lambda via the CreatFunction cloudtrail event and tagging it with the {now} UTC value but it is tagging the date that the policy was created, not when the lambda CreateFunction was triggered. I was told there is an issue open on this but didn't find it. Any plans to get that fixed as it will solve some big problems for us with enforcing on resources which don't have a CreationDate in their metadata
Is there a way to extract the LastModified time from the cloudtrail event and put it's value as the CreatedDate tag value?
Jamison Roberts
@jtroberts83
@kapilt - Also found and filed an issue with elasticsearch resource as it is now erroring out on accounts that have more than 5 elasticsearch domains with this error:
error:An error occurred (ValidationException) when calling the DescribeElasticsearchDomains operation: Please provide a maximum of 5 Elasticsearch domain names to describe.
cloud-custodian/cloud-custodian#5793
Jamison Roberts
@jtroberts83
This message was deleted
4 replies
I'm betting customized face masks is going to be the new hottest swag item at conferences now!
wallabyies
@wallabyies
Looks good!
Jamison Roberts
@jtroberts83
haha
Velmurugan Velayutham
@velmuruganvelayutham
Is there a way to terminate beanstalk environment based on the load balancer traffic?. I would like to terminate the environment if there is no request to load balancer/c name for last one week.
4 replies
Jamison Roberts
@jtroberts83
Is it possible to create a feature which would allow you to pull values from a resources metadata and apply it to an action? A couple usecases:
  1. aws.distribution - when enabling logging via the set-attributes it would be nice to have the bucket prefix be something like this: Prefix: '{resource.DomainName}' so that all the logs aren't dumped into the same directory and instead it pulls the DomainName value from the resource metadata and puts it as the bucket logging prefix
  1. aws.distribution/aws.lambda and a few others - being that several resource types don't have the resource created date in their metadata, being able to tag the '{resource.LastModifedTime}' as exact creation time would be nice as well, I know the {now} option currently doesn't work for lambda run policies so this would be one solution. We have to have the creation date for enforcement as we do monthly release cycles and our policies apply to all gear created AFTER that monthly release cycle date so we required to verify a resource was created after 'X' date, otherwise we are not allowed to enforce on it at this time.
Jamison Roberts
@jtroberts83
Hope everyone in the US has a great memorial day weekend!
KVInventoR
@KVInventoR

Hi all,

Guys, as you know a couple weeks ago EC2 key pairs started supporting tags.
Our issue that we had ton of keys for a long time and I got an idea to clean up them, as example:
every 6 hours run CC job to get over all EC2 instances and if EC2 has a key, find this key and update tags for key: LastUsedTime.
Keys, which doesn't have a tag: LastUsedTime or LastUsedTime more than 30 days, should be removed.

Is it possible to write all this steps with CC?
https://awsapichanges.info/archive/changes/20d670-ec2.html

2 replies
davidkshepherd
@davidkshepherd
@jtroberts83 There is an issue related to your question about using a distribution name here: cloud-custodian/cloud-custodian#5671
brunoeustaquio
@brunoeustaquio

hey guys, howdy?
I'm trying to run a policy over all my regions using the keyword all but it's not working:

% docker run -it \
  -v $(pwd)/output:/home/custodian/output \
  -v $(pwd)/ec2.yml:/home/custodian/ec2.yml \
  cloudcustodian/c7n run -v -s /home/custodian/output  ec2.yml --region=all  
2020-05-25 17:22:44,902: custodian.commands:DEBUG Loaded file ec2.yml. Contains 2 policies
Traceback (most recent call last):
  File "/usr/local/bin/custodian", line 11, in <module>
    load_entry_point('c7n', 'console_scripts', 'custodian')()
  File "/src/c7n/cli.py", line 359, in main
    command(config)
  File "/src/c7n/commands.py", line 107, in _load_policies
    policies += provider.initialize_policies(
  File "/src/c7n/resources/aws.py", line 608, in initialize_policies
    get_profile_session(options).client('ec2').describe_regions(
  File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258, in client
    return self._session.create_client(
  File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 831, in create_client
    client = client_creator.create_client(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 83, in create_client
    client_args = self._get_client_args(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 285, in _get_client_args
    return args_creator.get_client_args(
  File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 71, in get_client_args
    final_args = self.compute_client_args(
  File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 147, in compute_client_args
    endpoint_config = self._compute_endpoint_config(
  File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 218, in _compute_endpoint_config
    return self._resolve_endpoint(**resolve_endpoint_kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 300, in _resolve_endpoint
    return endpoint_bridge.resolve(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 360, in resolve
    resolved = self.endpoint_resolver.construct_endpoint(
  File "/usr/local/lib/python3.8/site-packages/botocore/regions.py", line 121, in construct_endpoint
    result = self._endpoint_for_partition(
  File "/usr/local/lib/python3.8/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
    raise NoRegionError()
botocore.exceptions.NoRegionError: You must specify a region.

running the latest img version
any hint on what am I missing here?

7 replies
Maki2020
@Maki2020
Hi, I was wondering if it was possible to integrate cloud custodian with HashiCorp Vault somehow. We use vault for handling short-lived credentials and the cloud custodian tool implementation is on our radar for Q3. Thank you so much.
10 replies
manitmalik
@manitmalik
Ran into another issue with "config-compliance". Hence kindly share inputs.
Use Case : Get the non compliant resources via config compliance by giving the security hub config rule name rather than using our own filters.
Issue : Resource count of non compliant resources is coming as 0 rather than matching with the security hub
- name: iam-user-remediate-non-compliant
  resource: aws.iam-user
  filters:
  - type: config-compliance
    rules: [securityhub-iam-user-unused-credentials-check-nv6rb]
    states: [NON_COMPLIANT]
11 replies
manitmalik
@manitmalik
image.png
Alicia Steen
@aliciasteen
Has anyone managed to filter SAML Identity Providers? I've found this issue cloud-custodian/cloud-custodian#4871 but was wondering if anyone had a working solution?
wjamka
@wjamka
Hi All, I have simple question regarding mark-for-op, can I create my op name? For example here:
- name: disable-rds-deletion-protection
  resource: rds
  mode:
    role: arn:aws:iam::{account_id}:role/ABCD
    type: periodic
    schedule: "rate(1 day)"
    execution-options:
        output_dir: s3://ABCD
  filters:
    - type: marked-for-op
      op: modify-db
  actions:
    - type: modify-db
      update:
        - property: 'DeletionProtection'
          value: false
        - property: 'PubliclyAccessible'
          value: false
1 reply
Srinivas Krishna
@srini_krishna_twitter
hello, I want to create the custodian policy to get the rds properties and see if the 'EnabledCloudwatchLogsExports': [
'string',
], is enabled or not. anybody have already done this , any help in the policy please?
Kapil Thangavelu
@kapilt
@srini_krishna_twitter the value filter allows for rich comparisions with any resource attribute
3 replies
fakiestyle
@fakiestyle
Hello, I have a couple of questions:
Does c7n-org module support gcp.audit mode policies? I want to try some gcp.audit polices in such way that Custodian's Cloud Function could watch events in other projects from same organization. Is it possible?
Kapil Thangavelu
@kapilt
@fakiestyle you don't need c7n-org for that, you just run c7n in the centralized project, c7n-org is for parallel fan out execution
11 replies
fakiestyle
@fakiestyle
policies:
  - name: no_ssh_from_internet
    resource: gcp.firewall
    mode:
      type: gcp-audit
      methods:
        - v1.compute.firewalls.insert
        - v1.compute.firewalls.patch
    filters:
      - type: value
        key: allowed[?IPProtocol=='tcp'].ports[]
        value: "22"
        op: contains
      - or:
        - type: value
          key: "sourceRanges | {range: join(', ', @)}.range"
          value: 0.0.0.0
          op: contains
        - type: value
          key: "sourceRanges | {range: join(', ', @)}.range"
          value: /0
          op: contains
    actions:
      - type: notify
        subject: no_ssh_from_internet
        to:
          - test@test.com
        format: json
        transport:
          type: pubsub
          topic: projects/root-cortex-265110/topics/custodian
how to set multiple projects logs subscriber?
Edward Moon
@edwardmoon

Is there a way in Cloud Custodian to detect public EBS snapshots?

I created a test EBS snapshot and set it to public. When I ran a custodian script to generate EBS schema, I didn't find anything in the schema that the snapshot was public.

Kapil Thangavelu
@kapilt
@edwardmoon cross-account filter
manitmalik
@manitmalik
This message was deleted
1 reply
manitmalik
@manitmalik
thought to share in case someone is also trying to get the results of the security hub findings compliance failures
policies:
  - name: Security Hub Non compliant resources
    resource: aws.iam-user
    filters:
        - type: finding
          query:
            ComplianceStatus:
                - Comparison: EQUALS
                  Value: FAILED
            Title:
                - Comparison: EQUALS
                  Value: "1.4 Ensure access keys are rotated every 90 days or less"
            RecordState:
                - Comparison: EQUALS
                  Value: ACTIVE
Edward Moon
@edwardmoon
@kapilt Can you give more details on "cross-account filter"? the only reference to account_id i found was in the advanced usage section which i don't think applies
6 replies
Jorge Bianquetti
@jbianquetti-nami
Hello team! We're heavy users of CloudCustodian in several cloud providers and I'm in love with it: great job! We want to consolidate all our garbage collection actions using it, but we lack Kubernetes support which is marked as Not Ready yet. Do we have an ETA for that? How can I help with that since there's little information on this?
8 replies
Gautami Kher
@gautamikhervista_gitlab
Hello Team, I am using cloud custodian for sending email using sqs as transport. I dont want to use the default template for email but to utilize my own template. However, while running command "c7n-mailer --run -c mailer.yml" I am getting following error;
error: An error occurred (InvalidParameterValue) when calling the SendRawEmail operation: Missing '"'
Can you please help me with same?
4 replies
Amarankit Srivastava
@amarankit-srivastava

Hello Team, Delete with additional options is supported for IAM users but not for Roles and Grps? ``` actions:

  - type: delete
    options:
      - inline-user-policies

```

Jamison Roberts
@jtroberts83
Hi @kapilt I see that version 9.2.0 is supposed to have tag support for codecommit and codepipeline but those don't show up in the schema as an option in 0.9.2.0
4 replies
Fidel Rodriguez
@fidelito

Is anyone having this error for vpc flow logs using s3 buckets? 2020-05-27 18:24:02,273: custodian.output:ERROR Error while executing policy
14:24:02 Traceback (most recent call last):
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/policy.py", line 291, in run
14:24:02 resources = self.policy.resource_manager.resources()
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/query.py", line 466, in resources
14:24:02 resources = self.filter_resources(resources)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/manager.py", line 109, in filter_resources
14:24:02 resources = f.process(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 299, in process
14:24:02 return self.process_set(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/filters/core.py", line 318, in process_set
14:24:02 resources = f.process(resources, event)
14:24:02 File "/usr/local/lib/python3.7/site-packages/c7n/resources/vpc.py", line 149, in process
14:24:02 fl['LogDestination'], destination)
14:24:02 KeyError: 'LogDestination'

```
policies:

  • name: vpc-flowlog-enable-s3-test-GovCloud
    resource: vpc
    filters:
    - not:
      - type: flow-logs
        enabled: true
        set-op: or
        op: equal
        traffic-type: all
        status: active
        destination: vpcflow-govcloud
2 replies
custodian version is 0.9.1
Todd Stansell
@tjstansell
@kapilt is the github CI environment broken? PRs don't seem to be able to run tests...
4 replies
Michael Nguyen
@micnguyen266

Hello, it seems I'm having trouble with this variable {account_id}. I'm running it on docker with version 0.8.46.0. Here is my policy, mailer.yml and output. I did a dryrun where it runs custodian and the mailer. Any ideas why I'm getting this error? botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://queue.amazonaws.com/ is not valid for this endpoint.

policies:
  - name: ec2-tag-compliance-notify-only
    resource: ec2
    description: |
       Scan EC2 resources that do not meet tag compliance policies.
    filters:
      - type: value
        key: "State.Name"
        op: ni
        value: ['terminated']
      - or: *tag-compliance-filters
    actions:
      - type: notify
        template: general_template.html
        priority_header: 2
        subject: "Enterprise Tagging - {{ policy['resource'] }} Compliance Report! [{{ account }} - {{ region }}] ***TEST EMAIL***"
        violation_desc: "is not tag compliant. Please see Cheat Sheet Link below and fix your tags!"
        action_desc: |
          resource out of compliance. Informational only. No marking, stopping or deleting of resources.
        to:
          - test@example.com
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/{account_id}/cloud-custodian-mailer

mailer.yml

queue_url: https://sqs.us-east-1.amazonaws.com/{account_id}/cloud-custodian-mailer
role: arn:aws:iam::{account_id}:role/cloud-custodian
from_address: test@example.com

Output:

2020-05-27 22:47:17,807: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:ap-northeast-1 count:349 time:4.05
2020-05-27 22:47:19,742: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:ap-southeast-2 count:31 time:1.84
2020-05-27 22:47:24,625: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:eu-central-1 count:64 time:4.86
2020-05-27 22:47:30,105: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:us-east-1 count:167 time:5.45
2020-05-27 22:47:34,732: custodian.policy:INFO policy:ec2-tag-compliance-notify-only resource:ec2 region:us-east-2 count:187 time:4.57
2020-05-27 22:47:35,449 - custodian-mailer - INFO - Downloading messages from the SQS queue.
Traceback (most recent call last):
  File "/usr/local/bin/c7n-mailer", line 11, in <module>
    load_entry_point('c7n-mailer==0.5.7', 'console_scripts', 'c7n-mailer')()
  File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/cli.py", line 253, in main
    processor.run()
  File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/sqs_queue_processor.py", line 117, in run
    for sqs_message in sqs_messages:
  File "/usr/local/lib/python3.8/site-packages/c7n_mailer-0.5.7-py3.8.egg/c7n_mailer/sqs_queue_processor.py", line 54, in __next__
    response = self.aws_sqs.receive_message(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address https://queue.amazonaws.com/ is not valid for this endpoint.
make: *** [cust-dryrun-report-prod] Error 1

When I hardcode the account number it works.

5 replies
Amarankit Srivastava
@amarankit-srivastava
Hello team, @kapilt @jtroberts83 , any advice on how to approach (to modify codes) to achieve "Action: Detach the policies from identity: role, user, group whenever policies has <service>:?? and effect: allow, without considering resource (resource: or resource:<S3 arn>) except for Get, Describe, List* in allow block in IAM Policies​"? Many thanks
3 replies
fakiestyle
@fakiestyle
@kapilt , the last release where gcp-audit mode is working is 0.8.46.1. Any higher version - resource not loaded in traceback of cloud function with any resource.
5 replies
KVInventoR
@KVInventoR

Hi all,
Does anyone used CC for select all s3 buckets which have turned on Versioning and don't have policy to delete old versions
I tried to write some filter like:

    filters:
      - type: value
        key: Versioning.Status
        value: Enabled
      - type: value
        key: Lifecycle.Rules[].NoncurrentVersionExpiration.NoncurrentDays
        value: absent
        op: not-in

but my filter looks wrong,
how it possible to write a correct filter?

ndesai15
@ndesai15
Is there anyway I can tell to custodian report command to extract out specific fields from output of "resources.json" instead default fields for a particular resource?
2 replies
Brian Bohrer
@BrianBohrer
With regards to Cloud Custodian Security Group Policies, is it feasible to enforce that every ingress rule must specify a port or ports? Ingress rule created with 'all' ports would be remediated basically. Thanks!
2 replies
tomarv2
@tomarv2
i am putting a policy that will look for tag: DeleteAfter: MM-DD-YY I thought I can do mark-for-op but that does seem to solve the problem, any suggestion?
3 replies
Gautami Kher
@gautamikhervista_gitlab
Hi Team, Is there any policy for aws S3 bucket to encrypt specific bucket? and not entire buckets in region
Jason Terlecki
@jterlecki_gitlab

When I run this policy

policies:
  - name: auto-clean-security-groups-egress
    resource: security-group
    description: |
      Remove security group internet egress at creation
    mode:
        type: cloudtrail
        role: arn:aws:iam::{account_id}:role/Vid-CloudCustodianClientRole
        events:
          - source: ec2.amazonaws.com
            event: CreateSecurityGroup
            ids: "responseElements.groupId"
        tags:
            Name: "auto-clean-security-groups-remediate-egress"
            supportgroup: "CLOUD"
            service: "CloudCustodian"
            costcenter: "{varcostcenter}"
            projectcode: "{varprojectcode}"
    actions:
      - type: set-permissions
        remove-egress:
          - IpProtocol: "-1"
            Cidr: "0.0.0.0/0"

I get the following error:
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in IpPermissions[0]: "Cidr", must be one of: FromPort, IpProtocol, IpRanges, Ipv6Ranges, PrefixListIds, ToPort, UserIdGroupPairs

Jason Terlecki
@jterlecki_gitlab
should I be using something like this instead?
  IpRanges:
    - CidrIp: "0.0.0.0/0"