Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

tsushan822

@tsushan822

Hi, I am using cloudcustodian to filter my EC2 AMI's. I have already a custodian policy as follows:

- name: ec2-ami-deregister-by-tag-filter
  resource: ami
  comment: |
    Check EC2 AMI's which are not having the following tags and deregister it.
  filters:
      - "tag:Owner": absent
      - "tag:Environment": absent
      - "tag:Purpose": absent
      - "tag:Retention": absent
  actions:
    - type: deregister

The problem with the above policy is that all tag name need to have a tagName with Capital Letters(i.e Owner, Environment, Purpose, Retention). I am planning on having some flexibility so allowing users to tag with Capitalizing the tagName(i.e owner, environment, purpose, retention). For that, I have added some AND and OR conditions but they are not working:

- name: ec2-ami-deregister-by-tag-filter
  resource: ami
  comment: |
    Check EC2 AMI's which are not having the following tags and deregister it.
  filters:
    - and :
      - "tag:Owner": absent
      - "tag:Environment": absent
      - "tag:Purpose": absent
      - "tag:Retention": absent
    - or :
      - "tag:owner": absent
      - "tag:environment": absent
      - "tag:purpose": absent
      - "tag:retention": absent
  actions:
    - type: deregister

What changes might be required on the following policy to achieve my goal.

1 reply

satvan23

@satvan23

Guys,

I have this policy to exclude security groups with Name Tag not containing "glue" but I still see that string in the email I get.

`` policies:

name: report-0-65535-ports-security-groups
resource: aws.security-group
filters:
- type: ingress
  IpProtocol: tcp
  FromPort: 0
  ToPort: 65535
- type: value
  key: "tag:Name"
  op: regex
  value: '^((?!glue).)*$'
  ``

1 reply

veenagurram

@veenagurram

Is there any filter to find out ELB with unhealthy hosts?

3 replies

KISStian

@KISStian

Is it currently possible to use tags present in the account config file for c7n-org as conditions and/or filters within policies?

KISStian

@KISStian

I did a little more searching and I saw in the docs that the account key can be used to get information from the config. Can’t believe I missed this.

KISStian

@KISStian

I have experience using c7n, but typically when it comes to just managing a few accounts. My team is currently decommissioning our existing governance solution and going to implement c7n across the organization which currently has 200+ AWS accounts.

We were planning on using c7n-org to distribute CloudTrail-based lambdas as well as a few daily periodic ones (kind of like a custodian sweep of things that may have been missed e.g. AWS issue with EventBridge). After reading through a number of conversations here, it is obvious that many others are doing something similar. However, it seems like there is a trade-off regarding a centralized a run (event-based lambda in master along with c7n-org polling) and a distributed setup across accounts, and I am having a hard time determining whether or not we should continue with the architecture path we have chosen.

Any help regarding the dilemma I am in, would be greatly appreciated.

Samarth Shivaramu

@s_samarth03_twitter

Hello, I am trying to create a CC policy to retrieve a report of all IAM users in all the AWS accounts who have access keys creation date greater than 90 days. Here's the policy written based on the documentation:

policies:
  - name: iam-user-access-keys-older-than-90-days
    description: Retrieve all IAM users whom have access keys older than 90 days
    resource: iam-user
    filters:
      - type: access-key
        key: Status
        value: Active
      - type: access-key
        match-operator: and
        key: CreateDate
        op: greater-than
        value: 90
        value_type: age

The command used to create the report is

c7n-org report -c ~/accounts.yml -s output --region all -u iam-user-audit.yml

No report is generated when the command is executed. I have checked the AWS accounts and there are multiple IAM user accounts that have IAM users having access keys created more than 90 days back. No errors are generated, but the report is blank as shown below:

Account,Region,Policy,UserName,CreateDate

Is the CC policy correct to retrieve the list of IAM user accounts?

KISStian

@KISStian

@s_samarth03_twitter Did you first execute c7n-org run as a dryrun first? I believe c7n-org report uses the results from the last run/dryrun to generate the report.

1 reply

farisbacker

@farisbacker

Hi, I am trying to catch s3 bucket with cross enviroment access. ie, If any Test or Prod environment S3 buckets are accessible in lower environment accounts. We have different OU for each enviroment accounts. Tried to catch this using cross acount filter with whitelist OU.

policies:
  - name: core-s3-bucket-cross-account
    resource: s3
    filters:
      - type: cross-account
        whitelist_orgids:
          - ou-xxx-xxxxx

but seems like, it catches only if we have explicitly mentioned OU in bucket access policy, it does not catch if bucket have access to specific account in OU. Is there anyway I can catch cross envirnment buckets ?

3 replies

Chaitanya Tyagi

@chay2199

Is there any way to get elbv2 listeners data using custodian?

1 reply

Steve Craig

@stevesworkgithub

Hi, I'm getting inconsistent results when trying to disable user-access keys when the user has two keys, one of which matches the age filter. I know this was an older issue, now fixed, but I think my particular problem is that I shouldn't be filtering on the user as well, only on the keys.


  - name: iam-active-keys-no-login
    region: eu-west-2
    resource: iam-user
    mode:
      type: periodic
      schedule: "cron(0 09-17 ? * MON-FRI *)"
      role: CustodianLambda
    filters:
      - type: value
        key: UserName
        op: in
        value:
            - testuser
      - type: credential
        key: access_keys.last_used_date
        value_type: age
        value: 760
        op: greater-than
    actions:
      - type: remove-keys
        disable: true
        age: 760

This ran perfectly during all tests, but when I run against production systems I'm getting some keys disabled that are only a few weeks old.

The other issue may be that we're using really high age values in order to minimise risk - but that seems unlikely. Thanks for any pointers in advance.

15 replies

SankarOps

@SankarOps

Hello all, how to set the workflow status using Cloud Custodian policy on AWS SecurityHub. I do not see any parameter/arguments in documentation? I want to set the findings as RESOLVED or FAILED based on the policy I write.

19 replies

pendyalal

@pendyalal

Hi all, we have a use case where we need to disable iam accesskeys > 90 days and we have few exemptions, like some access-key should be disable after 120 days and 180 days. we storing all that info in s3 bucket in a json file. Is there a way I can put all this exceptions in one policy?

17 replies

Brian Bohrer

@BrianBohrer

Hello - coming back to cloud custodian after some time away. I'm looking to create a policy that would identify/remove ingress rules with 0.0.0.0/0 from security groups, except when the ingress rule accepts that traffic from ports 22 and 80, does this seem possible?

2 replies

DigeratiDad

@digeratidad

Hello - Does anyone know if CC can monitor free IP space in VPCs? I don’t see anything off-hand but asking in case I’m missing it.

1 reply

Chaitanya Tyagi

@chay2199

Hello - Is there a way to get all attached policies to an IAM role in AWS.

KISStian

@KISStian

I experienced a situation where a CloudTrail-based policy failed took an exception on an action as a result of a conflicting operation already in progress. However, the policy continue executing other actions like notify. Is there any way to prevent this from occurring without having to include delays? The resource was cleaned up by a periodic policy that ran later, but two notifications went out even though only one was successful.

5 replies

moshe

@ohaionm_twitter

Hey all, I am trying to install custodian from the repository as instructed in the documentation:
$ python3 -m venv custodian
$ source custodian/bin/activate
$ git clone https://github.com/cloud-custodian/cloud-custodian.git
$ pip install -e ./cloud-custodian
$ pip install -e ./cloud-custodian/tools/c7n_gcp

It seems that it didn't distribute the c7n/ and maybe more packages. What am I missing ? Why when I install using the released packages the package indeed distributed? Same goes for c7n_gcp.

Thanks

5 replies

Marco Ceppi

@marcoceppi

Hey everyone! I've submitted a proposal for how we could implement policy testing for c7n policies.

cloud-custodian/cloud-custodian#6407

Looking for feedback on the proposal from a "👍" reaction on the issue or feedback in comments on the issue or the Google Document proposal are welcome ❤️

Jeff Welling

@jwcmd

Hey there! I'm trying to craft a filter that looks for detached GCP Disks, I can see an attribute in 'gcloud compute disks describe' but it doesn't show up in 'gcloud compute disks list'. Can I filter on a 'describe' attribute in cloudcustodian? Each of my attempts so far have failed but I'm pretty new to this..

Jeff Welling

@jwcmd

Ah nm, I got it :D

tomarv2

@tomarv2

In Azure, I am trying to find all the network-interfaces without NSG's attached to it.

Screen Shot 2021-01-21 at 6.42.33 PM.png

2 replies

Adam Kosmin

@windowsrefund

greetings. I'm familiar with c7n's -m flag to ship metrics to a cloud provider but know I'm going to need to get them into Datadog instead. Does anyone have a pointer on how to accomplish that?

1 reply

Adam Kosmin

@windowsrefund

n/m, looks like the standard DD integration is used or c7n_mailer

jbgachot

@jbgachot

Hello amazing community. I'm having some trouble setting up c7n_mailer with ses mail forwarding. My issue is : I'm running c7n-mailer (with the run flag, -c etc...) and I have an assume role setted up in my mailer.yml with the "role" key. It seems to work as expected for the retrieval of sqs messages, but the role doesn't seems to be used for ses sending of raw email. Is this normal behavior ?

3 replies

Jimi Sanchez

@jimilinuxguy

is it possible to have the json results that are created after running be minified? is there a way to over-ride the default ?

1 reply

Reginald Salisbury

@nixomancer_twitter

Hi, all. I'm trying to set up a policy that checks for aws.iam-group resources that are a) empty and older than 15 days or b) non-empty and haven't been used in more than 15 days. I've tried with something like this, but I can't seem to get it working - does this look about right to anyone?

filters:
    - type: has-users
       has-users: False
    - or:
        - type: usage
           match-operator: all
           TotalAuthenticatedEntities: 0
        - type: value
           value: CreateDate
           value_type: age
           op: greater-than
           value: 15
    - type: usage
       match-operator: all
       LastAuthenticated:
           type: value
           value_type: age
           op: greater-than
           value: 15

Morgan McEntire

@mmcenti

Trying to deploy a custodian policy as a config rule and running into this error:

botocore.errorfactory.InsufficientPermissionsException: An error occurred (InsufficientPermissionsException) when calling the PutConfigRule operation: The AWS Lambda function arn:aws:lambda:us-east-1:<acc>:function:<function_name> cannot be invoked. Check the specified function ARN, and check the function's permissions.

However, the role defined in the policy has config: and lambda: as permissions and I see the resource policy set up that looks correct. What am I missing here?

7 replies

satvan23

@satvan23

Guys, I have this policy to remove ports 21/23 if in a security group. But what happens is , it removes only port 21 not port 23. So, it removes only the first entry there. Any ideas ? I tried removing "or" but same problem.

`policies:

- name: security-group-revoke-21-23
  resource: security-group
  filters:
   - or:
      - type: ingress
        IpProtocol: tcp
        Ports: [21,23]
  actions:
    - type: remove-permissions
      ingress: matched

sirjana

@sirjana

Can anyone tell me the resource name of Lightsail instance. I have used aws.lightsail-instance and lightsail-instance but it's not working. I have used action: stop.

2 replies

Matt Clark

@matticulous

Hi all, I'm running into a problem with testing policies against lambda that I'm hoping someone can help provide direction on. Context: I have a policy with a super-simple filter based on tag presence (e.g. "tag:name: absent"). When I run this using custodian, I get expected results. When I run using the testing framework (test fixture, record_flight_data, etc), I'm getting very inconsistent results. Specifically, a very common case shows the tags attribute on the resource is empty when using the testing framework vs. custodian itself.

7 replies

mblonsky

@mblonsky

Hi folks, i understand based on the documentation that you can run a cloud-custodian instance against multiple accounts using c7n-org. I'm guessing AWS Organizations is involved in this process some way. Can you please tell me what "special" needs to be done to allow a cloud custodian instance to manage multiple accounts?

Steve-Groner

@Steve-Groner

@mblonsky AWS Organizations are not part of this process. c7n-org basically uses cross account roles to manage CC policies in other accounts. https://github.com/cloud-custodian/cloud-custodian/tree/master/tools/c7n_org

@mblonsky Hope these links help. https://cloudcustodian.io/docs/tools/c7n-org.html

mblonsky

@mblonsky

@Steve-Groner Thanks Steve for the info. It seems like the cross account access would be the trickiest part. Do you have any examples of how the correct policies were set up to allow this cross-account access? Thanks again

Steve-Groner

@Steve-Groner

@mblonsky the send link I sent you talks you through that

mblonsky

@mblonsky

@Steve-Groner Thank you Steve, i do see where it describes how to configure the policy file for c7n, i may be missing where the cross account access is configured within AWS

Steve-Groner

@Steve-Groner

@mblonsky Here is the role policy example

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<account_id_where_c7n-org_is_invoked>:root",
          "arn:aws:iam::<account_id_where_c7n-org_is_invokded>:role/CloudCustodian"
        ],
        "Service": [
          "lambda.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

That would be in added to the role in the receiving account. The account ID is from the account where you actually run CC from

mblonsky

@mblonsky

@Steve-Groner Thank's Steve, that is very helpful!

Christopher Dearie

@chris2fer_gitlab

Hi everyone. I'm trying to deploy the mailer on Azure but each time i do, the function runtime complains about missing azure.graphrbac. Well i gave up trying to get the function to work so now i have created a container to run on a container instance. I'm wondering what my authentication options are. It doesn't seem like MSI works for the mailer container.

4 replies

jmarkham24

@jmarkham24

Hi team, new to CC. Is there a way to inject environment variables into the policies? I am deploying an EC2 instance to run the policies with specific tags associated within. I then pull the ec2 describe tag properties from the EC2 and export them as environment variables. I'd like to then use those values to tag the lambda function deployed during a policy run. I can create the env variables via export. I can tag the lambda function with static tag values, but when I tried to insert an environment variable as the tag value it doesn't work. Is this possible? If so, does anyone have an example of a policy file with the correct format for injecting environment variables? Many Thanks

Christopher Pitstick

@cpitstick-argo

I would also like to know the above. For instance, when I tag an ec2 instance as idle with low CPU utilization, I'd like the value from CloudWatch to be put into the tag itself. Something like "cloud_custodian_cpu_idle": "5.67%"

But it looks like that may not be possible, as interpolation in tag values seems limited to only 3 variables right now?
https://github.com/cloud-custodian/cloud-custodian/blame/18c6d1ef72e6033b8b08b06761cc99c8772c2054/c7n/tags.py#L425

Christopher Pitstick

@cpitstick-argo

@Steve-Groner One annoying thing about the c7n-org addition is the orgaccounts plugin doesn't have its own docker image. So if you're, say, running c7n in Kubernetes like I am, you have to build the docker image yourself (https://github.com/cloud-custodian/cloud-custodian/blob/master/tools/c7n_org/scripts/orgaccounts.py)

Where communities thrive

People

Repo info

Activity