Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

Ben Khmelnytskiy

@benkhmelnytskiy

hello,

does anyone here have experience with running custodian tool in AWS Lambda and reading info from SQS?

for example, you send to SQS info about which policy and for which account it should be applied, Lambda is triggered by an event from SQS, gets all this info, and pass these parameters to custodian inside lambda container

1 reply

Abel

@Abikjose

@kapilt Do we have an action to create azure disk snapshot yet?

1 reply

aakshaik2

@aakifshaikh

Azure refer to this resource as Container Instances - Under WHAT resource type does Cloud Custodian covers it? Is there a mapping of what Azure calls and Cloud Custodian calls

3 replies

sadik13

@sadik13

Hi Team, How to get the age of the root account password? please provide your suggestions for preparing the policy. Thank you!!!

Adam Kosmin

@windowsrefund

Hello, I don't seem to be able to generate a report from S3 anymore despite the fact that my runs do succeed and store data in my bucket

I've rules out permissions, and have verified both run and reporting functionality using a local directory for -s

18 replies

Adam Kosmin

@windowsrefund

custodian report -s s3://my-damn-bucket/output policy.yaml --profile mylegitprofile

where policy.yaml contains just a single policy that uses the aws.s3 resource to look for buckets that don't have a set of tags. That policy.yaml is valid since it has produced daily outputs into the s3 bucket I'm trying to report against

Mihir Deshpande

@mihirvijdeshpande

Hello! Do we have any action that can enable encryption on EFS?

#6590

Hi Team, How to get the age of the root account password? please provide your suggestions for preparing the policy. Thank you!!!

#6589

Mihir Deshpande

@mihirvijdeshpande

Do we have any way of notifying if a custom domain is created with a specific TLSv?

3 replies

aakshaik2

@aakifshaikh

what is the command to see the installed c7n-azure version?

1 reply

aakshaik2

@aakifshaikh

Anyone ingesting cloud custodian logs from AZURE blob storage into SumoLogic SIEM? Need help please? Ping me.

1 reply

codehead1997

@codehead1997

I am using below policy to invoke a lambda for remediation, my question is how can i ensure that lambda gets invoked and executed properly? Is SQS integration for action type invoke-lambda is possible?
policies:

name: ec2-public-ip-check
resource: ec2
mode:
type: config-rule
role: arn:aws:iam::xxxxxxxxxxxx:role/ec2-public-ip-check-lambda-role
description: |
If a EC2 instance has Risk tag as "high" and a public IP attached, take some action.
filters:
- "tag:Risk": "High"
- "PublicIpAddress": present
  actions:
- type: invoke-lambda
  function: ec2-public-ip-remediation
  batch_size: 1
  async: true

Adam Kosmin

@windowsrefund

Using 0.9.10, which I've used successfully many times, I'm now seeing oddness where I can't report against data that I know has been output into my S3 bucket. My runs are triggered via a cronjob and I've verified the job has sent output into the bucket. However, I continue to see 0 hits when I run my reports. The crazy thing is that if I issue a run from my local binary (the same binary I use to report), I can then get the corresponding hits when I report. Would anyone be willing to get into the weeds of this with me? I'm convinced it's something obvious I'm overlooking

7 replies

nitro

@nitrocode

is there a way i can create a custodian lambda that can copy ec2 image tags from account A to accounts B, C, D ?

2 replies

jfricioni

@jfricioni

Is there a way for me to create two eventbridges for one lambda? I want to run on the 2nd and 3rd Saturday of a month to handle OS patching but I would prefer to have my accounts/regions as clean as possible and not have two lambdas that do the same thing but on different schedules

Michael Nguyen

@micnguyen266

Kind of a newbie question, but how do you guys upgrade Cloud-Custodian easily? I'm currently on version 0.9.4.0 and was wondering how can I upgrade without overwriting certain files such as the mailer.yml, makefile, Dockerfile, etc due to custom things I have added.
Just a little context on my env, I would push my changes to our Cloud-Custodian repo then a Jenkins job would kick off to create an image and send it to our main AWS account in ECR. Then I have fargate tasks that run on a Cloudwatch schedule that pulls from the Cloud Custodian image in ECR. Also running c7n-org from these tasks.

shravs_125

@shravs_125:matrix.org

[m]

I'm trying to publish reports to datadog - primary goal is to send a notification to team with list of resources which are not meeting tagging policies, Can someone help me how i can solve this @kapilt

srikanthcs

@srikanthcs

Does custodian support policies for apigatewayv2 (http and websocket api) ?
The docs have filters/actions for rest-api resource, and rest-api resource doesn't filter out http api.

indrajeet singh

@indraindrajit71:matrix.org

[m]

Hi team. Just started exploring custodian and testing on aws.
Created a simple yaml for stop and start an ec2 instance with a tag applied on the instance. Problem is this that if I stop an instance it works fine, but when I try to start it again, it gives error. To add: Before stopping the instance it was made sure that all checks has been passed in Status checks. However, after an interval of like 28 minutes approx, the same yaml gets executed successfully and the instance gets stopped or started. can someone please let me know if something wrong. I hope cusotdian checks the instances status checks post executing. Mnay thanks !

policies:
  - name: my-first-policy
    resource: aws.ec2
    filters:
      - "tag:custodian": present
    actions:    
      - stop

Error/warning it gives post starting the instance and using the same yaml with "start" in action

2021-04-02 19:47:57,891: custodian.policy:INFO policy:my-first-policy resource:aws.ec2 region:us-east-1 count:1 time:0.00
2021-04-02 19:47:57,892: custodian.actions:WARNING stop implicitly filtered 0 of 1 resources key:State.Name on running
2021-04-02 19:47:57,892: custodian.policy:INFO policy:my-first-policy action:stop resources:1 execution_time:0.00

After 28 minutes the same command gets executed successfully

custodian run  -s out custodian.yml --region=us-east-1
2021-04-02 20:11:03,541: custodian.policy:INFO policy:my-first-policy resource:aws.ec2 region:us-east-1 count:1 time:63.05
2021-04-02 20:12:05,418: custodian.policy:INFO policy:my-first-policy action:stop resources:1 execution_time:61.88

myoung34

@myoung34

Im hitting a mental wall…..

Im testing out c7n-org

Basically I have a role cloud-custodian in all my org accounts that can be assumed by a single one of them via stacksets

Thats working.
The stackset that creates the arn:aws:iam::{acct id}:role/cloud-custodian role has a policy:

            "Action": [
                "securityhub:BatchImportFindings"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }

But Im getting the error
2021-04-02 15:16:14,308: c7n_org:ERROR Exception running policy:iam-users-with-active-keys account:users region:us-east-1 error:An error occurred (AccessDeniedException) when calling the BatchImportFindings operation: User: arn:aws:sts::{acct id}:assumed-role/cloud-custodian/CloudCustodian is not authorized to perform: securityhub:BatchImportFindings on resource: arn:aws:securityhub:us-east-1::product/cloud-custodian/cloud-custodian

Is assumed-role different from my role permissions? Im not sure why its not picking up that permission

7 replies

Derek Egel

@egelnoteagle

I was wondering how I would be able to use filters to create a resources.json file that only lists s3 buckets (as well as any other resource such as ec2 instances, asg, etc...) with no tags at all

3 replies

Derek Egel

@egelnoteagle

policies:
  - name: tag-compliance
    resource: s3
    comment: | 
      Report on total count of non compliant buckets
    filters:
      - or:
        - tag:"*": absent

This returns all s3 buckets in the account

myoung34

@myoung34

@kapilt New feature request: cloud-custodian/cloud-custodian#6602

Open to feedback

Donato Azevedo

@donatoaz

Hi everyone! new to cloud-custodian here. I was wondering if there is a "pretty-print" solution that summarizes somehow the results of filters. I still have no actions set up, and am using it with a ReadOnly role. So before I start rolling my own solution to parse the resulting resources.json files, I just wanted to check with y'all

5 replies

aakshaik2

@aakifshaikh

aws.insight-rule is equivalent to AWS Cloudwatch Event Rules? Do we have aws.event-rule as separate resource types?

Donato Azevedo

@donatoaz

Is there a way I can have a policy for an ebs volume to make sure it's tagged with the same tag key and value as that of the ec2 instance it is attached to?

2 replies

Donato Azevedo

@donatoaz

Does custodian support notifying to a plain sns topic? I saw there is a generic webhook target (and probably I could do an API GW - SNS service integration easily) but just wanted to ask first

2 replies

Donato Azevedo

@donatoaz

Honest question from non-native english speaker: what's the deal with "maid_" as a prefix on tags?

6 replies

Donato Azevedo

@donatoaz

Me again... is there a way to put_metric using the value of tag as a dimension (or as a metric name for that matter)?

4 replies

Donato Azevedo

@donatoaz

Does anyone have any tips on how to manage DRYness? I'm seeing myself repeating a lot (I am implementing the tagging policy, and having to repeat many policies for each relevant resource type)

- name: ec2-instance-tagging-policy
  resource: ec2
  filter:
    - <<: *invalid_circle_tag

-name: ebs-volume-tagging-policy
  resource: ebs
  filter:
    - <<: *invalid_circle_tag

...

2 replies

cleo2525

@cleo2525

I'm trying to filter for s3 bucket policies that contain certain actions. I know I can use the 'has-statement' filter with 'action' option. But I was wondering if I could use the generic 'value' filter with 'op: regex' so I don't have to list out every action I am filtering for.

I pulled the available keys from the 's3' resource and I see the bucket policy is contained in the "Policy" key. I want to narrow my filter down to just look at the Action section of the policy when applying the regex. But I'm not sure how to specify that in the filter. The following filter isn't working.

policies:
  - name: s3-filter-test
    resource: s3
    filters:
      - type: value
        key: Policy.Statement.Action

princeshalem

@princeshalem

I'm trying to write a policy, which will automatically create a CloudWatch Event Rule triggered Lambda function in my account. But my policy is validate successfully . but it does not trigger or mail to me.

princeshalem

@princeshalem

olicies:

name: invalid-ip-address-login-detected
resource: account
description: |
Notifies on invalid external IP console logins
mode:
type: cloudtrail
role: CloudCustodian-QuickStart
events:
```
  - ConsoleLogin
```
filters:
- not:
  - type: event
    key: 'detail.sourceIPAddress'
    value: |
    '^((158.103.|142.179.|187.39.)([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
    .([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]))|(12.([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5])
    .([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]).([01]?[0-9]?[0-9]|2[0-4][0-9]|25[0-5]))$'
    op: regex
    actions:
- type: notify
  template: default.html
  priority_header: 1
  subject: "Login From Invalid IP Detected - [custodian {{ account }} - {{ region }}]"
  violation_desc: "A User Has Logged In Externally From A Invalid IP Address Outside The Company's Range:"
  action_desc: |
```
"Please investigate and revoke the invalid session along
with any other restrictive actions if appropriate"
```
  to:
  - davidprinceshalem@gmail.com
    transport:
    type: sqs
    queue: https://sqs.us-east-1.amazonaws.com/12345678900/cloud-custodian-mailer
    region: us-east-1

3 replies

pentagonal-proboscis

@pentagonal-proboscis

Hi there - I have created a policy in an attempt to return all AWS iam users + their credentials as follows:

policies:
- name: iam-users-inventory
  resource: iam-user
  region: us-east-1
  filters: 
    - or:
      - type: credential
        key: password_enabled
        value: false
      - type: credential
        key: password_enabled
        value: true

However, this doesn't return every user in my account - is there something I am obviously doing wrong here?

5 replies

Dave Van

@dvan1_gitlab

Hi, has anyone tried out azure multi-cloud support in the most recent release? Is it only supported in custodian or does c7n_org have support for it as well? I'm attempting to call ./c7n-org run -c azure_gov_subscription.yml -s output -u my_policy.yml --dryrun --region=AzureUSGovernment --verbose and it seems to produce an error correlating to commercial authentication endpoints login.microsoftonline.com instead of gov ones.

azure_gov_subscription.yml has the subscription name and ID that is located in Azure US Gov Cloud
my_policy.yml is a simple policy that should return the # of virtual machines under the specified subscription in azure_gov_subscription.yml

9 replies

cleo2525

@cleo2525

I'm having trouble getting the cross-account whitelist_conditions filter to work on my s3 policies. I have a test policy like this

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "OrgReadAccess",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::devprivbucket-#######",
                "arn:aws:s3:::devprivbucket-#######/*"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-##########"
                }
            }
        }
    ]
}

And my filter looks like this

policies:
  - name: public-access-policy-s3-mark
    resource: s3
    description: |
      tag bucket with public read access policy and send notifications
    filters:
      - type: check-public-block
      - type: cross-account
        everyone_only: true
      - type: cross-account
        whitelist_conditions: ["StringEquals"]

When I run this filter, it matches on buckets with the whitelisted condition. My understanding is that 'whitelisted_conditions" should make the filter skip the bucket policy

10 replies

Ben Khmelnytskiy

@benkhmelnytskiy

hey,
does anybody know why it happen?

2021-04-06 17:31:21,716: c7n.policies:WARNING policy:SecCCKmsRule resources:kms-key not available in region: us-east-1
2021-04-06 17:31:21,716: custodian.commands:WARNING Empty policy file(s). Nothing to do.

I'm trying to run custodian in lambda container and facing the above error; but when running the same policy from command line with exactly the same parameters - it works without any errors

1 reply

pentagonal-proboscis

@pentagonal-proboscis

Is it possible to specify which order c7n policies run in? Or to specify dependencies between them? e.g. run X after Y has finished?

10 replies

Jimmyd84

@Jimmyd84

For sqs custodian policy has anyone tried to used a cross account key for encryption? I tried and it didn't work.
actions:

   - type: set-bucket-encryption
     crypto: aws:kms
     key: xxxx

I added the full cross account alias and it appended the local account number in front of it.

3 replies

Jothibasu

@jblinuxclicks

Hi All.. i am new to Cloud-Custodian
Any one could you please help me to find out the AMi and Parents AMI to filter out the ec2 instance and stop action for the same
So we can enforce users should use the new AMI on every 60days, Please help me out

3 replies

rkalva

@rkalva

Hi All. I have a requirement where am trying to filter the data for nested JSON values. The value am trying to filter is "Type" and "Protocol" under the DefaultActions. Can anyone help. Below is the JSON and the syntax am trying which is unfortunately not working.

"c7n:MatchedListeners": [
      {
        "ListenerArn": "arn:aws:elasticloadbalancing:us-west-2:181167:listener/app/testhttpredirect/e2af8aa22/1c9c2c9c",
        "LoadBalancerArn": "arn:aws:elasticloadbalancing:us-west-2:181167:loadbalancer/app/testhttpredirect/e2af8aa22",
        "Port": 80,
        "Protocol": "HTTP",
        "DefaultActions": [
          {
            "Type": "redirect",
            "Order": 1,
            "RedirectConfig": {
              "Protocol": "HTTPS",
              "Port": "443",
              "Host": "#{host}",
              "Path": "/#{path}",
              "Query": "#{query}",
              "StatusCode": "HTTP_301"
            }
          }
        ]

Filter:

- type: listener
  key: DefaultActions.Type
  value: "redirect"
  op: eq

4 replies

Mike

@mikejgray

Value filter question...kind of random, but has anyone identified a way to do something like a csv2dict from a file but use the values as RegEx filters? Not that we need more sub-filters...but this would be handy for a particular use case we have.

2 replies

mogmismo

@mogmismo_twitter

with the aws.ec2 resource, I would like to use cloudtrail execution mode on the RunInstances event, but be able to use the reduce filter on the total number of instances in the account, not on the number of instances in the event itself (as found by the ids). Is this possible?

8 replies

Deborshi Choudhury

@choudhde

API gateway question... I need to implement a PRIVATE api gateway which should have a specific policy document, it should only allow traffic from a specific centralized VPC endpoint . Need to figure out if it's possible with cloud custodian. Any help would be much appreciated. Attaching the describe api ..

  {
    "id": "xxxxxxxxx",
    "name": "ssm_support_api",
    "description": "API supporting centralizaed SSM EC2 registration.",
    "createdDate": "2021-03-22T16:14:13+00:00",
    "apiKeySource": "HEADER",
    "endpointConfiguration": {
      "types": [
        "PRIVATE"
      ]
    },
    "policy": "{\\\"Version\\\":\\\"2012-10-17\\\",\\\"Statement\\\":[{\\\"Effect\\\":\\\"Deny\\\",\\\"Principal\\\":\\\"*\\\",\\\"Action\\\":\\\"execute-api:Invoke\\\",\\\"Resource\\\":\\\"arn:aws:execute-api:us-east-1:xxxxxxxxx:xxxxxxxxx\\/*\\\",\\\"Condition\\\":{\\\"StringNotEquals\\\":{\\\"aws:SourceVpce\\\":\\\"vpce-xxxxxxxxxxxxxx\\\"}}},{\\\"Effect\\\":\\\"Allow\\\",\\\"Principal\\\":\\\"*\\\",\\\"Action\\\":\\\"execute-api:Invoke\\\",\\\"Resource\\\":\\\"arn:aws:execute-api:us-east-1:xxxxxxxxx:xxxxxxxxxx\\/*\\\"}]}",
    "Tags": [],
    "c7n:MatchedFilters": [
      "policy"
    ]
  }
]

3 replies

jmahowald-slalom

@jmahowald-slalom

I'm not seeing anywhere if we could within the same policy specify both a config rule that is triggered on both config changes as well as periodic. This makes it hard from what I can see to update the policy for instance for IAM roles. We wanted to exclude service based roles and updated the policy. But I haven't successfully figured out how to rerun config for those items already marked non compliant without something like that ability.

Christopher Pitstick

@cpitstick-argo

Is there a way to get Cloud Custodian to consume Prometheus metrics on AWS instead of Amazon CloudWatch? We're using Node Exporter from Kubernetes clusters and we'd like to use Prometheus metrics to avoid the high cost of large quantities of AWS CloudWatch metrics.

1 reply

jfricioni

@jfricioni

Is there a document/video/etc on deploy custodian policies through github and github actions or the like? Trying to build out an actual pipeline to automate everything for my company and IT users who want to deploy policies out without having a dozen people ssh to an ec2 instance to deploy.

Where communities thrive

People

Repo info

Activity