Cloud Custodian/cloud-custodian

Rules engine for AWS management, DSL in yaml for query, filter, and actions on resources

Alexander Qiu

@aq17

Hi! Does anyone know if CC supports granular pattern matching? Specifically, for a GCP resource's set-iam-policy action, instead of listing all users in the remove-bindings list, being able to just say something like user:* ?

aakshaik2

@aakifshaikh

In order to use c7n-trailcreator, is there an instruction on how to create a sqlite db? Does the script creates by itself or no?

1 reply

jvoeller

@jvoeller

Hello,
I was trying to create policy that checks if a VNet subnet has a security group attached to it. I'd also settle for a filter that just outputs the VNet that has a subnet missing a security group.
Using a value filter on the vnet resource it shows all the subnets, but I found no way to iterate over them to check for an existing properties.subnets[].properties.networkSecurityGroup key. Using the present value does not work without a number in the brackets. Anyone got an idea how to get around that?

1 reply

Abel

@Abikjose

I'm getting this error

Error: Invalid base64-encoded string: number of data characters (53) cannot be 1 more than a multiple of 4 Unable to base64 decode slack_token, will assume plaintext.

at lambda

This is my mailer.

queue_url: https://sqs.ap-southeast-1.amazonaws.com/123456789075/-custodian role: arn:aws:iam::123456789075:role/custodian-lambda-sqs-readonly slack_token: xoxb-123456789432-1234567890000-BhfNXkkVnXfeJ49ypb5kUH4C from_address: abc@abc.com region: ap-southeast-1

1 reply

Used slack template as slack_template: slack_default

Abel

@Abikjose

I'm sending notification to both email and slack.

Any idea?

Abel

@Abikjose

@LykinsN @thisisshi @kapilt

paulc75-sco

@paulc75-sco

Hello all. I keep seeing the error bellow when creating policies for cloud trail events. I have been able to create it with EC2 and S3 no bother but on all other modules i see this error with different events. File "e:\venv\custodian\lib\site-packages\c7n\policy.py", line 618, in validate
assert e in CloudWatchEvents.trail_events, "event shortcut not defined: %s" % e
AssertionError: event shortcut not defined: CreateNatGateway

is it an authoring error on my part.

Sample code below.

policies:

name: VPC-Tag-Compliance
resource: vpc
mode:
type: cloudtrail
events:
```
- CreateNatGateway
- CreateNetworkAcl
- CreateNetworkInterface
- CreateRouteTable
- CreateSecurityGroup
- CreateVpc
- CreateVpnGateway
```
role: xxx
timeout: 900
actions:
- type: tag
  tags:
  application-name: test 1
  business-unit: test 2
  contact-email: test 3
  environment: test 4
  group-project: test 5
  operating-centre: test 6
  owner: test 7
  short-description: test 8
  use-context: test 9

6 replies

smithjamiej

@smithjamiej

Is there a way to put a AWS tag value into a violation_desc, action_desc, or email subject for reference on the alert

Ananth Balasubramanian

@linuxananth1976

Hey, I have a query
whether I'm missing anything or the behaviour is like that only?
in c7n_mailer aws lambda i don't see any logging in cloudwatch functions
logs except lambda standard logs i.e, start, end and report logging.
when I modified the code as below it works.

logging.getLogger('botocore').setLevel(logging.WARNING) ==> logger.setLevel(logging.INFO)

Can you please confirm the same if i.e, the case, can we have it as an argument to not disturb the code.

10 replies

Atul Jadhav

@tulJadhav_twitter

policies:

name: ec2-ssm-check
resource: ec2
filters:
- type: ssm
  key: PlatformName
  op: ne
  value_from:
  url: file:file.txt
  format : txt

have been using this policy to count the number of Ubuntu instances, file.txt contains 'Ubuntu' on the first line, the policy does not read value from the given file to match for value, hence also counts all other instances as well, what could be the mistake here

3 replies

Sonia Gurdian

@PendragonDay

Help, I'm trying to write a policy to detect if an rds-snapshot has been made public, then the action will be to delete it. However when looking at the output of the describe-db-snapshot cli command, there is one option that could be used for a generic filter which is "SnapshotType": however even when I have set the snapshot to public, the "SnapshotType" remains as "manual" there is nothing in the output that indicates the snapshot is public. How can I make Cloud Custodian detect this? This is my policy:

  - name: aws-rds-snapshot-PubliclyAccesible-rem
    resource: rds-snapshot
    description: Deletes RDS public snapshots.
    filters:
      - type: value
        key: SnapshotType
        op: eq
        value: public
    actions:
      - delete

The filter on the value: public is not detecting, because it remains manual even when I have made the snapshot public. Cross-account option is not equivalent because I can still make an rds-snapshot public without giving cross-account access to another account. deleting based on cross-account will delete snapshots that are not public and will not delete those that are public. Any help will be appreciate it!

4 replies

Pradeep Reddy

@prareed_twitter

Hello all,
When I run the policy and redirect the output to S3, it is saving as .gz. Is there a way to save the output in json or csv?
custodian run --output-dir s3://devops-bucket instance-type-stop-action.yml

2 replies

hamzazai2021

@hamzazai2021

Hi, is there any instruction on how to provide secure string for Smtp_password in mailer ?

8 replies

myoung34

@myoung34

am i missing something?

  - name: engine-admin-assume-role-detected
    resource: account
    description: A Team Engine admin sso assume role has occurred
    mode:
      type: cloudtrail
      role: arn:aws:iam::{account_id}:role/cloud-custodian
      events:
        - source: sts.amazonaws.com
          event: AssumeRoleWithSAML
          ids: "requestParameters.roleSessionName"
    filters:
       - type: event
         key: "detail.responseElements.assumedRoleUser.arn"
         op: regex
         value: "AWSReservedSSO_engine-.+?-admin_.*"

the lambda fired as expected, i took the payload and put it in payload.json:

± cat payload.json | jq .detail.responseElements.assumedRoleUser.arn
"arn:aws:sts::redact:assumed-role/AWSReservedSSO_engine-production-admin_4e122ecb4/marcus.young@redact.com"

so i know the regex is right:

but:

Filter #1 applied 1->0 filter: {"type": "event", "key": "detail.responseElements.assumedRoleUse
r.arn", "op": "regex", "value": "AWSReservedSSO_engine-.+?-admin_.*"}

Filtered from 1 to 0 account

4 replies

Ananth Balasubramanian

@linuxananth1976

Hello, I have couple of questions as below:

I want to know about whether c7n org execution logs in ec2 can be logged in CloudWatch log groups?
SQS Trigger instead of Scheduler can we have as Real time events trigger? the reason is i don't want to run c7n mailer Lambda every 5 mins to reduce the cost instead of it having trigger as events.

2 replies

Ajay Misra

@ajmsra

hello I have a policy for ec2-off hours and have configured slack notification when the instances are stopped or started

    actions:
      - start
      - type: notify
        to:
        {% if sns=='True' %}
          - arn:aws:sns:us-east-1:{{ aws_account_id }}:c7n-mailer
        {% endif %}
        {% if slack_channel %}
          - slack://#{{ slack_channel }}
        {% endif %}
        action_desc: The above EC2 instances are Stopped
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/{{ aws_account_id }}/c7n-mailer

I have not defined violation_desc but still when the notification comes on slack violation_desc is marked as blank

How can I make sure that a blank violation_desc is not posted as this is not a violation of any rule we are just stopping and starting instances

2 replies

@kapilt

Sean Benton

@seanmac5_twitter

Hi, we're just getting started on GCP and trying to understand what capability exists to filter on compute instances metrics. We're getting the error:

2021-06-07 14:11:56,859: custodian.commands:ERROR invalid policy file: test.yml error: instance.filters Invalid filter type {'type': 'metrics'}

Oddly the example in the doc refers to gcp.firewall (not gcp.instance) so am I just trying to do something that's not supported?

5 replies

pentagonal-proboscis

@pentagonal-proboscis

is there a way with the aws.rrset resource to only pull back records which are in a public hosted zone? I have managed to filter down to just A records and CNAME, but also want to filter on public zone.

5 replies

Abel

@Abikjose

Getting this error from function app.

Result: Failure Exception: ModuleNotFoundError: No module named 'azure.identity'. Troubleshooting Guide: https://aka.ms/functions-modulenotfound Stack: File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/dispatcher.py", line 305, in _handle__function_load_request func_request.metadata.entry_point) File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 42, in call raise extend_exception_message(e, message) File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 40, in call return func(*args, **kwargs) File "/azure-functions-host/workers/python/3.6/LINUX/X64/azure_functions_worker/loader.py", line 83, in load_function mod = importlib.import_module(fullmodname) File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/home/site/wwwroot/mailer_697dc9ec-d696-4962-94d9-c4e1fadb0455/function.py", line 11, in <module> from c7n_mailer.azure_mailer import handle File "/home/site/wwwroot/c7n_mailer/azure_mailer/handle.py", line 6, in <module> from c7n_azure.session import Session File "/home/site/wwwroot/c7n_azure/session.py", line 14, in <module> from azure.identity import (AzureCliCredential, ClientSecretCredential,

1 reply

I have done it exactly mentioned in the documentation but I am not able to receive email through sendgrid

I tried sending email to the sendgrid directly and it's working.

CurtisAndersenSysdig

@CurtisAndersenSysdig

Does anyone know if I am doing anything wrong?

 - type: notify
      slack_template: slack-custom-template
      slack_msg_color: '#00F0F0'
      violation_desc: 'test vi'
      to:
        - slack://#custodian-all
        - https://hooks.slack.com/services/T/B/lasdf
        - slack://foo@bar.com
      transport:
        type: sqs

The issue is that both the slack channel and the web hook send the correct notify message to their respective channels. However, the email doesn't seem to start a slack message with the user, but slack does notify the user through an email that they have messages waiting for them. When the email is opened however there is no message. Is there a special way that I need to preface direct messages?

8 replies

Ravindra babu

@Ravindrababu99_twitter

@kapilt

Below event based policy on redshift, policy shows successfully triggered in lambda monitoring but there is no log-stream generated and policy is not adding the user in tags(creatorid).

name: redshift-auto-tag-user
resource: redshift
mode:
type: cloudtrail
role: arn:aws:iam::account-number:role/CloudCustodian
events:
```
- source: redshift.amazonaws.com
  event: CreateCluster
  ids: "requestParameters.ClusterIdentifier"
```
filters:
- "tag:owner": absent
- "tag:creatorid": absent
  actions:
- type: auto-tag-user
  tag: creatorid

5 replies

Shivanjan Chakravorty

@Glitchfix

Hi everyone, Glitchfix here I am looking for some help. Is there any open source repositories to refer policies for any well known compliance?

3 replies

CurtisAndersenSysdig

@CurtisAndersenSysdig

Hi everyone, does any one know if there is a way to have a policy that takes all of the EC2 instances missing particular tags. Then using these EC2 instances could individual notify the owners that their particular instance is missing the specified tag.
Getting the EC2 instances missing the desired tag filtered out is repetitively simple. But, from what I can tell these filtered out instances are then sent over to actions of notify and notify sends all of them at once.

7 replies

Pradeep Reddy

@prareed_twitter

Hello, Is there any plan to support OCI integration?

3 replies

CurtisAndersenSysdig

@CurtisAndersenSysdig

I'm now working on making custom templates for slack notifications. Does anyone have any references of to help in learning how to format these properly>

3 replies

Ray Henson

@RayHenson_twitter

Hello, I am just starting to set up Cloud Custodian for PoC, and of course I need to show something more difficult than just enforcing tagging etc. I was asked to show if I could report when (AWS) an RDS instance is not part of a particular "Parameter Group" != "ssl-group". tried the following:
policies:

name: rds-pg
resource: rds
filters:
- type: value
  key: "DBParameterGroupName"
  op: eq
  value: "ssl-group"

but doesn't work, not sure how to get the JSMEPath expression right for :

        "DBParameterGroups": [
            {
                "DBParameterGroupName": "ssl-group",
                "ParameterApplyStatus": "in-sync"
            }
        ],

6 replies

Shawn L

@slaphitter

Hey folks. Trying to restrict checking the tags of S3 buckets to only run out of AWS us-east-1. When I deploy the policy to a lambda the conditions stanza seems to disappear. Is that expected behavior?

23 replies

    conditions:
      - type: value
        key: region
        op: equal
        value: "us-east-1"

CurtisAndersenSysdig

@CurtisAndersenSysdig

Random question is there a limit on the amount of violations that can be sent in a notification. because if the system finds 664 errors it sends 3 different notifications two of which have 250 elements. This was found in some testing that I was doing

2 replies

codehead1997

@codehead1997

@/all
Can we have a custodian policy which will raise alert if any non admin user attempts to use IAM.

2 replies

CurtisAndersenSysdig

@CurtisAndersenSysdig

I am currently working on a template for slack messages that will send the users with all the violations they have.

{% macro displayInstance(resources) %}
   {% for resource in resources %}
      {% if _lastOwner == getTag(resource, 'AutoTag_Creator') %}
         {{ format_resource(resource, policy['resource']) }}\n
      {% else %}
         AutoTagged Owner: {{ getTag(resource, 'AutoTag_Creator') }} TagedLastOwner: {{ _lastOwner }} {{ format_resource(resource, policy['resource']) }}\n
      {% endif %}
      {% set _lastOwner = getTag(resource, 'AutoTag_Creator') %}
   {% endfor %}
{% endmacro %}

The tagged last owner was so that I could see in side what was happening the problem is that _lastOwner is never set and if I put a set at the top the value never changes. Any ideas?

1 reply

jkoermeride

@jkoermeride

I'm attempting to setup some alerts around TLS 1.1 & TLS 1.2 for AWS resources. I found a way to configure this for elb & cloudfront, but have not been able to get it to work with custom domain names. I'm using the resource: rest-client-certificate. I even tried to looks at detail by just using the sample for rest-client-certificate. In both case I'm seeing the following error:
custodian validate apigateway_invalid_ciphers.yml
2021-06-15 16:20:12,801: custodian.commands:ERROR Configuration invalid: apigateway_invalid_ciphers.yml
2021-06-15 16:20:12,802: custodian.commands:ERROR Error on policy:apigateway-invalid-ciphers resource:rest-client-certificate
{'key': 'createdDate', 'value_type': 'age', 'value': 90, 'op': 'greater-than'} is not valid under any of the given schemas

Here is the sample code I'm trying:

policies:
  - name: apigateway-invalid-ciphers
    resource: rest-client-certificate
    filters:
      - key: createdDate
        value_type: age
        value: 90
        op: greater-than

Does anyone have any experience with this?

1 reply

CurtisAndersenSysdig

@CurtisAndersenSysdig

I have another question about limitations.
When using the line
- slack://tag/channel if there are say for example two EC2 instances however they have separate channel tags does Cloud Custodian just go with the first channel or does it send a message to both?

codehead1997

@codehead1997

@/all
I want to filter all the cloudtrail trails which are not logging. I came up with this policy-

policies:

name: cloudtrail-is-active-check
resource: aws.cloudtrail
filters:
- type: status
  key: IsLogging
  value: true

But this policy doesn't account for the fact that logging bucket doesn't exist or trail is not able to log in it.
I was wondering if there is any way to write a policy which covers all these cases.

Sujith M

@i0sync

hey guys, I'm new to CloudCustodian and need clarification around this - I have a situation - I have a policy which check iam-users-with-active-keys, and as an "action" I'm currently running another lambda function just to generate the report. I'm seeing the action lambda function does get event payload, which is the result of first custodian lambda function. Is there a way to access that event payload (possibly locally) ? The idea here is to have a way to be able to generate reports, I've run c7n-org report but the only output is an empty []. Kinda new to this, so any help would be greatly appreciated !

2 replies

rob hill

@pigeon_pie:matrix.org

[m]

ok this is my first day with custodian lol have created a policy got it to run in aws got it to report to csv , now trying to get itt to push to cloudwatch so i can build dashboard or reports . have run this custodian run --log-group=/custodian/test ec2.yml and asks for required -s output directory , isnt. the log group the output

1 reply

rowbot1

@rowbot1

I'm looking to extract the rules from https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-standards-fsbp-controls.html and apply them to cloud custodian. Has anyone done this before?

cleo2525

@cleo2525

Hi all, I need some clarification on checking the status of IAM user access keys. I need to check the following items:

if access key is enabled
and access key is older than 30 days
and access key has never been used or hasn't been used in greater than 60 days
mark the IAM user for op & notify the owner

6 replies

sadik13

@sadik13

Hi Team, i need to collect the security groups which is having the default SG and associated with any ec2 instance, so please suggest filters and actions for query.

Where communities thrive

People

Repo info

Activity