by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 15:43
  • 15:02
    alextodicescu starred cloud-custodian/cloud-custodian
  • 14:01
    85matthew opened #6029
  • 14:01
    85matthew labeled #6029
  • 13:16
    MichaelDavisTSN labeled #6028
  • 13:16
    MichaelDavisTSN opened #6028
  • 11:40
    kapilt commented #6018
  • 11:36
    kapilt commented #6020
  • 11:35
    kapilt commented #5987
  • 11:33
    kapilt synchronize #6027
  • 11:30
    kapilt commented #6025
  • 11:29
    kapilt commented #6026
  • 10:16
    kapilt synchronize #6027
  • 10:14
    kapilt ready_for_review #6027
  • 10:13
    kapilt synchronize #6027
  • 10:11
    kapilt synchronize #6027
  • 09:11
    kapilt synchronize #6027
  • Aug 10 20:56
    kapilt synchronize #6027
  • Aug 10 18:17
    kapilt edited #6027
  • Aug 10 18:16
    kapilt edited #6027
Stephen McCracken
@mckraken
Question here: Looking into the "invoke-lambda" action. what docs are there on what is passed to the lambda function? Parameters, payload, etc?
2 replies
Michael Nguyen
@micnguyen266
Just wondering how everyone organizes their policies. For example, you have a tag compliance policy that reports on resources that are not compliant. Do you put all aws resources in that one file like all_resources_tagging_report.yml or do you break it down by resource like ec2_tagging_report.yml, asg_tagging_report.yml, lambda_tagging_report.yml etc..
10 replies
allenb-github
@allenb-github
Hello everyone, I'm looking for a way to use CloudCustodian to find stack drift. I don't necessarily need to act on it, I just want it scan and be able to log it so I can follow up on it. Does CloudCustodian have anyways to work on CloudFormation stack drift? Or is that something planned for the future?
Ramachandran Seshadri
@rams3sh_gitlab
Hey All , Is there any possibility of using jmespath query in aws.iam-user resource's usage filter with further filters of type value.
Asking because I am unable to see the usage json details as part of the resources.json to experiment with some jmespath-fu.
Jamison Roberts
@jtroberts83
@rams3sh_gitlab if you just run a generic iam-user policy with some sort of usage filter then you can see what the outputted options are for jmespath filtering
3 replies
Jamison Roberts
@jtroberts83
@all We recently ran into an issue at the company I work for where a couple hundred of our AWS accounts reached the soft limit on Lambda Storage size which then caused our new policies to fail deployment and our customers got a little upset as they couldn't deploy any lambdas in their accounts either. Upon investigation we found that our custodian lambda functions had numerous versions, some having as few a 6 versions but some having 600+ versions. I wrote a custom script to go out and delete all the custodian lambda versions that were not the #Latest version and our lambda storage space went from 100% of the 75 GB down to 0.1% of the 75 GB! While this will not apply to everyone as it depends on how you deploy your lambda policies and the frequency at which you do so. Just thought it was something to be aware of when you are deploying lots and lots of lambda based policies. @kapilt Maybe having an option within aws.lambda for the delete action to delete all non-current versions of lambdas would be helpful. Something like this:
- name: cleanup-old-custodian-lambda-versions
  resource: lambda
  filters:
        - type: value
          key: FunctionName
          op: regex
          value: ^(custodian-?)\w+
  actions:
        - type: delete
          keep-latest: true
6 replies
Ramachandran Seshadri
@rams3sh_gitlab

Hey all,

Is there any way to disable / deactivate ssh-keys , mfa-devices etc.. of an aws iam user without deleting them ?

I referred official documentation , but couldn't find disable as an option for these user attributes.

A sample expected policy given below :-

policies:
      - name: disable-all-access-unless-valid
        resource: iam-user
        filters:
          - type: value
            key: UserName
            op: not-in
            value:
              - valid-user-1
              - valid-user-2
        actions:
          - type: delete
            disable : true  # Would be nice to have this option just like the case of remove-keys
            options:
              - console-access
              - access-keys   # remove-keys has disable: true option to disable access keys but not others
              - mfa-devices
              - ssh-keys
              - signing-certificates
              - service-specific-credentials

Is there any other way possible to achieve the same ?

3 replies
Ash
@abelmokadem
Hi all, any recommendation on how to deploy cloud custodian on AWS? I have some ideas on how I would deploy it. Just wondering if there are some ready made cloudformation templates.
3 replies
Steve Duys
@SteveDevOps
Hi there. Is it possible to watch multiple volume events in one yml ? examples?
Steve Duys
@SteveDevOps
(ie.. events: -CreateVolume. -AttachVolume)
4 replies
I can't get it to accept
Michael Nguyen
@micnguyen266
By default I believe Cloud Custodian can send email reports of up to 250 resources per email. This can be found under cloud-custodian/cloud-custodian-0.9.4.0/c7n/actions/notify.py. The # of resources reported can be changed here line 30: batch_size = 250. I was wondering instead of hardcoding this, can batch_size be added to the mailer.yml config instead?
4 replies
satvan23
@satvan23

Guys. So trying to create a policy for ebs createvolume. If volume is not encrypted or if aws default kms key is not used. But I get error, "must use a list for filters found:dict". So what's wrong in these filters ? The policy is as below.

policies:

  • name: delete-unencrypted-ebs-dev-corp
    region: us-east-2
    description: |
    Cloud Custodian will Terminate all unencrypted EBS volumes upon creation and
    send email to Creator/Owner.
    resource: ebs
    mode:
    role: arn:aws:iam::12345:role/ccustodian-ec2-role
    type: cloudtrail
    events:
    - CreateVolume
    filters:
    or:
    • Encrypted: false
    • type: default-ebs-encryption
      key: "alias/aws/ebs"
      state: false
      actions:
28 replies
Karl de Castro Fonseca
@KarlCF
So, regarding the use on c7n-org, can I provide the buckets for output inside each account? For example: I have previously created buckets with names {team-{account_id}-region} for logging outputs in each account. Is there a simple way to implement this?
3 replies
Dario Vianello
@dvianello
Hello! Has anyone managed to use CC to copy tags from a EKS cluster down to the ASGs at the back of the nodegroups?
3 replies
satvan23
@satvan23
Guys. So I have a c7n policy with Lambda running in one account and its good. And c7n-org runs with policies also ( not Lambda ). So, now the question is I have a Lambda running in a single account. Is it possible to use that Lambda from one account across multiple accounts. The Lambda deletes volumes on creation IF the volume is not encrypted.
2 replies
manitmalik
@manitmalik
Hi,
A very generic query. For sending message to sns, do we need to use the c7n mailer ?
4 replies
bhuvanesh_kj
@bhuvaneshkj
Hi,
Is there a way where we can change the output directory of policy output files locally in Azure? I basically want to get outputs of various subscription separately in different folders. Right now I see that the output files are overwritten when policies are executed on diff subscriptions.
1 reply
manitmalik
@manitmalik
Hi,
The message which i am getting when subscribing to the SNS topic which is mentioned in the policy seems to be encrypted. Is this the expected behavior ? How can i decrypt the message .. my objective is to attain the resource.json in the message so that i can parse the info in a lambda and do appropriate actions.
eJztVNty2jAQffdXePzaisiO8e2pHjKZ0hZy....................
5 replies
Mike
@mikejgray

Anyone else run into an issue where sending metrics to the account where c7n-org is run from fails quietly and sends info to CloudWatch instead?

c7n-org run -c test.yml -s ./test -u $POLICY --cache-period 0 -v --metrics aws://master?region=us-east-1 Master fails to run with an error, if I specify an account number it runs but sends info to the target account's CloudWatch instead of the specified account

11 replies
tomarv2
@tomarv2
@kapilt c7n-mailer installed using pip, is throwing error message, tried locally and inside docker:
root@8b79684d85a3:/# c7n-mailer
Traceback (most recent call last):
  File "/usr/local/bin/c7n-mailer", line 5, in <module>
    from c7n_mailer.cli import main
  File "/usr/local/lib/python3.8/dist-packages/c7n_mailer/cli.py", line 9, in <module>
    from c7n_mailer import deploy, utils
  File "/usr/local/lib/python3.8/dist-packages/c7n_mailer/deploy.py", line 19, in <module>
    from c7n.mu import (
ModuleNotFoundError: No module named 'c7n'
9 replies
Louis M.
@LMarkham
- type: value
          key: tag:owner
          op: regex
          value: '^[a-zA-Z0-9_.+-]+@(domain.com)$'
I am trying to get a tag filter that is anything besides xxxx@domain.com. If isnt xxxx@domain.com then it should fail. I am horrible with regex and appreciate any help.
12 replies
farisbacker
@farisbacker
I am trying to match elb based on target group port. Am I missing something here ? This give me incorrect zero result. There are target groups with port 22 under network load balancer.
 policies:
 - name: core-nlb-highrisk
   resource: app-elb
   filters:
      - Type: network
      - type: target-group
        key: "Port"
        value: 22
        op: eq
5 replies
tomarv2
@tomarv2
thanks @kapilt 0.9.3 mailer is not showing that error, but its still not deploying, showing size as 0MB:
c7n_1  | 2020-08-05 14:44:44,356 - custodian.azure.session - INFO - Authenticated [Principal | 12345]
c7n_1  | 2020-08-05 14:44:44,878 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Found Function Application "c7n-mailer-mailer-12345".
c7n_1  | 2020-08-05 14:44:45,171 - c7n_mailer.azure.deploy - INFO - Building function package for c7n-mailer-mailer-12345
c7n_1  | 2020-08-05 14:44:45,345 - c7n_mailer.azure.deploy - INFO - Function package built, size is 0MB
c7n_1  | 2020-08-05 14:44:45,345 - custodian.azure.function_app_utils - INFO - Publishing Function application
c7n_1  | 2020-08-05 14:44:47,167 - custodian.azure.function_package.FunctionPackage - INFO - Publishing Function package from /tmp/tmp0l_j8a53
c7n_1  | 2020-08-05 14:44:53,391 - custodian.azure.function_package.FunctionPackage - INFO - Function publish result: 202
c7n_1  | 2020-08-05 14:44:53,392 - custodian.azure.function_app_utils - INFO - Finished publishing Function application
16 replies
Louis M.
@LMarkham

Making a tag checker for EC2 on Run, Start instances and Create/DeleteTag events from CloudTrail. Got it checking the tags the way that I want and it can stop instances that aren't compliant.

    mode:
      type: cloudtrail
      role: arn:aws:iam::{account_id}:role/role-access
      events:
        - source: ec2.amazonaws.com
          event: RunInstances
          ids: "responseElements.instancesSet.items[].instanceId"
        - source: ec2.amazonaws.com
          event: StartInstances
          ids: "responseElements.instancesSet.items[].instanceId"
        - source: ec2.amazonaws.com
          event: CreateTags
          ids: "requestParameters.resourcesSet.items[].resourceId"
        - source: ec2.amazonaws.com
          event: DeleteTags
          ids: "requestParameters.resourcesSet.items[].resourceId"
    actions:
      - stop

The issue is that if I start a non compliant system it will not stop it and it gives me the error:

[WARNING]    2020-08-05T15:07:17.450Z    64ab1380-45dc-4ad4-9ed4-6091fde67e23    stop implicitly filtered 0 of 1 resources key:State.Name on running

Looks like the system needs to be running in AWS before it will Stop it.

Is there a work around for this? Our goal for this is to be proactive and not reactive on non compliance. We could use mark-for-op but that would leave the system running for a period of time and we would prefer to just stop it as soon as it goes non compliant.

18 replies
mattboret
@mboret
Hi, is it possible to set a default value for a variable? My use case, I want to be able to run a policy with custodian or c7n-org. With c7n-org the variable is set but not with Custodian, I would like to be able to run it as it without defining the variable.
policies:
  - name: ec2-loadbalancers-elb-idle
    resource: elb
    comment: |
      Policy to report idle EBS
    filters:
      - type: metrics
        name: RequestCount
        statistics: Sum
        days: "{elb_idle_days}"
        value: 0
        missing-value: 0
        op: equal
17 replies
jfcottrell
@jfcottrell
I'd like to create a policy for DynamoDB that compares ConsumedReadCapacityUnits to ProvisionedReadCapacityUnits on a table by table basis (and tags tables where the difference is greater than a predetermined amount). Is this even possible and, if so, what would this policy look like?
1 reply
Todd Stansell
@tjstansell
Question regarding ingress security group filters and the cidr checks. Say I want to remove any ingress rules who's Cidr is a subset of any one of a set of cidr blocks ... ideally pulling that list with values_from and an s3 bucket. With an op: in and a list of cidrs, it appears to only do a string match. It works if the cidr in the ingress permission is exactly one of the cidrs on the list. If I change it to value_type: cidr so it evalues these as cidr addresses, it only seems to match if value is a string. if it's a list of cidrs (even with just a single entry) it doesn't seem to match. Is this expected to work somehow?
6 replies
arafatheds
@arafatheds
@kapilt Do we have a way to filter whitelisted principles in vpc endpoint servies..
arafatheds
@arafatheds
Do we have a way to filter whitelisted principles in vpc endpoint servies..
David Barranco
@dbarranco

Hello everyone! :wave:

I wanted to ask you a quick question, looking for some advice on how to craft a webhook call in a Cloud Custodian policy.

My use case is the following:

I am trying to list GKE clusters based on a tag-value filter and would like to delete them.
Given that CloudCustodian doesn't have at this moment the DELETE operation for a GKE cluster, I was planning to use this webhook:

https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters/delete

But I am not sure how can I send the clusters name with Cloud Custodian (or modify any path parameter of the query):

DELETE https://container.googleapis.com/v1/{name=projects/*/locations/*/clusters/*}

Could I get some help? thanks in advance!

Anyone could help me with this?

28 replies
Bheesma
@Bheesma
I created a bunch of policies for testing and want to start over once again. How do I delete all the resources created by Cloud Custodian? This is for AWS.
1 reply
arafatheds
@arafatheds
Does custodian has resource for vpc endpoint services ?
5 replies
Steven Scoleri
@scoleri
does anyone know (offhand) if c7n-org run -t can do a NOT in tag?
2 replies
Jimi Sanchez
@jimilinuxguy
can the vpc flow log checks filter on tags or use a regex on the log group?
trinaryouroboros
@trinaryouroboros
hi maybe this is a simple question, all I'm trying to do is add a current date using like {{ variable }} in the c7n_mailer jinja code, similar to {{ account }} etc., I don't see any documentation that shows what variables are available to use
Michael Davis
@MichaelDavisTSN
first time trying the -t or --templates with mailer, and it's not working, still pulls the default location. this is mailer 0.6.3 and c7n 0.9.4. Anybody else have probs with that?
2 replies
trinaryouroboros
@trinaryouroboros
my question I found the answer to, it was actually in the github repo: https://github.com/cloud-custodian/cloud-custodian/tree/master/tools/c7n_mailer section "Writing an email template"
Ali Bajwa
@AliMBajwa
Hey all - is there a way to ensure IAM policy compliance with a defined 'reference' cfn template?
What were effectively trying to do is detect and correct IAM policies which are changed from a custom approved default policy - ideally we'd like like to be be run from lambda or even aws config but can't seem to wrap my head around how to do it with custodian.
1 reply
manitmalik
@manitmalik
Hi, Generic query on the skew. I can see that skew works on the condition i.e. current_date + skew >= target_date .. is there any other parameter or way we can use skew to have condition as current_date + skew = target_date. We are using skew to remind before remediation. Hence would like to remind only once.
Currently the issue is, event after remediation is done and tag is removed, skew still works for that day, considering i think it takes time for tags to get updated and reflect in the response of the policy
1 reply
sudheesh86
@sudheesh86
How create docker file for custodian image to run policies
David Barranco
@dbarranco

Hello!

I'd need some help finding out the reason why Custodian is filtering some resources in Google Cloud I don't see clear the behaviour of the tool.

Currently I have this policy:

policies:
- name: wipe-gke-clusters
  resource: gcp.gke-cluster
  comment: |
    Clean-up GKE clusters created by Jenkins for its
    CI tests.
  filters:
    - type: value
      key: "tag:created_by"
      value: "jenkins-test"
      op: eq
  actions:
    - type: webhook
      url: "https://container.googleapis.com/v1/"
      method: DELETE
      query-params:
        projects: "testproject"
        zones: resource.region
        clusters: resource.name

But any time I run Custodian I get:

2020-08-07 10:39:48,610: google_auth_httplib2:DEBUG Making request: POST https://oauth2.googleapis.com/token
2020-08-07 10:39:49,014: custodian.resources.kubernetescluster:DEBUG Filtered from 3 to 0 kubernetescluster
2020-08-07 10:39:49,014: custodian.policy:INFO policy:wipe-gke-clusters resource:gcp.gke-cluster region: count:0 time:0.51
2020-08-07 10:39:49,015: custodian.output:DEBUG metric:ResourceCount Count:0 policy:wipe-gke-clusters restype:gcp.gke-cluster scope:policy

I am running the tool with --regions all so it should be working in the region where this clusters are located

as far as I see in Google cloud, the GKE cluster has the correct label created_by: jenkins-test
any idea why it is not matching ? I'm on Custodian 0.8.46.0 btw
2 replies
Landon Dale
@landale4
Any ideas on how to grab a vpc_id when creating cloudwatch log group as part of the enable vpc flow logs policy actions? Trying to follow corporate naming convention when doing remediation.
    - type: set-flow-log
      DeliverLogsPermissionArn: arn:aws:iam::{account_id}:role/flowlogsrole
      LogGroupName: /{vpc_id}-vpcflowlogs/
2 replies
Conor
@lithiumoxide
Hi there. I'm looking to write some jinja2 templates for c7n-mailer, so that I can send to Slack custom details of resources found by a filter made with aws.ami. Does anyone have - or know of - any examples of such templates? I'm finding it hard to find examples online and would like to see something so I can make sure my template is structured well.
1 reply
Jamison Roberts
@jtroberts83
@CYarros10 boto3, you can view the code on Github
Steve Craig
@stevesworkgithub
Hi. I'm getting timeout errors on my c7n-mailer lambda as I'm running a huge policy check on tags which results in a legitimate 500K of policy matches over the course of the day. These are all being sent to Splunk, but am I right in thinking that there's a batch size of 250 sqs reads on the queue, so I'm potentially double counting ~250 of the messages? I don't have an obvious solution to this other than reduce the batch size or increase the lambda timeout, but is my basic assumption correct? Thanks very much.
Kapil Thangavelu
@kapilt
So azure mailer has a regression in 0.9.4 there’s a pr up to address, I’m hopeful to get 0.9.5 out this week to address
9 replies