Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 01:19
    marcoceppi synchronize #6496
  • Mar 01 23:38
    marcoceppi synchronize #6496
  • Mar 01 23:37
    thisisshi assigned #6199
  • Mar 01 23:32
    marcoceppi opened #6496
  • Mar 01 21:06
    kapilt unlabeled #6493
  • Mar 01 21:06
    kapilt labeled #6493
  • Mar 01 21:05
    kapilt labeled #6494
  • Mar 01 18:43
    Lucas-Irvine review_requested #6495
  • Mar 01 18:43
    Lucas-Irvine opened #6495
  • Mar 01 18:24
    ryanash999 opened #6494
  • Mar 01 18:24
    ryanash999 labeled #6494
  • Mar 01 18:21
    alfredgamulo commented #5758
  • Mar 01 15:47
    alfredgamulo closed #6491
  • Mar 01 15:47
    alfredgamulo commented #6491
  • Mar 01 15:24
    alfredgamulo synchronize #6491
  • Mar 01 15:17
    kapilt commented #6491
  • Mar 01 15:14

    kapilt on master

    docs - extend install instructi… (compare)

  • Mar 01 15:14
    kapilt closed #6477
  • Mar 01 05:50
  • Mar 01 05:24
    cguardia commented #6477
ohaionm
@ohaionm
Hey all,
I want to make custodian read my policies from github/bitbucket/gitlab, also when running periodic policies. To accomplish that I should use c7n-policystream tool? And if so, how I use it to instruct custodian to read from git instead of the file system?
Thanks!
3 replies
Matt Clark
@matticulous
Hi, I'm having an issue having c7n-org clean up lambdas using mugc. Things appear to work correctly when using the script natively, but not when executing via c7n-org.
Doesn't work: c7n-org run-script -c accounts.yaml -s c7n-org-out-mugc -a [account] "python scripts/mugc.py -c policy.yml --dryrun"
Works: python scripts/mugc.py -c policy.yml -r us-east-1 --assume [role_copied_from_c7n-org_account_config] --dryrun
Am I missing something important? (I am using the version of mugc from master btw)
4 replies
pjshort22
@pjshort22
Hi all, we are testing to upgrade cloud Custodian from 0.9.5.0 to 0.9.10.0, but have found issue with 0.9.10.0 and using our currect CSV files with value_from using uri, CSV2DICT example below. I know in release 0.9.7.0 where value_from was converted from lists to sets cloud-custodian/cloud-custodian#6043 This cause our CSV files holding Divisions or costscenter to be ignored and tag resources as invalid, when they are not invalid and work fine with 0.9.5.0. our CSV files are 1 line with comma seperated values ie "111","222,""345" etc and work with 0.9.5.0 but on 0.9.10.0 with change to Sets they are not recognized and only was to get 0.9.10.0 to recognize values is have them in a column CSV with a header and new line between value. But this format does not work with 0.9.5.0 the values are ignored. Has anyone else had this problem and how to fix so we can update to 0.9.10.0 without re creating all our CSV files adn without breaking current 0.9.5.0? Example from_value: - type: value
key: "tag:Division"
op: not-in
value_from:
url: s3:/s3bucket/ValidDivisions.csv
format: csv2dict
2 replies
aakshaik2
@aakifshaikh
Question regarding c7n-mailer: We are getting messages from AWS- Starting on April 01, 2021, AWS Lambda will no longer support the botocore.requests library in Python runtimes [1][2]. If you are using the cfn-response module for a Lambda-backed custom resource, you must modify your Lambda function resource's Code or Runtime property and update your stack(s) in order to get the latest version of the cfn-response module which removed its dependency on botocore.requests. If you do not update your Lambda function by April 01, 2021, you will no longer be able to create, update or delete your custom resource
7 replies
What's the plan for supporting c7nmailer python lambda after April 1st
Kristina Trump
@KristinaTrump_twitter
@kapilt can mugc.py be used to clean old Custodian Lambda versions?
2 replies
வேள்பாரி
@velpaari13_twitter
@kapilt while execution below policy with admin access in lambda
image.png getting below error
image.png unauthorizedoperation error.plz help to fix
3 replies
mach1na
@mach1na
Hey there folks. Hoping someone can lend a hand. Trying to setup garbage collection via mugc.py. I can run the script standalone but when trying to run via c7n-org we get a generic error: c7n_org:INFO error running script on account:account_name region:eu-west-1 script:./mugc.py -c policy.yml --prefix c7n --region eu-west-1 --dryrun``
4 replies
Mostafa Hadi
@ItsReallyHadi

Hi all,
I want to write a policy to catch all IAM-Roles with wide-open actions, such as *, s3:*, sqs:*, ... . which an example can be seen bellow.

      Policies:
        - PolicyName: default
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - sns:*
                  - sdb:*
                Resource:
                  - '*'

I have prepared following policy but seems like it has two issues:
1) does not accept * and return error
2) does not act as OR and it acts as AND

policies:
  - name: iam-role
    resource: iam-role
    filters:
      - type: check-permissions
        match: allowed
        actions:
#          - *
          - iam:*
          - s3:*
          - sdb:*
          - sns:*
          - sqs:*
          - rds:*

Is there any way that I make this work, with a regex pattern or any other way?

3 replies
chennarao
@uvemuch_gitlab
@here Anyone successfully implemented EC2-AMI usage check?
I've approved the list in one master AWS account and wanted to check if AMI usage is from that list
5 replies
jfricioni
@jfricioni
Hello, I'm running into an issue where my tagging compliance policy filter doesnt like the first two lines of the following snippet. I've tried removing the -and and trying different ways but it still will not catch it. I basically want to do if the first two lines are missing and any of the *tag-compliance-filters are missing then send email to Owner. What am I missing?
 filters:
      - and:
        - "tag:aws:autoscaling:groupName": absent
        - "tag:biorad:c7n:tag-compliance": absent
        - or: *tag-compliance-filters
7 replies
chennarao
@uvemuch_gitlab

here is my json file from s3:

{
    "us-east-1": [
        "ami-01",
        "ami-02"
    ],
    "us-east-2": [
        "ami-002",
        "ami-003",
        "ami-004"
    ]
}

c7n policy i'm using to compare values:

policies:
  - name: ec2-invalid-ami
    description: un approved ami
    comment: high
    resource: ec2
    filters:
      # - "State.Name": running
      - type: value
        key: ImageId
        op: not-in
        value_from:
          url: s3://bucketname/approved-amis/region-list.json
          expr: [0]."us-east-1"[]

I'm getting error Can someone correct this for me

2 replies
Michael Nguyen
@micnguyen266

Hello,
I'm trying to run Custodian to filter s3 buckets that are not following tag compliance.
The problem is that the email template is looking for the Name tag of the bucket, but not all the buckets have the Name tag so some rows in the email report shows a blank name.
My question is how can I grab the s3 bucket names the way how it's showing in the AWS console?

S3 policy

policies:  
  - name: s3-tag-compliance-notify-only
    resource: s3
    description: |
      Scan s3 buckets that do not meet tag compliance.
    filters:
      - type: value
        key: "State.Name"
        op: ni
        value: ['terminated']
      - or: *tag-compliance-filters
    actions:
      - type: tag
        tag: TagCompliant
        value: "no"
      - type: notify
        template: general_template.html
        priority_header: 2
        subject: "Enterprise Tagging - {{ policy['resource'] }} Compliance Report! [{{ account }} {{ account_id }} - {{ region }}]"
        violation_desc: "Your s3 bucket is not tag compliant. Please see Cheat Sheet Link below and fix your tags."
        action_desc: "s3 bucket out of compliance. Informational only. This report policy does not mark, stop or delete resources."
        to:
#          - resource-owner
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/01234567890/cloud-custodian-mailer

Email Template

    {% elif policy['resource'] == "s3" %}
      {% if resources[0]['GlobalPermissions'] is defined %}
        {% set columnNames = ['Name','GlobalPermissions'] %}
      {% elif 'tag' in policy['name'] and 'unmark' not in policy['name'] %}
        {% set columnNames = ['Name','Contact','Environment','Billing','Priority','BusinessUnit','Group','Owner','Services','Deploy','Invalid / Missing Tags'] %}
      {% else %}
        {% set columnNames = ['Name','Contact','Environment','Billing','Priority','BusinessUnit','Group','Owner','Services','Deploy'] %}
      {% endif %}
      {{ createTable(columnNames, resources, '80') }}

Running version 0.9.4.0

2 replies
colereynolds
@colereynolds
Good morning folks - I'm looking for some help or guidance filtering on GCP resource timestamps down to the hour. I see some AWS resources have a filter type like instance-age that have an hour option to allow that, but is there a way to accomplish the same with a generic filter value type like age? It appears the default filter behavior for the age value type is by days.
4 replies
mohinder6
@mohinder6
Hi,
We are using cloud-custodian to do a scheduled EC2 instance stop at 6PM EST everyday. We started observing an error: ResourceCountExceeded in lambda errors. I believe its because our account's instance count has grown and the list of EC2 instances is too big for the API call? Has anyone encountered this situation?
2 replies
ennkayhub
@ennkayhub
I am looking to filter S3 buckets that don't have 'Versioning' enabled. What should be the filter?
2 replies
Jose Manuel Holgueras Monedero
@josemanuelholgueras
Hello to all
I have been reading the documentation for a few days and I do not see how to delete or disable the resources created by c7n? is there any way to delete policies and associated resources for example in AWS?
2 replies
jmahowald-slalom
@jmahowald-slalom
I've got something weird with a basic policy with dynamodb. I've used the same filters on most all other resources in config-rule mode without any problems, and when I run my policy locally, or as a periodic lambda everything is hunky dory. But when I switch to config for it I get [ERROR] TypeError: unsupported operand type(s) for /: 'datetime.datetime' and 'float' Traceback (most recent call last): File "/var/task/custodian_policy.py", line 4, in run return handler.dispatch_event(event, context) File "/var/task/c7n/handler.py", line 165, in dispatch_event p.push(event, context) File "/var/task/c7n/policy.py", line 1140, in push return mode.run(event, lambda_ctx) File "/var/task/c7n/policy.py", line 853, in run resources = super(ConfigRuleMode, self).run(event, lambda_context) File "/var/task/c7n/policy.py", line 437, in run resources = self.resolve_resources(event) File "/var/task/c7n/policy.py", line 835, in resolve_resources return [source.load_resource(self.cfg_event['configurationItem'])] File "/var/task/c7n/resources/dynamodb.py", line 23, in load_resource resource['CreationDateTime'] = datetime.fromtimestamp(resource['CreationDateTime'] / 1000.0) [ERROR] TypeError: unsupported operand type(s) for /: 'datetime.datetime' and 'float' Traceback (most recent call last): File "/var/task/custodian_policy.py", line 4, in run return handler.dispatch_event(event, context) File "/var/task/c7n/handler.py", line 165, in dispatch_event p.push(event, context) File "/var/task/c7n/policy.py", line 1140, in push return mode.run(event, lambda_ctx) File "/var/task/c7n/policy.py", line 853, in run resources = super(ConfigRuleMode, self).run(event, lambda_context) File "/var/task/c7n/policy.py", line 437, in run resources = self.resolve_resources(event) File "/var/task/c7n/policy.py", line 835, in resolve_resources return [source.load_resource(self.cfg_event['configurationItem'])] File "/var/task/c7n/resources/dynamodb.py", line 23, in load_resource resource['CreationDateTime'] = datetime.fromtimestamp(resource['CreationDateTime'] / 1000.0)
jmahowald-slalom
@jmahowald-slalom
obviously the way that it loads up the metadata when running in a local mode vs. running in config is slightly different, but I've certainly never seen it manifest itself before.
jmahowald-slalom
@jmahowald-slalom
ugh. Looking at the code that is crapping out, it’s not common, and the pr that added it has the great phrase “This is definitely a hack” https://github.com/cloud-custodian/cloud-custodian/pull/3516#issue-251551314. Going to make an isolated policy that is easier to share and reproduce to create an issue first.
jmahowald-slalom
@jmahowald-slalom
Filed an issue cloud-custodian/cloud-custodian#6470 . BTW, I filed an issue yesterday and it got fixed four hours ago. That's certainly a record for me on any non internal project. Wow
Kapil Thangavelu
@kapilt
@jmahowald-slalom maybe you can contribute on the next one :-)
jmahowald-slalom
@jmahowald-slalom
You're just too quick. I was going to add some debugging locally to see what the values were at the time, and you've already got a PR up for it.
1 reply
Deborshi Choudhury
@choudhde
Hello everyone, been trying to solve a problem with c7n, but the actions for this resource type seems to be limited, or I could be missing something. I have a route 53 hosted zone and that hosted zone should only have NS, SOA & TXT type record for an"acm" use case, any other record type like CNAME, A or Alias should be deleted upon creation. Is there any support for "ChangeResourceRecordsSets" to perform "DELETE" like action?
DigeratiDad
@digeratidad
Hello, when I’m using c7n-mailer with a custom template how do I include the template?
2 replies
Ramachandran Seshadri
@rams3sh_gitlab

Hey All, I was looking out if there would be a way to identify services that are unused for a period of n days through usage filter for iam entities (group / role) and if found unused beyond n days, add a deny statement to the existing policy / new policy . I am looking out for this as a method for enforcing least privilege model in AWS something that repokid from netflix does, but through c7n.
I have following two issues :-

i. I am able to do a filter based on usage , but I was not able to figure out the services that were unused dynamically which could be passed to an action set to deny those alone.
ii. I could'nt find an action in role and user which could create a policy dynamically with passed unused services from filter and attach it to the user / role.

It would be really helpful if anyone can provide a solution for this (if it exists), if not any hacky way through c7n is also appreciated !!

Ananth Balasubramanian
@linuxananth1976
Hey Guys, I have a question not sure whether it answered or not yet. If answered please paste the reference link with this reply.
My requirement is as follows:
I need to validate/evaluate the existing standard custom tag having service keys and values from my array list of values.
If exists we can ignore the resource of notify/disable/deletion else I need to take action.
for eg:
I had list of array valid service tags value as
["abc", "xyz", "def", "jkl" ]
if my resource had prefix with above valid tags like below
service: abc-ui
As per the above example, the resource should be ignored.
service: dummy-pack
Action should be taken for the above one.
It would be very much helpful and thanks if I get the solution for the above.
5 replies
Ricardo
@ricardoandre_gitlab

Hey, guys. Maybe this is a trivial question, but believe me, I've been reading documentation and I still can't find it.

I am using a simple policy to check my AWS iam users keys expiration, and I want to send an email to the specific user without Tags. Previously, by using contact_tags: ['myTag'] in the mailer worked as expected. But now I changed all the names of my iam users to use email ids . e.g before it was sam, now it's sam@mail.com.
How can I reference the username itself when using the notify action?

  - name: iam-user-password-105days-older
    resource: iam-user
    ...truncated...
    actions:
      - type: notify
        subject: "[custodian] IAM - Password Not Changed"
       ...truncated...
        to:
          - **TO_WHO? That's my big question :( I don't want to use tags, but I want to access the actual username who already has the email on it**
        transport:
         type: sqs
          ...truncated...
Ananth Balasubramanian
@linuxananth1976

Hey Guys, I have a question not sure whether it answered or not yet. If answered please paste the reference link with this reply.
My requirement is as follows:
I need to validate/evaluate the existing standard custom tag having service keys and values from my array list of values.
If exists we can ignore the resource of notify/disable/deletion else I need to take action.
for eg:
I had list of array valid service tags value as
["abc", "xyz", "def", "jkl" ]
if my resource had prefix with above valid tags like below
service: abc-ui
As per the above example, the resource should be ignored.
service: dummy-pack
Action should be taken for the above one.
It would be very much helpful and thanks if I get the solution for the above.

Thanks I got solution for the above and have below too..
One more query will it possible to declare as variable or make it as argument file the values?

4 replies
Abel
@Abikjose
Hello Guys, I have a question. How do I know the number of API calls cloud custodian make to the AWS account? (To be safe that it doesn't hit the throttle )
1 reply
JT
@superman3

i am trying to filter my KMS keys
i have 1 aws managed KMS key (can not be tagged) and 1 customer managed key called testkey1 (that has been tagged properly)

Im using the below filter but testkey1 keeps being picked up by the filter. any ideas what im doing wrong please ?

filters:
  - or:
    - "tag:costcentre": absent
    - "tag:live": absent
    - not:
      - type: value
        key: "tag:live"
        value_type: normalize
        value: ['yes', 'no']
        op: in
  - and:      
    - not:
      - type: value
        key: "AliasName"
        value_type: normalize
        value: 'alias/aws/'
        op: contains
4 replies
sarojatester
@sarojatester
Hi, I am trying to create a policy with cloudtrail mode for CreatePolicy event (iam-policy).
6 replies
Michael Davis
@MichaelDavisTSN
Anybody know which age filter works with security groups?
Update: I think this will work:
- type: value
  value_type: age
  key: CreationDate
  op: ge
  value: 30
2 replies
jfricioni
@jfricioni
I'm having an issue with my account name not showing up when running a tagging compliance policy. I'm able to get the account name to show up when I run the lambda in the account where the SQS/mailer live but the other account I am using to test does not show up. I am able to pull account ID (removed from picture) and the region pulls fine as well, here is also the snippet of code for the notification.
- type: notify
  template: default.html
  priority_header: 2
  subject: "EC2 - Instance Tagging Out of Compliance!!! - {{ account }} - {{ account_id }} - {{ region }}"
  violation_desc: |
    Your EC2 instance is out of compliance and is missing mandatory tag(s)
  action_desc: |
    Include all required tags on the EC2 instance
     to stop receiving daily notifications. You can find the tagging wiki here - 
  to:
    - resource-owner
2 replies
image.png
cleo2525
@cleo2525
Hello, is there a away to set a filter for s3 buckets that don't have a global-grant? Specifically, I'm looking to filter for s3 buckets that have certain tags and also don't have public read ACLs [READ, READ_ACP] for grantees AllUsers and AuthenticatedUsers. Thank you for any help.
3 replies
Ananth Balasubramanian
@linuxananth1976

looking for some idea, I have two different templates, one templates is used to open a Jira ticket and one is used to send email to users, how I can accomplish it in one action, I was thinking this would work, but no, is creating a new policy the only option:

actions:
  - type: notify
    subject: "S3 bucket has not been Tagged with Owner tag"
    template: jira.html
    to:
      - jira@tomarv2.com
    transport:
      type: sns
      topic: "arn:aws:sns:us-east-2:123456789:custodian-mailer-nonprod-us-notify"
  - type: notify
    subject: "S3 bucket has not been Tagged with Owner tag"
    template: default.html
    to:
      - varun@tomarv2.com
    transport:
      type: sns
      topic: "arn:aws:sns:us-east-2:123456789:custodian-mailer-nonprod-us-notify"

Hello, Can I please have a document or anybody configured for Jira ticket creation as a notification with Jira token API? I got this quote but couldn't find the template or how to use it. Please let me know if there is any link that exists for my requirement. Thanks in advance.

2 replies
DigeratiDad
@digeratidad
Hello everyone! I’m curious how everyone is managing their CC policies and lambdas across 100s of accounts. At my org we have 100s of AWS accounts and 50+ CC policies across all those accounts and regions. I’ve started getting some push back on the number of lambdas deployed. Anyone else having those issues and if so how do you manage it?
2 replies
DigeratiDad
@digeratidad
Does anyone have an example of using webhook actions with slack? I basically want the end-user to hit “proceed” before an action takes place.
pendyalal
@pendyalal
@kapilt , In custodian schema, I see aws.transit-attachment.filters.event, but when I ran the custodian policy in the cloudtrial mode, it says resource:aws.transit-attachment does not support cloudtrail mode policies.
4 replies
 policies:
  - name: transitgateway-attachments
    resource: aws.transit-attachment
    description: Notify when some attaches transitgateway.
    mode:
      type: cloudtrail
      role: CustodianLambdaExecutionRole
      memory: 256
      timeout: 180
      runtime: python3.8
      events:
        - source: ec2.amazonaws.com
          event: CreateTransitGatewayVpcAttachment
          ids: requestParameters.CreateTransitGatewayVpcAttachmentRequest.TransitGatewayId 
    filters:
      - or:
        - not:
          - type: event
            key: requestParameters.CreateTransitGatewayVpcAttachmentRequest.TagSpecifications.Tag[].Contact
            value: 'test'
            op: contains
          - type: event
            key: requestParameters.CreateTransitGatewayVpcAttachmentRequest.TagSpecifications.Tag[].Project
            op: ne
            value: Infrastructure
            value_type: normalize
    actions:
      - type: notify
aakshaik2
@aakifshaikh
Anyone please share your custodian policy to identify S3 bucket is exposed to the world/public (looking at both bucket policy and ACL)
Ananth Balasubramanian
@linuxananth1976

Hey, there is one requirement from my side. i.e., I need to have both in a single filter
value: regex
value_from: s3 URL

for eg:-
I have an array of standard tags in my file.json as below

    "service": ["abc", "xyz", "lmn"]
}

filters were like below:
- type: value key: "tag:service" op: in value_from: url: s3://bucket_name/key/file.json

value_regex:
- type: value key: "tag:service" op: regex value: '^(abc|xyz|lmn)([\-].*)?$'

will it be possible to have both for tag validation?
Please let me know if there is any solution for this one. Thanks in advance.

Ramachandran Seshadri
@rams3sh_gitlab
Created a small playground tool for creating and testing c7n-mailer templates for my use case. Thought of sharing it here in case any one else has same use case as mine. Link : https://github.com/rams3sh/c7n-mailer_playground
1 reply
Ricardo
@ricardoandre_gitlab

Hey, guys. Maybe this is a trivial question, but believe me, I've been reading documentation and I still can't find it.

I am using a simple policy to check my AWS iam users keys expiration, and I want to send an email to the specific user without Tags. Previously, by using contact_tags: ['myTag'] in the mailer worked as expected. But now I changed all the names of my iam users to use email ids . e.g before it was sam, now it's sam@mail.com.
How can I reference the username itself when using the notify action?

    resource: iam-user
      - type: notify
        to:
          - **TO_WHO? That's my big question :( I don't want to use tags, but I want to access the actual username who already has the email on it**
Ryan Ash
@ryanash999

Good morning,
Is there any way to accomplish the following in a single policy outside of a custom lambda/etc?

rest-stage - Does not have associated webACL
AND
rest-api - Is not PRIVATE

mahisid
@mahisid
Hi Need help with a custodian policy
I am building a policy to detect root login . I want to send the event which triggered the check to be forwarded to slack. I tried {{ event }} for violation_desc but i am not getting the intended output on slack.
policies:

  - name: root-user-login-detected
    resource: account
    description: |
      Notifies Security and Cloud Admins teams on any AWS root user console logins
    mode:
       role: custodian-role
       type: cloudtrail
       events:
          - ConsoleLogin
    filters:
       - type: event
         key: "detail.userIdentity.type"
         value_type: swap
         op: in
         value: Root
    actions:
      - type: notify
        slack_msg_color: danger
        violation_desc: {{ event }}
        action_desc: Root user login detected! Take action if this is not intended login.
        to:
          - xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        transport:
          type: sqs
          queue: https://sqs.us-east-2.amazonaws.com/xxxxxxxxxxxxxxxxxxxxxxxx
          region: us-east-2
2 replies
Ryan Ash
@ryanash999
I heard Kapil in a podcast mention current/future features to allow Custodian to behave like Terraform Sentinel. This would be allowing us to move compliance left by looking at cloudformation/terraform. Does anyone know if this is future or if anything exists today.
4 replies