aws/aws-sdk-core-ruby

Please join http://gitter.im/aws/aws-sdk-ruby instead.

Christian Bay

@valleybay

I am trying to use this gem with Rails, but where do I actually configure the require?

do I just include it in config/initializers/aws.rb?

Gerardo Santana

@santana

is there a reason for allowing each Aws::SharedCredenials instance to have a different path to the credentials file?

if there’s none, we could parse the ini file once (instead of per instance) and provide a list of profiles from a class method, available to all instances

Trevor Rowe

@trevorrowe

@santana The purpose of supporting a path is to allow the instance profile credentials to be loaded in environments where a/the home directory is not available for the current user, but the file is available on disk.

Wouldn’t the following work?

Aws::InstanceProfileCredentials.new.each_profile do |profile_name, credentials|
  # yielded creds are Aws::Credentials
end

Ryan Romanchuk

@rromanchuk

Really need this functionality in V2 http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/SNS/Message.html

Crypto should be the first thing to get ported

Trevor Rowe

@trevorrowe

@rromanchuk Are you referencing the #authentic? helper method on AWS::SNS::Message?

Ryan Romanchuk

@rromanchuk

@trevorrowe yeah sorry, extremely timesaving method when dealing with https SNS delivery

Trevor Rowe

@trevorrowe

@rromanchuk Would you like to take a look at a pull-request I made to add sns message authentication?

aws/aws-sdk-ruby#741

Ryan Romanchuk

@rromanchuk

@trevorrowe amazing!

kevin-staiger

@kevin-staiger

I have noticed that sqs .poll and receive_message both automatically delete the message after receiving, is there any way to not delete it?

Erik Straub

@brkattk

Hey everyone. I’m attempting to get a stubbed Aws::AssumeRoleCredentials instance with the following

        aws_sts_client = Aws::STS::Client.new( credentials: credentials,
                                               region: DEFAULT_AWS_REGION)

        Aws::AssumeRoleCredentials.new( client: aws_sts_client,
                                        role_arn: profile[:role_arn],
                                        role_session_name: "#{profile[:name]}-session",
                                        duration_seconds: DEFAULT_AWS_TOKEN_DURATION)

credentials is either an instance of Aws::SharedCredentials or Aws::Credentials

My issue is I keep seeing this error:

NoMethodError:
       undefined method `access_key_id' for nil:NilClass
# /Users/brkattk/.rbenv/versions/2.2.2/gemsets/project-name/gems/aws-sdk-core-2.0.37/lib/aws-sdk-core/assume_role_credentials.rb:39:in `refresh'
# /Users/brkattk/.rbenv/versions/2.2.2/gemsets/project-name/gems/aws-sdk-core-2.0.37/lib/aws-sdk-core/refreshing_credentials.rb:20:in `initialize'
# /Users/brkattk/.rbenv/versions/2.2.2/gemsets/project-name/gems/aws-sdk-core-2.0.37/lib/aws-sdk-core/assume_role_credentials.rb:29:in `initialize'
# ./lib/project-name/configuration/aws_parser.rb:106:in `new'
# ./lib/project-name/configuration/aws_parser.rb:106:in `assume_role_credentials'
# ./lib/project-name/configuration/aws_parser.rb:83:in `build_credentials'
# ./lib/project-name/configuration/aws_parser.rb:63:in `block in validate_profiles'
# ./lib/project-name/configuration/aws_parser.rb:60:in `each'
# ./lib/project-name/configuration/aws_parser.rb:60:in `validate_profiles'
# ./lib/project-name/configuration/aws_parser.rb:49:in `validate!'
# ./lib/project-name/configuration/aws_parser.rb:29:in `parse!'
# ./spec/lib/project-name/configuration/aws_parser_spec.rb:186:in `block (9 levels) in <top (required)>'
# ./spec/lib/project-name/configuration/aws_parser_spec.rb:189:in `block (9 levels) in <top (required)>'

I’m assuming the general Aws.config[:stub_responses] = true does not construct the actual responses you need, just simply stops the HTTP call from happening, right?

Trevor Rowe

@trevorrowe

Thats correct. You can specify the data a stub should return, but by default it returns sample empty responses.

Erik Straub

@brkattk

ok. I guess I might as well use webmock and do it myself then. do you happen to know where I might find an example response? edit: http://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html

Trevor Rowe

@trevorrowe

If you can send an actual live request, then the SDK can provide you the http response. For example:

resp = aws_sts_client.assume_role(
  role_arn: profile[:role_arn],
  role_session_name: "#{profile[:name]}-session",
  duration_seconds: DEFAULT_AWS_TOKEN_DURATION)

resp.context.http_response.status_code
resp.context.http_response.headers
resp.context.http_response.body_contents

Erik Straub

@brkattk

I have an app that uses multiple different AWS credentials to do various things and eventually upload a file to S3. I’ve got an array of environments like so

[
  {
    :name=>"default”,
    :bucket_name=>”my.bucket”,
    :bucket_region=>”us-east-1”,
     :credentials=>#<Aws::SharedCredentials profile_name="default" path="/Users/brkattk/.aws/credentials”>
  },
  {
    :name=>”secondary",
    :bucket_name=>”my.bucket”,
    :bucket_region=>”us-east-1”,
    :credentials=>#<Aws::SharedCredentials profile_name=“secondary" path="/Users/brkattk/.aws/credentials”>
  }
]

I’ve got a Report class that will upload a CSV to an environment’s designated bucket:

class Report
  att_reader :csv, :environment
  def initialize(environment, csv)
    @environment = environment
    @csv = csv
  end

  def upload!
    s3_bucket.put_object(
          body: csv.to_csv,
          key: csv.to_filename,
          content_type: 'text/csv')
  end

  private

    def s3_resource
      ::Aws::S3::Resource.new(client: s3_client)
    end

    def s3_bucket
      s3_resource.bucket environment[:bucket_name]
    end

    def s3_client
      ::Aws::S3::Client.new(credentials: environment[:credentials], region: environment[:bucket_region])
    end
end

I’m looping each environment and producing a report, however, it is uploading both files to the last environment’s bucket.

Am I going insane?

Erik Straub

@brkattk

is Aws::S3::Client meant to be a singleton?

Trevor Rowe

@trevorrowe

No, you can construct as many instances of Aws::S3::Client as you require.

This should not affect your issue, but you can reduce some of that code by not constructing the Resource and bucket:

S3::Object.new(bucket_name, key).put(body:’…’, content_type: ‘…')

Also, your issue is that it is “uploading both files to the last environment’s bucket” — in the example you gave above, they list the same bucket name.

Erik Straub

@brkattk

Aws::SharedCredentials.new profile_name: ‘blah’

does not do what it advertises @trevorrowe

regardless of bucket name, it should tell which account by the credentials passed to Aws::S3::Client, right?

my issue is that the default doesn’t even have a bucket named my.bucket

I’ve got a test for this:

I have 2 profiles in my ~/.aws/credentials that are completely separate AWS accounts (one is my personal, one for my company)

I created an ran this script. I do not have a bucket named reports in my personal AWS account, but this puts the first object in my company account’s report bucket

require 'rubygems'
require 'aws-sdk'

creds1 = Aws::SharedCredentials.new(profile_name: 'erik')
s3_client1 = Aws::S3::Client.new(region: 'us-east-1', credentials: creds1)
s3_resource1 = Aws::S3::Resource.new(client: s3_client1)
s3_bucket1 = s3_resource1.bucket('reports')
object = s3_bucket1.objects.first
p object

Trevor Rowe

@trevorrowe

What is Aws::SharedCredentials.new doing if not what you expect? The ability to write to a bucket is not strictly limited to a single account. That is the default, only the bucket owner may write, but you can open the public for public writes, or even white listed users and accounts.

Erik Straub

@brkattk

ugh.. hang on this might be my own stupidity

Trevor Rowe

@trevorrowe

When you construct a shared credentials object, inspect the #access_key_id and #secret_access_key. Are they for the correct profile?

Erik Straub

@brkattk

:cry: :gun: I definitely had a policy on that bucket that was open to any authenticated user

now I’m getting the appropriate AccessDenied exception

Peter Mounce

@petemounce

Hi @trevorrowe - could you give me a steer on how to create a waiter for a windows ec2 password being updated since <some date>?

The reason being is that I have ec2 instances where ec2config is configured to set a new administrator password on each launch. A reboot counts (apparently), and so i'm in a position where i'm baking an AMI, and need to orchestrate reboots, and therefore need to wait for the the password to both be available and different.

The EC2:GetPasswordData operation helpfully has a timestamp - so I think i just need to know how to create the waiter which takes a date time as a parameter to compare against

Trevor Rowe

@trevorrowe

Currently the client waiter interface only accepts inputs that are valid request parameters. That said, there is a custom waiter interface on the resource classes where you can pass a block and wait on it to return a truthy value.

instance = Aws::EC2::Resource.new.instances.first
old_timestamp = instance.password_data.timestamp
instance.reboot
instance.wait_until { |i| i.password_data.timestamp != old_timestamp }

Peter Mounce

@petemounce

@trevorrowe thanks; perfect!

one thing i did run into was that response in the block was nil on at least the first attempt, which seemed odd

i wanted to do

... do |attempts, response|
  puts "#{attempts}; timestamp this time: #{response.password_data.timestamp}"
end

but response was nil

Trevor Rowe

@trevorrowe

@petemounce Do you mean for the before_wait callback?

Peter Mounce

@petemounce

@trevorrowe yes

      proc = Proc.new do |attempts, response|
        logger.debug "Waiting for password data timestamp to be newer", {pwd_timestamp: instance.password_data.timestamp, newer_than: since}
      end
      instance.wait_until(max_attempts: 30, delay: 10, before_wait: proc) { |i| i.password_data.timestamp != since }

i is an Instance resource

Peter Mounce

@petemounce

@trevorrowe is there a way to globally change the global waiter? I was using the standard settings, and I'm getting RequestLimitExceeded exceptions, presumably because of all the Describe* requests.

to combat that, i've changed my waiters to look like, for example,

w.max_attempts = 15
w.interval = 0
w.before_attempt do |a, resp|
  pause_exponentially n
  logger.debug "Waiting for password...", {instance_id: id, attempt_count: a}
end

def pause_exponentially(n, seed=2, exp=1.4)
  sleep(seed * (exp ** n))
end

it would be really handy to be able to change that in just a single place (with the capability probably of still being able to configure the seed and exponent-even-if-that's-the-wrong-mathematical-term for each waiter)

Peter Mounce

@petemounce

actually, one thing i thought of - it would be nice to just tell the waiter definition the maximum time it should wait, rather than do maths to figure out what that will end up being (regardless of whether the strategy is linear or exponential)

Trevor Rowe

@trevorrowe

There is not an accessible way to change the default waiter value currently. Currently, the waiter provider is a private interface, and is subject to change. If you are comfortable poking into something that may change in the future, you could do the following:

EC2::Client.waiters.instance_variable_get("@waiters")[:instance_running][:delay] = 20

I’d be open to suggestions on how to make this more flexible, until then, I left the interfaces marked @api private.

Where communities thrive

People

Repo info

Activity