Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Matt Benson
    @mbenson
    but how is the region used outside AWS? Please forgive my newness to the platform.
    Agim Emruli
    @aemruli
    well except for a few exception every service is reachable outside AWS through an service endpoint
    for example rds.ap-southeast-1.amazonaws.com
    Matt Benson
    @mbenson
    sure...
    Agim Emruli
    @aemruli
    so you can run your application on your laptop and use the AWS services
    you can even have stack in cloud formation and use those services
    the only thing here is that we need to know the region
    so that the client can make the right call
    the exception is onyl elasticache
    that service is only reachable inside the same security group or vpc
    therefore its quite common to run your app also outside AWS but still consume AWS services
    Matt Benson
    @mbenson
    okay; I won't pretend to fully understand it, but if there is some code that specifically uses the region knowledge to contact other services then that is good enough
    Agim Emruli
    @aemruli
    yes
    Matt Benson
    @mbenson
    I suppose that's why it's pulled from what appear to be various instances of the AmazonWebserviceClientFactoryBean then
    Agim Emruli
    @aemruli
    yeah every service client have a region, and the factory bean manages that
    Matt Benson
    @mbenson
    thanks for making it clear. In this case I have an app that I'm running on AWS but this particular one isn't actually consuming AWS services... I just want the AmazonEc2InstanceDataPropertySource autoconfigured. But I guess I'll just have to deal with it. :P
    Matt Benson
    @mbenson
    argh; I really wanted to use that with YAML subdocument profile negation but that won't be released until Spring Boot 1.4.0 :(
    Ryan Gardner
    @ryangardner
    FYI I just submitted a pull request to let people use the DefaultAWSCredentialsProvider when running with the spring boot starter - spring-cloud/spring-cloud-aws#170
    Ryan Gardner
    @ryangardner
    The docs currently state that the DefaultAWSCredentialsProvider is used for services - which seems only be true if you aren’t using the spring boot starters. The ContextCredentialsAutoConfiguration always creates a custom credentials provider chain that gets used for the amazon service calls - over time things have been added (like profile support) to that chain, but it’s still doesn’t support all the options of the default one
    Ryan Gardner
    @ryangardner
    has anyone here had a chance to look at spring-cloud/spring-cloud-aws#170 yet?
    Ryan Gardner
    @ryangardner
    This message was deleted
    Felipe Adorno
    @FelipeAdorno
    Hi guys I need use spring-cloud-stream-sink-s3 but I have a lot of problems, I have need sink local file to s3 bucket any help?
    Danilo Gomes
    @danilocgomes_twitter
    Hey, guys, how are you? Can I ask you something about spring cloud aws messaging on this room?
    Am I in the right place? :D
    Danilo Gomes
    @danilocgomes_twitter

    I want know if I can control the number of consumers when using Spring Cloud AWS Messaging and @MessageMapping annotation.
    If I understood, I should create my own AsyncTaskExecutor. I really don't know if this is the right way.

    Let me know if you could help me, guys. Thanks in advance.

    Kyle Anderson
    @andersonkyle
    @danilocgomes_twitter I've got a similar question. The default polling settings for SQS by way of the @SqsListener annotation appear to be extremely aggressive in their frequency. SQS charges per request and the Spring Cloud AWS defaults seem to be around 16 requests per second. I would like to use long pollling and adjust the interval. If anyone can point me to where I can set that I'd appreciate it. Obviously this can easily be done by using the Amazon SDK directly but I'm hoping there's a way to do this within Spring.
    Danilo Gomes
    @danilocgomes_twitter

    @kanderson450 Thanks for your attention. Actually I have the same need but in order to increase or decrease rps according to queue (it'll be decided by profile, each one with its configurations). I would like to hardly increase the number of consumers to deal with 150 rps.

    If I'm right, I should pass other instance of AsyncTaskExecutor to SimpleMessageListenerContainer
    https://github.com/spring-cloud/spring-cloud-aws/blob/master/spring-cloud-aws-messaging/src/main/java/org/springframework/cloud/aws/messaging/listener/SimpleMessageListenerContainer.java

    But, is this the right way?

    Spring Cloud AWS Buildmaster
    @spring-cloud-aws-buildmaster
    You should use the wait-time-out property on the listener container <aws-messaging:annotation-driven-queue-listener wait-time-out="60" />
    This defines the time the poller has to wait when no messages are in the queue (long polling), this is actually a feature provided by the AmazonSQS client.
    If you use Java Config you can set that property using the SimpleMessageListenerContainerFactory
    Danilo Gomes
    @danilocgomes_twitter

    Thanks, guys. But that isn't my real question. Let's suppose that my application is consuming 50 records per second. If I want scale up to 150 rps, the only option that I have is scaling horizontally or can I increase (and control) my number of consumers?

    protected AsyncTaskExecutor createDefaultTaskExecutor() {
            String beanName = getBeanName();
            ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
            threadPoolTaskExecutor.setThreadNamePrefix(beanName != null ? beanName + "-" : DEFAULT_THREAD_NAME_PREFIX);
            int spinningThreads = this.getRegisteredQueues().size();
    
            if (spinningThreads > 0) {
                threadPoolTaskExecutor.setCorePoolSize(spinningThreads * DEFAULT_WORKER_THREADS);
    
                int maxNumberOfMessagePerBatch = getMaxNumberOfMessages() != null ? getMaxNumberOfMessages() : DEFAULT_WORKER_THREADS;
                threadPoolTaskExecutor.setMaxPoolSize(spinningThreads * maxNumberOfMessagePerBatch);
            }
    
            // No use of a thread pool executor queue to avoid retaining message to long in memory
            threadPoolTaskExecutor.setQueueCapacity(0);
            threadPoolTaskExecutor.afterPropertiesSet();
    
            return threadPoolTaskExecutor;
    
        }

    Can I do it just overwriting AsyncTaskExecutor configuration?

    That is a citation of SimpleMessageListenerContainer.java
    Danilo Gomes
    @danilocgomes_twitter
    Can you see what I'm facing? Does it make sense?
    Ryan Gardner
    @ryangardner
    @danilocgomes_twitter I ran into the same problem about not being able to easily ramp up the number of concurrent executors for the message listener so I ended up switching out my message polling and using camel instead
    for the others asking… if you want to do something with adding long polling, here’s what I put in my app’s config:
        public QueueBufferConfig queueBufferConfig(@Value("${QUEUE_CONSUMER_COUNT}") int consumerCount) {
            QueueBufferConfig config = new QueueBufferConfig();
            // use long polling to avoid going nuts with empty fetches
            config.setLongPoll(true);
            config.setLongPollWaitTimeoutSeconds(20);
            // more in-flight batches can help maximize throughput per box
            config.setMaxInflightReceiveBatches(consumerCount * 2);
            config.setMaxDoneReceiveBatches(consumerCount+1);
            return config;
        }
    
    
    
    
        @Lazy
        @Bean(destroyMethod = "shutdown")
        public AmazonSQSAsync amazonSQS( QueueBufferConfig queueBufferConfig, RegionProvider regionProvider, AWSCredentialsProvider awsCredentialsProvider) {
            AmazonSQSAsyncClient amazonSQSAsyncClient;
            amazonSQSAsyncClient = new AmazonSQSAsyncClient(awsCredentialsProvider);
    
            if (regionProvider != null) {
                amazonSQSAsyncClient.setRegion(regionProvider.getRegion());
            }
    
            return new AmazonSQSBufferedAsyncClient(amazonSQSAsyncClient, queueBufferConfig);
        }
    Simon
    @sbuettner
    Is there a possibility to disable the ec2 metadata lookup during startup thats happening in the OnAwsCloudEnvironmentCondition?
    Kyle Anderson
    @andersonkyle
    @sbuettner cloud.aws.stack.auto=false
    Danilo Gomes
    @danilocgomes_twitter

    Hey @ryangardner! I'm so sorry for delay. I was in a really really critical project and I need to change my focus.
    And, I noted that this chat isn't very active.

    So, thank you so much for reply me. Seriously you need change your solution for Camel?
    Is there no native solution for spring cloud aws?

    I'll try overwrite AsyncTaskExecutor mentioned previously and I'll get back with the results.

    Danilo Gomes
    @danilocgomes_twitter

    Hey @ryangardner , I should confess that I've also used camel :D

    Thanks for your suggestion!

    Andy Czerwinski
    @czerwina
    How do people build and test locally their microservices if you use the spring-cloud-aws annotations in their code?
    Kyle Anderson
    @andersonkyle
    @czerwina I don't have an elegant answer, other than using Profile specific credentials. For example, for local development, I have a set of credentials on my local machine that are tied to a completely separate account from production. Same thing for staging. AWS Account segregation is what I do.
    @czerwina This can introduce config drift over time, but that's the best I've got.
    Andy Czerwinski
    @czerwina
    @kanderson450 so you need to deploy to aws for each developer environment? Don't think our infrastructure dudes would appreciate paying for the hardware even if cheap.
    We use redis, oracle xd, rabbitmq locally which translate to elasticated, rds and sqs
    *Elasticache
    Kyle Anderson
    @andersonkyle
    @czerwina I'd rather take the hit then use completely different components. RabbitMQ is nothing like SQS (even with an abstraction layer), and there's a lot of configuration that is specific to each. That's more config drift than separate AWS environments. And I'm not suggesting you need to deploy to AWS for local testing. You can make use of SQS, S3 and other services while running your app on your laptop. This isn't perfect either but SQS messages cost a miniscule amount.
    Andy Czerwinski
    @czerwina
    @kanderson450 didn't think of just exposing the services to the devs. U think via api gateway or vpn to the vpc?
    Kyle Anderson
    @andersonkyle
    @czerwina Depends on the service as they are not all available as private VPC endpoints. For example, SQS is not but S3 is. Your custom application services should be able to run locally all at once using docker-compose.