Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    Matt Benson
    @mbenson
    argh; I really wanted to use that with YAML subdocument profile negation but that won't be released until Spring Boot 1.4.0 :(
    Ryan Gardner
    @ryangardner
    FYI I just submitted a pull request to let people use the DefaultAWSCredentialsProvider when running with the spring boot starter - spring-cloud/spring-cloud-aws#170
    Ryan Gardner
    @ryangardner
    The docs currently state that the DefaultAWSCredentialsProvider is used for services - which seems only be true if you aren’t using the spring boot starters. The ContextCredentialsAutoConfiguration always creates a custom credentials provider chain that gets used for the amazon service calls - over time things have been added (like profile support) to that chain, but it’s still doesn’t support all the options of the default one
    Ryan Gardner
    @ryangardner
    has anyone here had a chance to look at spring-cloud/spring-cloud-aws#170 yet?
    Ryan Gardner
    @ryangardner
    This message was deleted
    Felipe Adorno
    @FelipeAdorno
    Hi guys I need use spring-cloud-stream-sink-s3 but I have a lot of problems, I have need sink local file to s3 bucket any help?
    Danilo Gomes
    @danilocgomes_twitter
    Hey, guys, how are you? Can I ask you something about spring cloud aws messaging on this room?
    Am I in the right place? :D
    Danilo Gomes
    @danilocgomes_twitter

    I want know if I can control the number of consumers when using Spring Cloud AWS Messaging and @MessageMapping annotation.
    If I understood, I should create my own AsyncTaskExecutor. I really don't know if this is the right way.

    Let me know if you could help me, guys. Thanks in advance.

    Kyle Anderson
    @andersonkyle
    @danilocgomes_twitter I've got a similar question. The default polling settings for SQS by way of the @SqsListener annotation appear to be extremely aggressive in their frequency. SQS charges per request and the Spring Cloud AWS defaults seem to be around 16 requests per second. I would like to use long pollling and adjust the interval. If anyone can point me to where I can set that I'd appreciate it. Obviously this can easily be done by using the Amazon SDK directly but I'm hoping there's a way to do this within Spring.
    Danilo Gomes
    @danilocgomes_twitter

    @kanderson450 Thanks for your attention. Actually I have the same need but in order to increase or decrease rps according to queue (it'll be decided by profile, each one with its configurations). I would like to hardly increase the number of consumers to deal with 150 rps.

    If I'm right, I should pass other instance of AsyncTaskExecutor to SimpleMessageListenerContainer
    https://github.com/spring-cloud/spring-cloud-aws/blob/master/spring-cloud-aws-messaging/src/main/java/org/springframework/cloud/aws/messaging/listener/SimpleMessageListenerContainer.java

    But, is this the right way?

    Spring Cloud AWS Buildmaster
    @spring-cloud-aws-buildmaster
    You should use the wait-time-out property on the listener container <aws-messaging:annotation-driven-queue-listener wait-time-out="60" />
    This defines the time the poller has to wait when no messages are in the queue (long polling), this is actually a feature provided by the AmazonSQS client.
    If you use Java Config you can set that property using the SimpleMessageListenerContainerFactory
    Danilo Gomes
    @danilocgomes_twitter

    Thanks, guys. But that isn't my real question. Let's suppose that my application is consuming 50 records per second. If I want scale up to 150 rps, the only option that I have is scaling horizontally or can I increase (and control) my number of consumers?

    protected AsyncTaskExecutor createDefaultTaskExecutor() {
            String beanName = getBeanName();
            ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
            threadPoolTaskExecutor.setThreadNamePrefix(beanName != null ? beanName + "-" : DEFAULT_THREAD_NAME_PREFIX);
            int spinningThreads = this.getRegisteredQueues().size();
    
            if (spinningThreads > 0) {
                threadPoolTaskExecutor.setCorePoolSize(spinningThreads * DEFAULT_WORKER_THREADS);
    
                int maxNumberOfMessagePerBatch = getMaxNumberOfMessages() != null ? getMaxNumberOfMessages() : DEFAULT_WORKER_THREADS;
                threadPoolTaskExecutor.setMaxPoolSize(spinningThreads * maxNumberOfMessagePerBatch);
            }
    
            // No use of a thread pool executor queue to avoid retaining message to long in memory
            threadPoolTaskExecutor.setQueueCapacity(0);
            threadPoolTaskExecutor.afterPropertiesSet();
    
            return threadPoolTaskExecutor;
    
        }

    Can I do it just overwriting AsyncTaskExecutor configuration?

    That is a citation of SimpleMessageListenerContainer.java
    Danilo Gomes
    @danilocgomes_twitter
    Can you see what I'm facing? Does it make sense?
    Ryan Gardner
    @ryangardner
    @danilocgomes_twitter I ran into the same problem about not being able to easily ramp up the number of concurrent executors for the message listener so I ended up switching out my message polling and using camel instead
    for the others asking… if you want to do something with adding long polling, here’s what I put in my app’s config:
        public QueueBufferConfig queueBufferConfig(@Value("${QUEUE_CONSUMER_COUNT}") int consumerCount) {
            QueueBufferConfig config = new QueueBufferConfig();
            // use long polling to avoid going nuts with empty fetches
            config.setLongPoll(true);
            config.setLongPollWaitTimeoutSeconds(20);
            // more in-flight batches can help maximize throughput per box
            config.setMaxInflightReceiveBatches(consumerCount * 2);
            config.setMaxDoneReceiveBatches(consumerCount+1);
            return config;
        }
    
    
    
    
        @Lazy
        @Bean(destroyMethod = "shutdown")
        public AmazonSQSAsync amazonSQS( QueueBufferConfig queueBufferConfig, RegionProvider regionProvider, AWSCredentialsProvider awsCredentialsProvider) {
            AmazonSQSAsyncClient amazonSQSAsyncClient;
            amazonSQSAsyncClient = new AmazonSQSAsyncClient(awsCredentialsProvider);
    
            if (regionProvider != null) {
                amazonSQSAsyncClient.setRegion(regionProvider.getRegion());
            }
    
            return new AmazonSQSBufferedAsyncClient(amazonSQSAsyncClient, queueBufferConfig);
        }
    Simon
    @sbuettner
    Is there a possibility to disable the ec2 metadata lookup during startup thats happening in the OnAwsCloudEnvironmentCondition?
    Kyle Anderson
    @andersonkyle
    @sbuettner cloud.aws.stack.auto=false
    Danilo Gomes
    @danilocgomes_twitter

    Hey @ryangardner! I'm so sorry for delay. I was in a really really critical project and I need to change my focus.
    And, I noted that this chat isn't very active.

    So, thank you so much for reply me. Seriously you need change your solution for Camel?
    Is there no native solution for spring cloud aws?

    I'll try overwrite AsyncTaskExecutor mentioned previously and I'll get back with the results.

    Danilo Gomes
    @danilocgomes_twitter

    Hey @ryangardner , I should confess that I've also used camel :D

    Thanks for your suggestion!

    Andy Czerwinski
    @czerwina
    How do people build and test locally their microservices if you use the spring-cloud-aws annotations in their code?
    Kyle Anderson
    @andersonkyle
    @czerwina I don't have an elegant answer, other than using Profile specific credentials. For example, for local development, I have a set of credentials on my local machine that are tied to a completely separate account from production. Same thing for staging. AWS Account segregation is what I do.
    @czerwina This can introduce config drift over time, but that's the best I've got.
    Andy Czerwinski
    @czerwina
    @kanderson450 so you need to deploy to aws for each developer environment? Don't think our infrastructure dudes would appreciate paying for the hardware even if cheap.
    We use redis, oracle xd, rabbitmq locally which translate to elasticated, rds and sqs
    *Elasticache
    Kyle Anderson
    @andersonkyle
    @czerwina I'd rather take the hit then use completely different components. RabbitMQ is nothing like SQS (even with an abstraction layer), and there's a lot of configuration that is specific to each. That's more config drift than separate AWS environments. And I'm not suggesting you need to deploy to AWS for local testing. You can make use of SQS, S3 and other services while running your app on your laptop. This isn't perfect either but SQS messages cost a miniscule amount.
    Andy Czerwinski
    @czerwina
    @kanderson450 didn't think of just exposing the services to the devs. U think via api gateway or vpn to the vpc?
    Kyle Anderson
    @andersonkyle
    @czerwina Depends on the service as they are not all available as private VPC endpoints. For example, SQS is not but S3 is. Your custom application services should be able to run locally all at once using docker-compose.
    The infrastructure services that are AWS specific (e.g. S3, SQS, SNS) can be called remotely from your laptop with Profile specific credentials tied to a completely separate environment (or even a seperate account).
    Steve Nera
    @xatnys
    Heya, I was taking a look at RDS configuration support for Aurora and noticed that support for it was explicitly added back in October 2015. I was testing read replica support and it wasn't detecting the read replicas I had created.
    From what I understand, Amazon very recently (in January specifically) updated aws-java-sdk-rds to add a field for properly obtaining Aurora read replicas. The existing API that spring-cloud-aws is using results in an empty list for read replica identifiers if you're using an Aurora cluster.
    Alain Sahli
    @alainsahli
    Thanks for mentioning that! We are going to look into it.
    Felipe Adorno
    @FelipeAdorno
    Hy everione, I create a register to create all my aws queus using annotations is interesting make a PR for this?
    I create the SqsConsumer to this and I use the parameters to create a SQS queue
    public @interface SqsConsumer {
    
        String[] value() default {};
    
        String delaySeconds() default "0";
    
        String maximumMessageSize() default "262144";
    
        String messageRetentionPeriod() default "345600";
    
        String policy() default "";
    
        String receiveMessageWaitTimeSeconds() default "0";
    
        String redrivePolicy() default "";
    
        String visibilityTimeout() default "30";
    }
    Kyle Anderson
    @andersonkyle

    Are you saying that you want to create the SQS queue if it doesn't exist, wherever the @SqsListener is defined? Something like:

    @SqsListener(value = "myQueue", createQueue=true)
    public void queueListener() {}

    Any clarification you can provide would be great. Thanks.

    Felipe Adorno
    @FelipeAdorno
    @andersonkyle I'll do this, thanks for reply :)
    Al Grant
    @bigalnz
    anyone here connected Intellij via the plug to AWS?
    Al Grant
    @bigalnz
    anyone here using AWS???
    Al Grant
    @bigalnz
    I have my webapp running in AWS beanstalk. Each time a connection is made it prints the IP address to the console - but its printing 127.0.0.1 - is that because beanstalk is using a proxy or?
    Kyle Anderson
    @andersonkyle
    @bigalnz Yes, definitely an AWS user but I can't say that I've used the IntelliJ plugin...
    Is your question specific to this repository?
    JonathanAaron
    @JonathanAaron
    I know this is AWS, but has anyone use Spring-Cloud Zuul and Eureka with Kubernetes?
    Kyle Anderson
    @andersonkyle
    @JonathanAaron Can't say that I have. Are you running into an issue?
    Gaurav Rawat
    @gauravbrills
    Is there any plan to support dimension publication through spring cloud aws actuator . Specially with relation to instance id's in case of multiple ec2 nodes publishing metrics of the same app .
    Nagilla Venkatesh
    @nagillavenkatesh
    Hi all,