spring-cloud/spring-cloud-aws

Integration for Amazon Web Services APIs with Spring

hdutra

@hdutra

Hello all,

hdutra

@hdutra

I'm trying use the spring-cloud-aws-jdbc in my project, but when I receive a large number of user I have a problem, It seems like my pool of connections is full.

Exception: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:100; busy:100; idle:0; lastwait:30000]

I have used C3P0 in it with no problem. Is possible I use C3P0 for pool connection instead Tomcat pool?

2 replies

Mateusz Mrozewski

@mateuszmrozewski

Hi. Is mvn package reformatting sources of spring-cloud-aws for you as well? Do I need to do any additional setup not to have such behavior?

ayusuf

@ayusuf

Hi, does anybody know how to use multiple AWS keys in a single Spring Cloud AWS project?

harinjy

@harinjy

HI, I am trying to integrate spring cloud with aws secrets manager. I attached an instance profile to my EC2 which allows reading parameters from Secrets manager. I am getting connection timeout error. It is trying to connect to cloudformation.us-east-1.amazonaws.com. I guess the call is routing to internet and firewall is blocking it. I sthere way i can configure my spring boot project to use cloudformation endpointso that it won't go to public internet? I am following this..https://cloud.spring.io/spring-cloud-static/spring-cloud-aws/2.1.0.RELEASE/single/spring-cloud-aws.html#_integrating_your_spring_cloud_application_with_the_aws_secrets_manager

Raj

@rajjaiswalsaumya

How to connect to aurora db rds ?

Robert Wiesner

@rowi1de

Hey guys ... question to spring-cloud-aws / multiple listeners on one queue

 class A{
  @SqsListener(value = "some-queue", deletionPolicy = ON_SUCCESS)
  public void onMessageReceived1(String payload, @Header(TYPE) String messageType) {}
 }

 class B {
  @SqsListener(value = "some-queue", deletionPolicy = ON_SUCCESS)
  public void onMessageReceived2(@Payload SomeMessagePayload message, @Header(TYPE) String messageType) {}
 }

would that work? btw. is the Method Name taken into account?

I got "Ambiguous handler methods mapped for destination " when both methods are called "onMessageReceived"

2 replies

Kris Krishna

@kriskrishna

Anybody used AWS MSK with spring-cloud-aws

nbenjamin

@nbenjamin

Hi all, When i am using @SqsListener with Object its not getting the value

    @SqsListener("Demo")
    public void processMessage(Person person) {
        System.out.println("Message Received - " + person.getName());
    }

I get the value - Message Received - null

When i debug i can see the GenericMessage payload with actual value. I thought If jackson library is in class path message converstion happens automatically

I have configured QueueMessageHandlerFactory with StrictContentTypeMatch- false

    @Bean
    public QueueMessageHandlerFactory queueMessageHandlerFactory() {
        QueueMessageHandlerFactory factory = new QueueMessageHandlerFactory();
        MappingJackson2MessageConverter messageConverter = new MappingJackson2MessageConverter();

        //set strict content type match to false
        messageConverter.setStrictContentTypeMatch(false);
        factory.setArgumentResolvers(Collections.singletonList(new PayloadMethodArgumentResolver(messageConverter)));
        return factory;
    }

nbenjamin

@nbenjamin

Getting this error

org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [com.nbenjam.springcloudawsmessaging.domain.Person] for GenericMessage [payload={
"Type" : "Notification",
"MessageId" : "6f18d976-c801-58a8-be7a-8fdc47069ac8",
"TopicArn" : "arn:aws:sns:us-east-1:88s8s8s:demo-topic",
"Message" : "{\"name\":\"Ryan-Adam\",\"age\":10}",
"Timestamp" : "2020-01-28T19:34:31.951Z",
"SignatureVersion" : "1",

Taylor Bruce

@tbruce731CH

Hello All, I want to be able to extend message visibility in SQS. How can I do this using Spring Framework? Currently I have it working well and deserializing my body payload to an object however I want to be able to get the ReceiptHandle for the message in order to extend the message visibility as my application does work.
Thanks!

Fernando Montes

@ferzerkerx

Hi
I've upgraded from spring-boot from 2.2.2.RELEASE to 2.2.4.RELEASE. and this seems to change the behavior of the sent metrics to aws (I'm getting lot's of The parameter MetricData.member.7.Dimensions.member.5.Value is required.) I'm using io.micrometer:micrometer-registry-cloudwatch:1.3.2 (I've also tried 1.3.3), on a separate note I've just noticed that io.micrometer:micrometer-registry-cloudwatch is now deprecated and that there is a io.micrometer:micrometer-registry-cloudwatch2 (I've tried this one but it fails to be picked up by spring)

so the questions are:

how can I debug which metric is causing problems? any logs I can enable?
seems like io.micrometer:micrometer-registry-cloudwatch2 is they way to go now is it compatible with spring-cloud-aws?
have you encountered something similar? and how did you fix it

thanks in advance

@nbenjamin I've seen that problem, in a nutshell you seem to be expecting a raw message, but you are getting a wrapped message, try to change your mapping from:

foo(Person person){} to foo (@NotificationMessage Person person) I'm assuming this is the listener (your queue on aws should not send the raw message)

Taylor Bruce

@tbruce731CH

Also I believe I am having a memory leak in the spring cloud aws library - Ive removed all business logic from my application and simply have a SQSListener that spits out the message to stdout. After running through about 30k messages across 10 k8s pods the memory starts to run away. Any pointers?

junior6886

@junior6886

Do I need spring cloud to use spring cloud aws?

1 reply

Harish Chava

@hchava

Hello There, Is there a way to monitor any new objects creation events on Aws S3 bucket using Spring?
If so, Please provide the best practices. Fyi, I've only read access for a 3rd party S3 bucket using aws access and secret keys.

Bernardo Martins

@bsamartins

@hchava, best way is setting up a rule in S3 that reacts to certain events and will push them to SQS or similar.
You can then have a listener in Spring that consumes those events.
https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

Harish Chava

@hchava

@bsamartins Thank you for the reply! unfortunately I don't have any other access to the bucket apart from the read access using certain prefix. we're consumers for a Third party S3 bucket.

Maciej Walkowiak

@maciejwalkowiak

👋

Oleksandr

@giova333

👋

Eddú Meléndez Gonzales

@eddumelendez

hi!

Eddú Meléndez Gonzales

@eddumelendez

have been thinking about 2 things looking at the current source code. 1. Remove the @Autowire annotations and go for constructor injection or use ObjectProvide for those with required=false. 2 . Move the cloudwatch support to v2

Maciej Walkowiak

@maciejwalkowiak

Hi @eddumelendez! :+1: to both ideas. I am thinking about solid re-do of configuration classes. For 2.3.x I think we can drop XML support and also drop @Enable classes. This would allow us move all configuration completely to autoconfigure module and use Spring Boot @Conditional annotations. There we could use ObjectProviders

I'll put today/tomorrow together my thoughts on 2.3.x and 3.x. I see 3.x as move to AWS SDK v2 - this will be a lot of work and I would like to drop everything we don't need and don't want to support just to reduce amount of code to migrate.

I started working already on 2.3.x but everything is up to discussion and your input @eddumelendez will be extremely welcome!

Spencer Gibb

@spencergibb

:+1: @eddumelendez

jarc0der

@jarc0der

Hi @maciejwalkowiak !
Currently looking at the spring cloud messaging. Seems a lot of code are based on spring messaging framework.
Looking at you reactive-sqs PoC looks more readable and understandable. Of course it's PoC and more features should be implemented.
Are you considering migration from spring messaging mechanism for sqs/sns support on spring cloud?

Maciej Walkowiak

@maciejwalkowiak

Hi @jarc0der! Ideally we do not migrate away from Spring Messaging as there are some downstream projects that depend on Spring Cloud AWS like Spring Integration AWS and I am afraid we could break them. Perhaps there is a way to use reactive approach and keep it Spring Messaging compatible (just guessing, I am still digging into this topic). Ideally whatever abstraction we come up with should let us build something like Spring Cloud Stream SQS. I did a POC for that long time ago https://github.com/maciejwalkowiak/spring-cloud-stream-binder-sqs but since it's based on Spring Cloud AWS it also has all the flaws of current implementation of SQS listener.

Philip Riecks

@rieckpil_twitter

Hey :wave: I just submitted an issue to optimize the integration test setup(spring-cloud/spring-cloud-aws#572). It would be great if you can share some thoughts on this :thumbsup:

Philip Riecks

@rieckpil

Does the Intellij_Spring_Boot_Java_Conventions file supersedes the eclipse-code-formatter.xml? While reading the README in the coding conventions it says to install the Eclipse Formatter for IntelliJ while in the IntelliJ IDE setup one should use the coding conventions from Spring Cloud Build?

Maciej Walkowiak

@maciejwalkowiak

@rieckpil IDE setup is another thing that needs to be refreshed. I did not use eclipse-code-formatter plugin at all. Code is being formatter through spring-javaformat Maven plugin automatically when you build the project

3 replies

Shawn Johnson

@shawnjohnson

I feel like I am missing a major concept with the SqsListener - How/where is it determined how frequently the SQS queue is checked?

Rory Byrne

@synek

Is it possible to register two listeners for a single SQS queue? One or the other should be used to handle the event, depending on the structure of the message body. Example:

@Slf4j
@Component
public class MyListener {

  @SqsListener("my-queue")
  public void handler(Foo foo) {
    log.debug(foo);
  }

  @SqsListener("my-queue")
  public void otherHandler(Bar bar) {
    log.debug(bar);
  }

}

My queue receives different messages for different events, so I'd like to have a different handler for each event type

Maciej Walkowiak

@maciejwalkowiak

@synek you would need to implement this dispatching logic yourself. It's meant to have single @SqsListener per queue

Mete Alpaslan Katırcıoğlu

@mkatircioglu

Hi @maciejwalkowiak I have provided configuration properties for AmazonRdsDatabaseAutoConfiguration as you suggested in spring-cloud/spring-cloud-aws#598 but I couldn't find how I can bind AnnotationAttributes to configuration properties for AmazonRdsInstanceConfiguration. How can I handle that? I would be very happy, if you can guide me.

Rory Byrne

@synek

Hi folks there's a bug with the Spring Cloud AWS documentation website - code examples are illegible due to the text color matching the background color. https://cloud.spring.io/spring-cloud-aws/2.2.x/reference/html/

1 reply

Or is it just me? I suppose it's possible that my browser is broken

Rory Byrne

@synek

Petar Tahchiev

@ptahchiev

Hey everyone, I'm searching all over the internet to see if it possible to delete a file on AWS S3 using the spring-cloud API, and I found this: https://stackoverflow.com/questions/51304075/how-to-delete-file-from-s3-using-spring-cloud-aws
which claims that is is not possible. Is this really the case??? This seems like a very basic functionality

2 replies

Andy Dior VP of Product

@DwyerBandEmpire_twitter

I apologize if this has been covered, but I was looking at the Spring Cloud roadmap and noticed this in the 2020.0.0-M1 release:

Spring Cloud AWS and Spring Cloud GCP are no longer part of the release train. They will continue to be part of Hoxton as long as it is supported – at least thru June of 2021.

(https://spring.io/blog/2020/04/17/spring-cloud-2020-0-0-m1-released). Having read through a few PRs in the project, it looks like Spring Cloud AWS 3.0 will be a major release. Is it just the case that Spring Cloud AWS is no longer part of the Spring Cloud release train?

Spencer Gibb

@spencergibb

@DwyerBandEmpire_twitter yes, it will be released on it's own. It will move to a new github org and have a new maven groupid and new package names (all still undecided yet)

1 reply

Juan Carlos Ledesma

@juancarlos_la_twitter

Hi guys. I want to implement concurrency (multiple workers) in the SqsListeners. I already have several projects that use that same implementation (as a library). Ideally, keep the implementation with the annotation (@SqsListener) and make the changes in the configuration class

I read an AWS documentation about that, but it's a bit old, so I don't know if I need to change my implementation, or if there is a way to set the concurrency attributes in the config class.

Link: https://aws.amazon.com/blogs/developer/using-amazon-sqs-with-spring-boot-and-spring-jms/

The following is the implementation that I currently have, but I don't know how to set the concurrency attributes:

Config class

public class SqsConfig {

    @Bean
    public QueueMessagingTemplate queueMessagingTemplate(AmazonSQSAsync amazonSQSAsync) {
        return new QueueMessagingTemplate(amazonSQSAsync);
    }

    @Bean
    public AmazonSQS sqsClient() {
        return AmazonSQSClientBuilder.defaultClient();
    }
}

Consumer class:

@Service
public class Consumer implements ISqsStandardConsumer{

    @Override
    @SqsListener("SQS-concurrency-test")
    public void receiveStandardMessage(String message) {
        System.out.println(message);
    }  
}

I really appreciate your comments

2 replies

Mete Alpaslan Katırcıoğlu

@mkatircioglu

spring-cloud-aws is using cloud.aws prefix for configuration properties. Would it be good to rename as spring.cloud.aws in next major release 3.0.0.M1? What do you think? @maciejwalkowiak @eddumelendez

2 replies

Pedro Marques

@pedrorlmarques_twitter

Hello I don't know if this is a bug or simple spring cloud aws doesn't support this.
In order to have access to my aws account from AWS CLi, I have to login through a npm app called aws-azure-login and this redirects me to my organization login page and creates and all the information necessary on .aws/config file and .aws/credentials.  

.aws/config

  [profile landing-zone-iam]
azure_tenant_id=7e4c72b8-86d8-4e4a-90d8-4f7b7b2513c4
azure_app_id_uri=https://signin.aws.amazon.com/saml
azure_default_username=email@email.com
azure_default_role_arn=
azure_default_duration_hours=10

.aws/credentials

  [landing-zone-iam]
aws_access_key_id=ASIAVAXUICQBJKDAHRSU
aws_secret_access_key=z+uN+nklhHLC5RbAQQK8AnMgIV8vAbFIr3iJR5Cg
aws_session_token=FwoGZXIvYXdzEGsaDGWJ/5ZJZgsV3mRkaiKpAttDP2k7Pkcog04eQvT64bqtTf6cyGW5A+tGBC1vCo6HWnqFQCorPTyaftKk4EDvlenwMk8tYBMgtuGk+I9fMcuFpczuEipEKC4Bmbff9NlwZf7NyqY0xMLG28SQG7p9RZ2OX7mQ0OxT9r7xGCpmt6+nuuykHQRM+vjIAejFu8IgOJvxsYqCB+LuNkgVeT9gEjxAsJKKpD3wyJbiyDCJd+RVDEW+smxQ1fO3PqAX0H9F8MOJfaKpzb8m6bMKgWH59Vg9yb6wW5SnoKcR9m/kt546VO1FYwpnEgWHk/chFivey0iBTxWHz3CMA8ga7e5jtmFNt2/vPLZCoXHyyE194l9i7oPbOGNQn/gpkLF7B43N+s/t/KQe8Uek+xp4tjEtar2+nxFZfZTFLyiclKn5BTIq5dpEXmpQmCkCNgjkpx+kBXNESDnoGu0sTSBleltUemcqZkOT3I9PRC8o
aws_expiration=2020-08-20T20:02:04.000Z

[app-dev]
role_arn=arn:aws:iam::792079637043:role/admin-role
region=eu-west-1
source_profile=landing-zone-iam
external_id=1234
role_session_name=session_name

I can use the cli e.g aws s3 ls - -profile app-dev and I have results  If I try to use spring cloud aws I cannot run the application   

application.properties

  spring.application.name=demo-aws
cloud.aws.stack.auto=false
cloud.aws.credentials.use-default-aws-credentials-chain=true
cloud.aws.region.use-default-aws-region-chain=true
cloud.aws.credentials.profile-name=app-dev

Exception: 

Caused by: com.amazonaws.SdkClientException: Unable to load region information from any provider in the chain
at com.amazonaws.regions.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:59) ~[aws-java-sdk-core-1.11.792.jar:na]  Does Spring Cloud AWS support this kind of authentication ?  

ctruzzi

@ctruzzi

Hello all,
I am trying to figure out if there are any Spring Bean Scopes that work with Spring Cloud AWS for the entire process of a @SqsListener call. I'd like to autowire certain data through the call cycle but it appears like request doesn't work and I don't believe any of the other scopes by default support that.

Corneil du Plessis

@corneil

I'm trying to deploy a Spring Boot application to AWS Beanstalk. A year ago I wrote an article on how to do it but now those same instructions fail: https://www.baeldung.com/spring-boot-deploy-aws-beanstalk
I've tried with an without Spring Cloud AWS enabled. In the one profile I exclude:

  org.springframework.cloud.aws.autoconfigure.cache.ElastiCacheAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.context.ContextCredentialsAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.context.ContextInstanceDataAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.context.ContextRegionProviderAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.context.ContextResourceLoaderAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.context.ContextStackAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.jdbc.AmazonRdsDatabaseAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.mail.MailSenderAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.messaging.MessagingAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.metrics.CloudWatchExportAutoConfiguration

In the other profile I only excluded:

org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration,\
  org.springframework.boot.autoconfigure.liquibase.LiquibaseAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.cache.ElastiCacheAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.mail.MailSenderAutoConfiguration,\
  org.springframework.cloud.aws.autoconfigure.messaging.MessagingAutoConfiguration

So the AmazonRdsDatabaseAutocConfiguration should be active.
I tried with the RDS database as part of Beanstalk application and as a separate RDS database instance.

It can't be this difficult.

Davy De Waele

@ddewaele

I know it is possible to retrieve ec2 instance data : https://cloud.spring.io/spring-cloud-aws/reference/html/#using-instance-metadata) when using spring-cloud-aws, but is there also a way to retrieve ECS / Fargate meta data when your spring boot service is running in an ECS / Fargate container ?

Gayan Weerakutti

@gayanW

Adding spring-cloud-starter-aws-secrets-manager-config to 2.3.4.RELEASE - Hoxton.SR8, project throws the following:

com.amazonaws.util.EC2MetadataUtils : Unable to retrieve the requested metadata (/latest/meta-data/instance-id). 
Failed to connect to service endpoint: 

com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
        at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[aws-java-sdk-core-1.11.792.jar:na]
....

Caused by: java.net.SocketTimeoutException: connect timed out

But downgrading the springCloudVersion to Hoxton.SR6 do fix it.

Where communities thrive

People

Repo info

Activity