monix/monix-connect

@here greetings everyone, i'm very new to Fargate/ECS/Batch so forgive my ignorance, but I have a docker image/container thats running a scala batch job. it uses the aws-sdk-v2, hadoop client, and monix to make a bunch of parallel api calls to an http endpoint, bring the data back, do some transforms, and then make a bunch of parallel writes to s3. everything works fine locally, but when i run it as a batch job with fargate i get this error:

java.nio.file.AccessDeniedException: ...
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by ContainerCredentialsProvider : com.amazonaws.AmazonServiceException: Too Many Requests (Service: null; Status Code: 429; Error Code: null; Request ID: null)
when i try to write the data to s3 as parquet files with code that looks very similar to the example from the docs :

val parquetWriter: ParquetWriter[GenericRecord] = {
AvroParquetWriter.builderGenericRecord
.withConf(conf)
.withSchema(schema)
.build()
}

// returns the number of written records
val t: Task[Long] = {
Observable
.fromIterable(elements) // Observable[Person]
.map( => personToGenericRecord())
.consumeWith(ParquetSink.fromWriterUnsafe(parquetWriter))
}
it doesnt happen with every write taking place in the job, but it does with a lot of them. Here are my hadoop settings if that helps:

conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem")
conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
conf.set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.ContainerCredentialsProvider")

The reading ive done has made me more confused about the whole thing, like maybe its container settings for using IAM for creds, but it half works as is? i dunno,
I would just like this process to run like it does in docker on my local machine, any help would be much appreciated.