Where communities thrive

  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
Repo info
Ryosuke Iwanaga
This is a Gitter room for aws-fluent-plugin-kinesis. Feel free to ask questions here.
Ryosuke Iwanaga
Finally released v2.0.0! If you have any trouble with this version, please let me know.
Ryosuke Iwanaga
Released v2.0.1 to fix AWS SDK version conflict against s3 plugin.
Hey can someone share the kubernetes configmap for kinesis
fluentd configmap
Hello, is the fluent-kinesis-plugin windows compatible?
I wasn't sure from the docu
has anyone got experience with Rekognition and video kinesis stream
Hello, I have a question when using compression, Do you decompress message in lambda before sending to firehose? especially when using kinesis_streams_aggregated, the standard KCL and firehose doesn't support zlib compressed message. so when do you usually decompress them?
I'm new here, so i don't wanna make you waste your time guys... I have a problem trying to connect multiple consumer to kinesis shard... Someone know how could i do it? I'm using Java
Thank you for your help

Hi Everyone, I am using kinesis agent to stream from application logs and lambda function to separate the logs based on some indices. However I am seeing the below error intermittently.

{ "error": { "root_cause": [ { "type": "remote_transport_exception", "reason": "[2f4424726309a9300ffdd0939e5cca77][x.x.x.x:9300][indices:data/write/bulk[s]]" } ], "type": "es_rejected_execution_exception", "reason": "rejected execution of processing of [195195722][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[misc-logs-2020-11-16][0]] containing [index {[misc-logs-2020-11-16][_doc][J9_q0HUBq-O-de1zDKJd], source[{\"message\":\"\\"End of file or no input: Operation interrupted or timed out (60 s recv delay) (60 s send delay)\\"\"}]}], target allocation id: yx7QdcoSRkecIXzgqLLeLg, primary term: 1 on EsThreadPoolExecutor[name = 2f4424726309a9300ffdd0939e5cca77/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@5205cb35[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 98568339]]" }, "status": 429 }

I am using multiline option set in agent.json to read multiline log statements together.

AWS Kinesis Agent offers settings like number of nodes and shards. However unsure where queue size may be increased as elasticsearch.yaml doesn't seem to be accessible. Any help would be appreciated.

1 reply