Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 21 14:42
    singhajit89 commented #210
  • Jan 16 02:06
    simukappu closed #196
  • Jan 16 02:06
    simukappu commented #196
  • Jan 14 20:28
  • Jan 08 11:20
    singhajit89 commented #210
  • Jan 05 02:32
    simukappu commented #210
  • Jan 05 02:21
    simukappu commented #210
  • Jan 04 10:22
    simukappu labeled #210
  • Jan 04 10:20
    simukappu commented #210
  • Jan 04 10:19
    simukappu commented #210
  • Jan 04 10:19
    simukappu commented #210
  • Jan 04 10:19
    simukappu labeled #210
  • Jan 04 10:19
    simukappu unlabeled #210
  • Jan 04 10:05
    simukappu commented #201
  • Jan 04 10:05
    simukappu labeled #201
  • Dec 29 2020 07:41
    singhajit89 commented #210
  • Dec 24 2020 17:27
    chris-piekarski starred awslabs/aws-fluent-plugin-kinesis
  • Dec 23 2020 13:55
    simukappu unlabeled #150
Ryosuke Iwanaga
@riywo
This is a Gitter room for aws-fluent-plugin-kinesis. Feel free to ask questions here.
Ryosuke Iwanaga
@riywo
Finally released v2.0.0! If you have any trouble with this version, please let me know.
Ryosuke Iwanaga
@riywo
Released v2.0.1 to fix AWS SDK version conflict against s3 plugin.
ND
@najeebdanish
Hey can someone share the kubernetes configmap for kinesis
fluentd configmap
AlexFeng217
@AlexFeng217
Hello, is the fluent-kinesis-plugin windows compatible?
I wasn't sure from the docu
jamestmmadmin
@jamestmmadmin
has anyone got experience with Rekognition and video kinesis stream
Timesking
@timesking
Hello, I have a question when using compression, Do you decompress message in lambda before sending to firehose? especially when using kinesis_streams_aggregated, the standard KCL and firehose doesn't support zlib compressed message. so when do you usually decompress them?
qohat-lulo
@qohat-lulo
@qohat-lulo
Hello,
I'm new here, so i don't wanna make you waste your time guys... I have a problem trying to connect multiple consumer to kinesis shard... Someone know how could i do it? I'm using Java
Thank you for your help
neethujacobmec
@neethujacobmec

Hi Everyone, I am using kinesis agent to stream from application logs and lambda function to separate the logs based on some indices. However I am seeing the below error intermittently.

{ "error": { "root_cause": [ { "type": "remote_transport_exception", "reason": "[2f4424726309a9300ffdd0939e5cca77][x.x.x.x:9300][indices:data/write/bulk[s]]" } ], "type": "es_rejected_execution_exception", "reason": "rejected execution of processing of [195195722][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[misc-logs-2020-11-16][0]] containing [index {[misc-logs-2020-11-16][_doc][J9_q0HUBq-O-de1zDKJd], source[{\"message\":\"\\"End of file or no input: Operation interrupted or timed out (60 s recv delay) (60 s send delay)\\"\"}]}], target allocation id: yx7QdcoSRkecIXzgqLLeLg, primary term: 1 on EsThreadPoolExecutor[name = 2f4424726309a9300ffdd0939e5cca77/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@5205cb35[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 98568339]]" }, "status": 429 }

I am using multiline option set in agent.json to read multiline log statements together.

AWS Kinesis Agent offers settings like number of nodes and shards. However unsure where queue size may be increased as elasticsearch.yaml doesn't seem to be accessible. Any help would be appreciated.

1 reply