ovh/kubernetes

Let's talk about our Kubernetes As a Service Product #OVH

Bmagic

@bmagic

Ok I have some strange log message on logs metrics server :

E0915 10:32:05.134618       1 summary.go:97] error while getting metrics summary from Kubelet node-c4260b85-c8c7-4f7d-88e9-id( [IP]:10250): Get https://[IP]:10250/stats/summary/: dial tcp [IP]2:10250: getsockopt: no route to host

Nicolas Steinmetz

@nsteinmetz

@MrDienns did you check the Certificate object was celle created first ? (kubectl describe secrets/YourCertificate if I remember well ?)

@MrDienns so all is working ? sorry I spent most of my week-end in the garden :-)

Thomas Coudert

@thcdrt

Hello @bmagic , can you send me in private your cluster id and the node IP please ?

Michał Frąckiewicz

@SystemZ

any tips how to get all container logs sent to ovh log data platform?

I'm searching for something similar to this but with TLS output
https://github.com/roffe/kube-gelf

as far as I know, logstash doesn't support TLS for GELF protocol as input and fluentd doesn't support syslog output <facepalm>

Joël LE CORRE

@jlecorre_gitlab

Hello @SystemZ Maybe this documentation can help you?
I'm not sure if it meets your needs: https://docs.ovh.com/fr/logs-data-platform/kubernetes-fluent-bit/

Michał Frąckiewicz

@SystemZ

Looks promising, I'll check it :)

Jawher Moussa

@jawher_twitter

Hi ! I tried to follow the fluent-bit tutorial above, but it references the tag fluent/fluent-bit:1.3, but that version does not exist (the daemonset pods fail with ImagePullErr. According to dockerhub, the latest version is 1.2.2.

Using that version (1.2), the fluent-bit pods continuously log [2019/09/17 12:01:58] [error] [out_gelf] error encoding to GELF

Michał Frąckiewicz

@SystemZ

Yeah, I encountered this too

I tried :latest version of image and it didn't work either

Thomas Coudert

@thcdrt

Hi @jawher_twitter , checking with the Logs team

Jawher Moussa

@jawher_twitter

Thanks 🤗

Michał Frąckiewicz

@SystemZ

it's probably related to filter-kubernetes.conf
fluent/fluent-bit#1070

Thomas Coudert

@thcdrt

Yes, there is an issue on the 1.2 version that has been fixed on master but which is not yet released.

We're seeing how we can get around the problem until this modification has been prodded

Michał Frąckiewicz

@SystemZ

I'll try to build docker image from master

Jawher Moussa

@jawher_twitter

Thank you @thcdrt , I appreciate the prompt response ! I'll try building fluentbit from master :+1:

Thomas Coudert

@thcdrt

Ok @jawher_twitter , keep me updated :)

Michał Frąckiewicz

@SystemZ

I'll already pushed it systemzz/fluent-bit:latest

so far it doesn't show that error, I'll check if logs are being sent

yep, it works, only docker image needs to be changed

no errors, logs visible in graylog

Thomas Coudert

@thcdrt

Great, thanks @SystemZ :)

We will push it on one of our repo and modify doc until fluentbit release its new version

Michał Frąckiewicz

@SystemZ

:thumbsup:

Michał Frąckiewicz

@SystemZ

I see that calico-node is pretty chatty, any idea how to ignore containers from sending logs to graylog ?

arduinio

@arduinio

you could use pipeline in graylog to remove or prevent

i thimnk it will be possible

think

Michał Frąckiewicz

@SystemZ

on selfhosted, yea, on ovh offer I think it's not available

arduinio

@arduinio

i use my custom graylog stack

sorry

Michał Frąckiewicz

@SystemZ

I used my too but each malfunction of graylog + ES debugging melted my brain too much

Thomas Coudert

@thcdrt

Hello @SystemZ , you can use this filter => https://docs.fluentbit.io/manual/v/1.2/filter/grep

It allow to exclude some logs

Thomas Coudert

@thcdrt

We deployed the docker image on ovhcom/fluent-bit. We're updating the doc too.

Thomas Coudert

@thcdrt

Pull request on ovh/docs is opened => ovh/docs#339

Michał Frąckiewicz

@SystemZ

thanks! grep works wonderful in fluent bit
I added something like this to the end of filter-kubernetes.conf config map:

    [FILTER]
        Name    grep
        Match   *
        Exclude kubernetes_container_name calico-node
    [FILTER]
        Name    grep
        Match   *
        Exclude kubernetes_container_name wormhole

Thomas Coudert

@thcdrt

@SystemZ Glad to hear !

bcarriou

@bcarriou

Hi, I'am not sure but can you confirm me if my etcd cluster is UP on my cluster ?
I have some message on my RAncher cluster about that

E0917 15:25:22.936453 9 leaderelection.go:286] Failed to update lock: etcdserver: request timed out
2019/09/17 15:25:24 [ERROR] ClusterController local [cluster-stats] failed with : etcdserver: request timed out

Philippe Vienne

@PhilippeVienne_gitlab

@bcarriou Same on my side since days and etcd is fine, just liveness is not replying because on another network.

Thomas Coudert

@thcdrt

Hello @bcarriou , where do you see those logs ?

bcarriou

@bcarriou

@PhilippeVienne_gitlab ok thanks :)
@thcdrt It is on my Rancher cluster but it is due because etcd isn't accessible now => thx @PhilippeVienne_gitlab

bcarriou

@bcarriou

Hi, I tried to upgrade version from 1.12 to 1.13 but I have the state Erreur in my OVH UI.
Some info:

k get nodes
NAME       STATUS                        ROLES    AGE    VERSION
worker-1   Ready                         <none>   202d   v1.12.10
worker-2   Ready                         <none>   158d   v1.12.10
worker-3   NotReady,SchedulingDisabled   <none>   92d    v1.12.10

bcarriou

@bcarriou

my cluster id: 1aa61333-b08c-4ea9-abc0-82f8b39b5cb2

SebastienJardin

@SebastienJardin

Hi @bcarriou I'm working on your problem, we can switch on DM if you want?

bcarriou

@bcarriou

@SebastienJardin yes thx

Where communities thrive

People

Repo info

Activity