Where communities thrive

Join over 1.5M+ people
Join over 100K+ communities
Free without limits
Create your own community

Explore more communities

normation/rudder

Continuous Auditing & Configuration - please speak english and be nice ;)

People

Repo info

Activity

ism93

@ism93

if you have ideas please tell me

Nicolas Charles

@ncharles

It's because there are too many inventory to process at the same time

hence the full queue

ism93

@ism93

how can i flush the queue?

Francois Armand

@fanf

@ism93 it should empty by itself as fast as possible. You can check status by API: https://docs.rudder.io/api/#api-Inventories-queueInformation

are you in 6.0.z? Or 5.0.z?

ism93

@ism93

i am in 5.0.12

ok i will check with api

Francois Armand

@fanf

@ism93 in 5.0, it's an other local only API, see: https://docs.rudder.io/reference/5.0/reference/architecture.html#_inventory_workflow_from_nodes_to_root_server

hum

actually: https://docs.rudder.io/reference/5.0/reference/architecture.html#_queue_of_inventories_waiting_to_be_parsed

ism93

@ism93

yes, the queue is full

curl http://localhost:8080/endpoint/api/info
{"queueMaxSize":50, "queueFillCount":50, "queueSaturated":true}

Francois Armand

@fanf

is there a lot of inventories coming? If not, the queue should not remain saturated at all. Is there errors in log file?

(in /var/log/rudder/webapp/2020_02_14.stderrout.log)

ism93

@ism93

yes there is many errors on the log file,

errors is like : Rejecting Inventory Nodes

2020-02-14 16:01:02] INFO inventory-processing - Watch new inventory file 'rmmMYNODE-4171843a-1e1d-4f9f-80c6-c12a35232191.ocs' with signature available: process.
[2020-02-14 16:01:03] WARN inventory-processing - Rejecting Inventory 'rmmMYNODEl-4171843a-1e1d-4f9f-80c6-c12a35232191.ocs' for Node '4171843a-1e1d-4f9f-80c6-c12a35232191' because processing queue is full.
[2020-02-14 16:20:10] INFO inventory-processing - Watch new inventory file 'rmmMYNODE-4171843a-1e1d-4f9f-80c6-c12a35232191.ocs' with signature available: process.

this is a log about my node which is not in pending nodes

ism93

@ism93

I can restart the watcher so?

Francois Armand

@fanf

@ism93 : Watch new inventory file tells that new inventories are correctly seens and processing starts

but you should also have corresponding "inventory processed" (or inventory saved in...)

warning about queue full are expected since your queue is full. I was wondering for stacktraces or "ERROR"

ism93

@ism93

hello @fanf, the issue persists, i still have that kind of errors:

[2020-02-17 09:21:20] ERROR inventory-processing - Rejecting Inventory 'MYNODE-3fec0e5e-1790-4d3c-9596-63bb23ba7ad6.ocs' for Node '3fec0e5e-1790-4d3c-9596-63bb23ba7ad6' because the Inventory signature is not valid: the Inventory was not signed with the same agent key as the one saved within Rudder for that Node. If you updated the agent key on this node, you can update the key stored within Rudder with the following command on the Rudder Server: '/opt/rudder/bin/rudder-keys change-key 3fec0e5e-1790-4d3c-9596-63bb23ba7ad6 <your new public key>'. If you did not change the key, please ensure that the node sending that inventory is actually the node registered within Rudder

but the node is already on server, and the key was not changed, i don't understand why it is in error

I have 7110 line in errors in the /var/log/rudder/webapp/2020_02_17.stderrout.log file

awk '{print $3}' /var/log/rudder/webapp/2020_02_17.stderrout.log | grep ERROR | wc -l

Francois Armand

@fanf

@ism93 : this error is different from previous ones. Here, the inventory is not refused because of performance (processing queue fully filled), but because of consistance. It may be because there is an other node with the same UUID, for example if you are duplicating server from a common master.

ism93

@ism93

But I still have this error: curl http://localhost:8080/endpoint/api/info
{"queueMaxSize":50, "queueFillCount":50, "queueSaturated":true}

Francois Armand

@fanf

@ism93 ok, just that they are different things and need to be processed appart, else we won't be able to progress. Do you have recent log saying that inventories are accepted?

if not, when if the last time you had one ?

Nicolas Charles

@ncharles

how long does it take to accept an inventory according to the logs?

sr57

@sr57

I got this error : Cannot update configuration policy, on one of my node, a mobile PC that changed location. Any explanation? How investigate?

Alexis Mousset

@amousset

You can check if it is in one of the authorized networks for your server (or relay)

you can also check if you can connect from node to server on port 5309 (using something like telnet)

ism93

@ism93

@fanf no, i don't have log that says that inventory has been accepted, but i did: rudder agent factory-reset on one node: and the log say : tailf /var/log/rudder/webapp/2020_02_17.stderrout.log | grep rmm-p2000305
[2020-02-17 14:45:51] INFO inventory-processing - Watch new inventory file 'MYNODEfl-13634d7b-cc34-4be4-8078-f9fcb189ea88.ocs' with signature available: process.
[2020-02-17 14:45:51] WARN inventory-processing - Rejecting Inventory MYNODE-13634d7b-cc34-4be4-8078-f9fcb189ea88.ocs' for Node '13634d7b-cc34-4be4-8078-f9fcb189ea88' because processing queue is full.
[2020-02-17 14:46:04] INFO inventory-processing - Watch new inventory file MYNODE-84d5e1cf-8b59-4134-94ec-d648c040c912.ocs' with signature available: process.
[2020-02-17 14:46:05] WARN inventory-processing - Rejecting Inventory 'MYNODEl-84d5e1cf-8b59-4134-94ec-d648c040c912.ocs' for Node '84d5e1cf-8b59-4134-94ec-d648c040c912' because processing queue is full.

the queue is still full

Francois Armand

@fanf

@ism93 if you don't have any successfully accepted inventories, something is wrong with rudder. Inventory processing should not be longer than a couple of seconds. Is it possible to restart rudder-jetty? It seems something is blocked (but I can't tell what)

ism93

@ism93

@fanf yes the principal issue is that since friday,

sr57

@sr57

@amousset
1) authorized. Local network (ie same as rudder server)
2) telnet rudder 5309 : Ok

Alexis Mousset

@amousset

then you can have a look at logs from cf-serverd in /var/log/message or /var/log/syslog, you should see a reason for the refusal

ism93

@ism93

@fanf after the restart, the node is not accepted yet

sr57

@sr57

@amousset

#012 rudder-agent could not get an updated configuration from the policy server. #012 This can be caused by a network issue, an unavailable server, or if this #012 node was deleted from the Rudder root server. #012 Any existing configuration policy will continue to be applied without change. #012

Strange msg for me, I understand that the node was deleted! ?

Alexis Mousset

@amousset

@sr57 On the Rudder server, I forgot to precise it.

sr57

@sr57

@amousset Thaks! :-) <wan ip> not in allowconnects! After having putting back my laptop to the "rudder network", it connects via the internet ip and not the local ip!

Alexis Mousset

@amousset

@sr57 good!

Francois Armand

@fanf

@ism93 how many inventory files is there in 1/ /var/rudder/inventories/incoming and 2/ /var/rudder/inventories/accepted-nodes-updates ? Can you send us the log file /var/log/rudder/webapp/2020_02_17.stderrout.log ?

(not in the public chan, either by email or at least in private chan)