Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    ism93
    @ism93
    if you have ideas please tell me
    Nicolas Charles
    @ncharles
    Hi
    It's because there are too many inventory to process at the same time
    hence the full queue
    ism93
    @ism93
    how can i flush the queue?
    Francois Armand
    @fanf
    @ism93 it should empty by itself as fast as possible. You can check status by API: https://docs.rudder.io/api/#api-Inventories-queueInformation
    are you in 6.0.z? Or 5.0.z?
    ism93
    @ism93
    i am in 5.0.12
    ok i will check with api
    Francois Armand
    @fanf
    hum
    ism93
    @ism93
    yes, the queue is full
    curl http://localhost:8080/endpoint/api/info
    {"queueMaxSize":50, "queueFillCount":50, "queueSaturated":true}
    Francois Armand
    @fanf
    is there a lot of inventories coming? If not, the queue should not remain saturated at all. Is there errors in log file?
    (in /var/log/rudder/webapp/2020_02_14.stderrout.log)
    ism93
    @ism93
    yes there is many errors on the log file,
    errors is like : Rejecting Inventory Nodes
    2020-02-14 16:01:02] INFO inventory-processing - Watch new inventory file 'rmmMYNODE-4171843a-1e1d-4f9f-80c6-c12a35232191.ocs' with signature available: process.
    [2020-02-14 16:01:03] WARN inventory-processing - Rejecting Inventory 'rmmMYNODEl-4171843a-1e1d-4f9f-80c6-c12a35232191.ocs' for Node '4171843a-1e1d-4f9f-80c6-c12a35232191' because processing queue is full.
    [2020-02-14 16:20:10] INFO inventory-processing - Watch new inventory file 'rmmMYNODE-4171843a-1e1d-4f9f-80c6-c12a35232191.ocs' with signature available: process.
    this is a log about my node which is not in pending nodes
    ism93
    @ism93
    I can restart the watcher so?
    Francois Armand
    @fanf
    @ism93 : Watch new inventory file tells that new inventories are correctly seens and processing starts
    but you should also have corresponding "inventory processed" (or inventory saved in...)
    warning about queue full are expected since your queue is full. I was wondering for stacktraces or "ERROR"
    ism93
    @ism93
    hello @fanf, the issue persists, i still have that kind of errors:
    [2020-02-17 09:21:20] ERROR inventory-processing - Rejecting Inventory 'MYNODE-3fec0e5e-1790-4d3c-9596-63bb23ba7ad6.ocs' for Node '3fec0e5e-1790-4d3c-9596-63bb23ba7ad6' because the Inventory signature is not valid: the Inventory was not signed with the same agent key as the one saved within Rudder for that Node. If you updated the agent key on this node, you can update the key stored within Rudder with the following command on the Rudder Server: '/opt/rudder/bin/rudder-keys change-key 3fec0e5e-1790-4d3c-9596-63bb23ba7ad6 <your new public key>'. If you did not change the key, please ensure that the node sending that inventory is actually the node registered within Rudder
    but the node is already on server, and the key was not changed, i don't understand why it is in error
    I have 7110 line in errors in the /var/log/rudder/webapp/2020_02_17.stderrout.log file
    awk '{print $3}' /var/log/rudder/webapp/2020_02_17.stderrout.log | grep ERROR | wc -l
    Francois Armand
    @fanf
    @ism93 : this error is different from previous ones. Here, the inventory is not refused because of performance (processing queue fully filled), but because of consistance. It may be because there is an other node with the same UUID, for example if you are duplicating server from a common master.
    ism93
    @ism93
    But I still have this error: curl http://localhost:8080/endpoint/api/info
    {"queueMaxSize":50, "queueFillCount":50, "queueSaturated":true}
    Francois Armand
    @fanf
    @ism93 ok, just that they are different things and need to be processed appart, else we won't be able to progress. Do you have recent log saying that inventories are accepted?
    if not, when if the last time you had one ?
    Nicolas Charles
    @ncharles
    how long does it take to accept an inventory according to the logs?
    sr57
    @sr57
    I got this error : Cannot update configuration policy, on one of my node, a mobile PC that changed location. Any explanation? How investigate?
    Alexis Mousset
    @amousset
    You can check if it is in one of the authorized networks for your server (or relay)
    you can also check if you can connect from node to server on port 5309 (using something like telnet)
    ism93
    @ism93
    @fanf no, i don't have log that says that inventory has been accepted, but i did: rudder agent factory-reset on one node: and the log say : tailf /var/log/rudder/webapp/2020_02_17.stderrout.log | grep rmm-p2000305
    [2020-02-17 14:45:51] INFO inventory-processing - Watch new inventory file 'MYNODEfl-13634d7b-cc34-4be4-8078-f9fcb189ea88.ocs' with signature available: process.
    [2020-02-17 14:45:51] WARN inventory-processing - Rejecting Inventory MYNODE-13634d7b-cc34-4be4-8078-f9fcb189ea88.ocs' for Node '13634d7b-cc34-4be4-8078-f9fcb189ea88' because processing queue is full.
    [2020-02-17 14:46:04] INFO inventory-processing - Watch new inventory file MYNODE-84d5e1cf-8b59-4134-94ec-d648c040c912.ocs' with signature available: process.
    [2020-02-17 14:46:05] WARN inventory-processing - Rejecting Inventory 'MYNODEl-84d5e1cf-8b59-4134-94ec-d648c040c912.ocs' for Node '84d5e1cf-8b59-4134-94ec-d648c040c912' because processing queue is full.
    the queue is still full
    Francois Armand
    @fanf
    @ism93 if you don't have any successfully accepted inventories, something is wrong with rudder. Inventory processing should not be longer than a couple of seconds. Is it possible to restart rudder-jetty? It seems something is blocked (but I can't tell what)
    ism93
    @ism93
    @fanf yes the principal issue is that since friday,
    sr57
    @sr57
    @amousset
    1) authorized. Local network (ie same as rudder server)
    2) telnet rudder 5309 : Ok
    Alexis Mousset
    @amousset
    then you can have a look at logs from cf-serverd in /var/log/message or /var/log/syslog, you should see a reason for the refusal
    ism93
    @ism93
    @fanf after the restart, the node is not accepted yet
    sr57
    @sr57

    @amousset

    #012 rudder-agent could not get an updated configuration from the policy server. #012 This can be caused by a network issue, an unavailable server, or if this #012 node was deleted from the Rudder root server. #012 Any existing configuration policy will continue to be applied without change. #012

    Strange msg for me, I understand that the node was deleted! ?

    Alexis Mousset
    @amousset
    @sr57 On the Rudder server, I forgot to precise it.
    sr57
    @sr57
    @amousset Thaks! :-) <wan ip> not in allowconnects! After having putting back my laptop to the "rudder network", it connects via the internet ip and not the local ip!
    Alexis Mousset
    @amousset
    @sr57 good!
    Francois Armand
    @fanf
    @ism93 how many inventory files is there in 1/ /var/rudder/inventories/incoming and 2/ /var/rudder/inventories/accepted-nodes-updates ? Can you send us the log file /var/log/rudder/webapp/2020_02_17.stderrout.log ?
    (not in the public chan, either by email or at least in private chan)