normation/rudder

Continuous Auditing & Configuration - please speak english and be nice ;)

th333boo

@th333boo

hello people

I'm wonder how to uninstall pacakge directive from the wrong group

If anyone could help me out on that

Alexis Mousset

@amousset

il you want to cancel a directive effect the usual way is to apply a new directive that undoes what the other did

th333boo

@th333boo

will it purge

let me check

you've mentioned directive not technique right ?

pmg

@pmg7557_twitter

Good morning All, I have often and randomly(?) the error : "Status Missing red" for some node(s) , most of the time for directive "Job schdeduled" (opening time 0-23) but sometime for others (inventory for instance)
It'is ennnoying because, it arrived on nodes which shloud be 100% ok and this alert me on false pb.
What should be the cause? How can we analyse this random problem?
Rudder version : 6.02 - Protocol : Https

François Armand

@fanf

@pmg7557_twitter standard analysis process:

check report protocol used (in settings > General > [Reporting protocol]). Old syslog-based, wih UDP, will sometime loose reports because of random network problems (but UDP is necessary vs TCP to avoid global syslog contention). If possible, use new "HTTPS only" based reporting - but need all node in Rudder 6.0 at least;
if already in "HTTPS only", we will need a case by case analysis. So eatch time it happens, we need to look at the report leading to error (technical log for the faulty run). Most of the time, missing can be related to a bug in the technique (a missing report on some code path).

pmg

@pmg7557_twitter

@fanf Https only (last line of my post), in the technilcal logs, there is no value "Error" or "Missing" ! Strange?

François Armand

@fanf

oh sorry, missed that line

@pmg7557_twitter ok, strange. Can you screenshot the error message ?

pmg

@pmg7557_twitter

I"ll do it, but I saw also that my logs are full with such lines "Need to insert line ..." concerning ssh directive in audit mode, so the other logs lines can be over 1000?

@fanf I did not see how to link a file (screenshot)?

pmg

@pmg7557_twitter

On the compliance report I see :
For the node / inventory pb : None Message : empty Status in red : Missing
For the node / Job scheduled pb : /sys_maj/Commun/tmp_check Message ; [Missing report #0] Status in red : Missing

@fanf I must leave, can you tell me in what log I can see more that 1000 lines, I'll check it later. Thanks in advance

François Armand

@fanf

for screenshot, I just copy & paste them here, and gitter do some magic

cc @ncharles @amousset @peckpeck for the missing reports

Nicolas Charles

@ncharles

@pmg7557_twitter could it be possible to add a touch in your script, to check if it is effectively not run when in missing status?

could you add comment to https://issues.rudder.io/issues/18203 saying that it also happens when schedule is 0 - 23?

On one of the nodes having missing reports, do grep "jobSchedule" /var/rudder/cfengine-community/outputs show that command was executed this night?

pmg

@pmg7557_twitter

x-special/nautilus-clipboard
copy
file:///home/philippe/Bureau/Capture%20d%E2%80%99%C3%A9cran%20du%202020-10-05%2015-29-07.png

touch : done , let's wait tomorrow to see the result

Nicolas Charles

@ncharles

the image didn't do anyhing

pmg

@pmg7557_twitter

grep "jobSchedule" /var/rudder/cfengine-community/outputs/previous
2020-10-05T12:06:59+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@None@@_sys_maj_Commun_sauve_bd2_3_0_44cd6dc6_d863_456c_92b8_95619a978d61@@2020-10-05 12:06:53+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Scheduling _sys_maj_Commun_sauve_bd2_3_0_44cd6dc6_d863_456c_92b8_95619a978d61 was correct
2020-10-05T12:06:59+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@None@@_sys_maj_Commun_tmp_check_3_0_44cd6dc6_d863_456c_92b8_95619a978d61@@2020-10-05 12:06:53+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Scheduling _sys_maj_Commun_tmp_check_3_0_44cd6dc6_d863_456c_92b8_95619a978d61 was correct
2020-10-05T12:06:59+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/sauve_bd2@@2020-10-05 12:06:53+00:00##83766f80-4627-4d1c-af94-888246a2173c@#The command will be run at a random time after 00:00 on this node
2020-10-05T12:06:59+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/tmp_check@@2020-10-05 12:06:53+00:00##83766f80-4627-4d1c-af94-888246a2173c@#The command will be run at a random time after 00:00 on this node

The 2 commands seems correct but both have Missing Status

@ncharles Magic is gone?

Nicolas Charles

@ncharles

there's only log_info logs and not execution

"previous" file is only the last run

you would need to detect when result_ disappeared during last 24h

pmg

@pmg7557_twitter

https://issues.rudder.io/issues/18203 updated

@ncharles there's only log_info logs and not execution : Ok I'll try to find last execution logs

pmg

@pmg7557_twitter

/var/rudder/cfengine-community/outputs/cf_philippe31601698010_Sat_Oct3_06_06_50_2020_0x7f786a056700:2020-10-03T04:07:01+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@None@@_sys_maj_Commun_sauve_bd2_3_0_44cd6dc6_d863_456c_92b8_95619a978d61@@2020-10-03 04:06:52+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Scheduling _sys_maj_Commun_sauve_bd2_3_0_44cd6dc6_d863_456c_92b8_95619a978d61 was correct
/var/rudder/cfengine-community/outputs/cf_philippe31601698010_Sat_Oct3_06_06_50_2020_0x7f786a056700:2020-10-03T04:07:01+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@None@@_sys_maj_Commun_tmp_check_3_0_44cd6dc6_d863_456c_92b8_95619a978d61@@2020-10-03 04:06:52+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Scheduling _sys_maj_Commun_tmp_check_3_0_44cd6dc6_d863_456c_92b8_95619a978d61 was correct
/var/rudder/cfengine-community/outputs/cf_philippe31601698010_Sat_Oct3_06_06_50_2020_0x7f786a056700:2020-10-03T04:07:01+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/sauve_bd2@@2020-10-03 04:06:52+00:00##83766f80-4627-4d1c-af94-888246a2173c@#The command will be run at a random time after 00:00 on this node
/var/rudder/cfengine-community/outputs/cf_philippe31601698010_Sat_Oct3_06_06_50_2020_0x7f786a056700:2020-10-03T04:07:01+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/tmp_check@@2020-10-03 04:06:52+00:00##83766f80-4627-4d1c-af94-888246a2173c@#The command will be run at a random time after 00:00 on this node
/var/rudder/cfengine-community/outputs/cf_philippe31601784427_Sun_Oct4_06_07_07_2020_0x7f786a056700:2020-10-04T04:07:19+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@None@@_sys_maj_Commun_sauve_bd2_3_0_44cd6dc6_d863_456c_92b8_95619a978d61@@2020-10-04 04:07:10+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Scheduling _sys_maj_Commun_sauve_bd2_3_0_44cd6dc6_d863_456c_92b8_95619a978d61 was correct
/var/rudder/cfengine-community/outputs/cf_philippe31601784427_Sun_Oct4_06_07_07_2020_0x7f786a056700:2020-10-04T04:07:19+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@None@@_sys_maj_Commun_tmp_check_3_0_44cd6dc6_d863_456c_92b8_95619a978d61@@2020-10-04 04:07:10+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Scheduling _sys_maj_Commun_tmp_check_3_0_44cd6dc6_d863_456c_92b8_95619a978d61 was correct
/var/rudder/cfengine-community/outputs/cf_philippe31601784427_Sun_Oct4_06_07_07_2020_0x7f786a056700:2020-10-04T04:07:19+00:00 R: @@jobScheduler@@result_success@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/sauve_bd2@@2020-10-04 04:07:10+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Job returned a success return code after the last completed execution (/sys_maj/Commun/sauve_bd2)
/var/rudder/cfengine-community/outputs/cf_philippe31601784427_Sun_Oct4_06_07_07_2020_0x7f786a056700:2020-10-04T04:07:19+00:00 R: @@jobScheduler@@result_success@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/tmp_check@@2020-10-04 04:07:10+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Job returned a success return code after the last completed execution (/sys_maj/Commun/tmp_check)
/var/rudder/cfengine-community/outputs/cf_philippe31601784427_Sun_Oct4_06_07_07_2020_0x7f786a056700:2020-10-04T04:07:19+00:00 R: @@jobScheduler@@log_info@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/sauve_bd2@@2020-10-04 04:07:10+00:00##83766f80-4627-4d1c-af94-888246a2173c@#The command will be run at a random time after 00:00 on this node
/var/rudder/cfengine-community/outputs/cf_philippe31601784427_Sun_Oct4_06_07_07_2020_0x7f786a056700:2020-10-04T04:07:19+00:00 R: @@jobScheduler

If I understand, my jobs were correct at 6h07 yesterday, and the status is Missing now (15h54)

Nicolas Charles

@ncharles

yes, so it lost the status somewhere between 4h07 and 15h54

what's the last time you had a result_ for josScheduler ?

grep jobScheduler /var/rudder/cfengine-community/outputs/* | grep result

pmg

@pmg7557_twitter

@ncharles I saw that the last execution was yesterday!

grep jobScheduler /var/rudder/cfengine-community/outputs/* | grep result

/rudder/cfengine-community/outputs/cf_philippe31601842021_Sun_Oct4_22_07_01_2020_0x7f786a056700:2020-10-04T20:07:14+00:00 R: @@jobScheduler@@result_success@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/sauve_bd2@@2020-10-04 20:07:04+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Job returned a success return code after the last completed execution (/sys_maj/Commun/sauve_bd2)
/var/rudder/cfengine-community/outputs/cf_philippe31601842021_Sun_Oct4_22_07_01_2020_0x7f786a056700:2020-10-04T20:07:14+00:00 R: @@jobScheduler@@result_success@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/tmp_check@@2020-10-04 20:07:04+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Job returned a success return code after the last completed execution (/sys_maj/Commun/tmp_check)
/var/rudder/cfengine-community/outputs/cf_philippe31601849212_Mon_Oct5_00_06_52_2020_0x7f786a056700:2020-10-04T22:07:31+00:00 R: @@jobScheduler@@result_success@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/sauve_bd2@@2020-10-04 22:06:56+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Job returned a success return code after the last completed execution (/sys_maj/Commun/sauve_bd2)
/var/rudder/cfengine-community/outputs/cf_philippe31601849212_Mon_Oct5_00_06_52_2020_0x7f786a056700:2020-10-04T22:07:31+00:00 R: @@jobScheduler@@result_success@@32377fd7-02fd-43d0-aab7-28460a91347b@@44cd6dc6-d863-456c-92b8-95619a978d61@@0@@Job@@/sys_maj/Commun/tmp_check@@2020-10-04 22:06:56+00:00##83766f80-4627-4d1c-af94-888246a2173c@#Job returned a success return code after the last completed execution (/sys_maj/Commun/tmp_check)

Nicolas Charles

@ncharles

and you don't have it a 00:00:06 today ?

i'd be interested in the file from today at 00:00:06 (about)

pmg

@pmg7557_twitter

@ncharles
and you don't have it a 00:00:06 today ? : Not seen

cf_philippe31601849212_Mon_Oct5_00_06_52_2020_0x7f786a056700 : 2189 lines. How I send it to you?

Nicolas Charles

@ncharles

i sent you a private message with my email

Nicolas Charles

@ncharles

@pmg7557_twitter I tried to answer to your mail but got refused. The file you sent contains the result_, the file I need would rather be named Mon_Oct__5_0206 and contain run fro m2020-10-05 00:06 because of time zone

pmg

@pmg7557_twitter

@ncharles I send you the new file

pmg

@pmg7557_twitter

"Missing report" - new elt : sometimes Scheduled Jobs are not lauched?
I use the cde : grep jobScheduler /var/rudder/cfengine-community/outputs/ | grep launched | grep sauve_bd2 | cut -d' ' -f 1
*sauve_bd2 is the name og my job

Previous node (the one with missing report now) (node rebooted yesterday afternoon) :
/var/rudder/cfengine-community/outputs /cf_n_di1601426996_Wed_Sep_30_02_49_56_2020_0x7f79e7f33700:2020-09-30T00:50:10+00:00
/var/rudder/cfengine-community/outputs/cf_n_di1601513420_Thu_Oct1_02_50_20_2020_0x7f79e7f33700:2020-10-01T00:50:33+00:00
/var/rudder/cfengine-community/outputs/cf_n_di1601599788_Fri_Oct2_02_49_48_2020_0x7f79e7f33700:2020-10-02T00:50:03+00:00
/var/rudder/cfengine-community/outputs/cf_n_di1601686219_Sat_Oct3_02_50_19_2020_0x7f79e7f33700:2020-10-03T00:51:06+00:00
/var/rudder/cfengine-community/outputs/cf_n_di1601859016_Mon_Oct__5_02_50_16_2020_0x7f79e7f33700:2020-10-05T00:50:29+00:00

Node that has Missing Status yesterday
/var/rudder/cfengine-community/outputs/cf_philippe31601590052_Fri_Oct2_00_07_32_2020_0x7f786a056700:2020-10-01T22:09:21+00:00
/var/rudder/cfengine-community/outputs/cf_philippe31601762852_Sun_Oct4_00_07_32_2020_0x7f786a056700:2020-10-03T22:08:02+00:00
/var/rudder/cfengine-community/outputs/cf_philippe31601935626_Tue_Oct6_00_07_06_2020_0x7f786a056700:2020-10-05T22:07:38+00:00

Nicolas Charles

@ncharles

Thank you for the files. It seems that it simply stop to reports at 00:07. Just to be sure I understand correctly, do the next run restores the status, or is it still missing ?

Status is persisted for 1440 minutes - which is maybe too little on a non 5 minutes schedule

pmg

@pmg7557_twitter

@ncharles status, or is it still missing? Now status is Ok for Node Philippe3 but Missing for Node Di.

Where communities thrive

People

Repo info

Activity