[Solved] Cron jobs run twice e.g., at 6:25 again at 7:35

Kernels & Hardware, configuring network, installing services

[Solved] Cron jobs run twice e.g., at 6:25 again at 7:35

Postby w4kh » 2020-01-13 19:07

Recently (not sure exactly when this started) cron jobs started running twice. I was on 9.x and had no issues, but upgraded to Buster (10) and had no issue (yet) eventually upgraded to 10.1 and then 10.2 and the problem seemed to start.
At first, cron.hourly (which has ONLY the .placeholder ) was erroring and sending an email telling me that it could not find root, which was in the user field of /etc/crontab... reading man pages later, I tried `cp -p /etc/crontab /etc/old.crontab` and then `crontab -e` and removed root from user field... I checked /var/spool/cron/crontabs/root and after the edit to remove "root" from the entry, I found:
Code: Select all
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.cwzBzw/crontab installed on Thu Jan  9 08:28:57 2020)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
17 *   * * *   cd / && run-parts --report /etc/cron.hourly
25 6   * * *   cd / && run-parts --report /etc/cron.daily
47 6   * * 7   cd / && run-parts --report /etc/cron.weekly
52 6   1 * *   cd / && run-parts --report /etc/cron.monthly
#
1 0    * * 3   cp -p /etc/wed_motd /etc/motd
1 0    * * 0-2,4-6 cp -p /etc/old_motd /etc/motd
#
20 22 5,9,13,17,22,27 * * /usr/local/bin/B4BU
32 22 5,9,13,17,22,27 * * /usr/local/bin/runBU
2 0  1 * * /usr/local/bin/L_B4BU
12 0 1 * * /usr/local/bin/L_runBU
#

After exiting the edit of crontab, I found that jobs were being run twice! Particularly a problem for the two jobs " /usr/local/bin/B4BU" and " /usr/local/bin/L_B4BU" which are supposed to "trim" (delete) older backups to make room for newer "runBU" (partclone.* and dd" backups of filesystems 1, 2, and 4... the "L_" versions are for capturing the end of month backups... running these twice deletes one of the backup generations intended to be kept. And, running the jobs in cron.[daily|hourly|weekly|monthly] means logwatch, rkhunter, tripwire and others get run twice in close succession with odd results.

I tried removing crontab, but cron goes nuts every minute and will soon overflow log files:
Code: Select all
Jan 13 12:56:01 DebServ cron[854]: (CRON) STAT FAILED (/etc/crontab)
Jan 13 12:57:01 DebServ cron[854]: (CRON) STAT FAILED (/etc/crontab)


And worse... cron jobs still are getting run twice!

Others have suggested rebooting, but the runBU jobs do reboot the machine so that filesystem SDA2 is backed up clean. I have exhausted my thinking and need some new ideas for getting the twice running jobs to get pared down to running once. A final thought: the machine is on 24/7, so anacron really isn't needed. What REALLY is the impact of purging the anacron package?
Last edited by w4kh on 2020-02-20 18:22, edited 1 time in total.
4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11)
MB: Gigabyte 970A-D3P
CPU: AMD FX-8350 @4000.000 MHz cache: 2048 KB
RAM: 32GB (4x8GB) Unbuffered/Unregistered
LTO-5 SAS Tape on LSI SAS9211 controller
Video: GeForce 8400 GS to VIZIO E320VA
User avatar
w4kh
 
Posts: 89
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: Cron jobs running twice e.g., once at 6:25 and again at

Postby arzgi » 2020-01-14 11:16

w4kh wrote:A final thought: the machine is on 24/7, so anacron really isn't needed. What REALLY is the impact of purging the anacron package?

Code: Select all
$ aptitude show anacron
... clip..
 
 This package is pre-configured to execute the daily jobs of the Debian system.  You
 should install this program if your system isn't powered on 24 hours a day to make sure
 the maintenance jobs of other Debian packages are executed each day.
Homepage: http://sourceforge.net/projects/anacron/
Merkit: admin::automation, implemented-in::c, interface::daemon, role::program,
        scope::utility


If you don't need, just remove.
arzgi
 
Posts: 533
Joined: 2008-02-21 17:03
Location: Finland

Re: Cron jobs running twice e.g., once at 6:25 and again at

Postby w4kh » 2020-01-15 13:08

arzgi is confirming what I have seen elsewhere, so I will purge the anacron package and see what happens... The machine is on 24/7/365 - a server - so anacron is not truly needed...
I will do the purge and report back.
Now, my curiosity is crying for more information... I didn't just add anacron when I updated to Buster, it had been installed for some time... So what triggered the problem of having anacron and cron running jobs twice? It only started recently, sometime during the 10.1 or 10.2 updates.
4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11)
MB: Gigabyte 970A-D3P
CPU: AMD FX-8350 @4000.000 MHz cache: 2048 KB
RAM: 32GB (4x8GB) Unbuffered/Unregistered
LTO-5 SAS Tape on LSI SAS9211 controller
Video: GeForce 8400 GS to VIZIO E320VA
User avatar
w4kh
 
Posts: 89
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: Cron jobs running twice e.g., once at 6:25 and again at

Postby arzgi » 2020-01-15 13:20

w4kh wrote:arzgi is confirming what I have seen elsewhere, so I will purge the anacron package and see what happens... The machine is on 24/7/365 - a server - so anacron is not truly needed...
I will do the purge and report back.
Now, my curiosity is crying for more information... I didn't just add anacron when I updated to Buster, it had been installed for some time... So what triggered the problem of having anacron and cron running jobs twice? It only started recently, sometime during the 10.1 or 10.2 updates.


From my experience, I'd say apt is quite stable, and if you don't mix foreign stuff, keeps system sane.
But if you still are unsure, check first /etc/anacrontab, comment out the commands.
arzgi
 
Posts: 533
Joined: 2008-02-21 17:03
Location: Finland

Re: Cron jobs running twice e.g., once at 6:25 and again at

Postby neuraleskimo » 2020-01-15 13:46

w4kh wrote:Now, my curiosity is crying for more information... I didn't just add anacron when I updated to Buster, it had been installed for some time... So what triggered the problem of having anacron and cron running jobs twice? It only started recently, sometime during the 10.1 or 10.2 updates.

Take a look at /etc/crontab. What time is /etc/cron.daily set to run?
User avatar
neuraleskimo
 
Posts: 142
Joined: 2019-03-12 23:26
Location: Bloomington, Indiana, USA

Re: Cron jobs running twice e.g., once at 6:25 and again at

Postby w4kh » 2020-01-16 16:07

This is really odd...
Yesterday I removed anacron ("apt-get remove anacron") and found the anacron configuration files and removed each after checking to see that there was only anacron settings included. I thought I had found the path out of the woods..

This morning, however, I had TWO logwatch runs, both started at 6:25... That was startling.
Not one to give up easily, I set about finding (and removing any residue of anacron:
Code: Select all
root@Deb10:/# apt-get purge anacron
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  anacron*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
(Reading database ... 218789 files and directories currently installed.)
Purging configuration files for anacron (2.3-28) ...
Processing triggers for systemd (241-7~deb10u2) ...
[ Rootkit Hunter version 1.4.6 ]
File updated: searched for 181 files, found 147


This seemed to mean that some files might still be hanging around... And I remembered having pre-pended "0" to the anacron tasks in the /etc/cron.* directories, so I set out to find them all:
Code: Select all
root@Deb10:/# find . -depth -iname *anacron*
./home2/etc-usr/.cron.monthly/0anacron
./home2/etc-usr/.cron.weekly/0anacron
./home2/etc-usr/.cron.d/anacron
./home2/etc-usr/.cron.daily/0anacron
./var/lib/systemd/timers/stamp-anacron.timer
./var/cache/apt/archives/anacron_2.3-28_amd64.deb
./usr/share/augeas/lenses/dist/tests/test_anacron.aug
./usr/share/augeas/lenses/dist/anacron.aug
./usr/share/doc/cron/README.anacron


and then deleted them all:
Code: Select all
root@Deb10:/# rm /home2/etc-usr/.cron.monthly/0anacron
root@Deb10:/# rm /home2/etc-usr/.cron.weekly/0anacron
root@Deb10:/# rm /home2/etc-usr/.cron.daily/0anacron
root@Deb10:/# rm /home2/etc-usr/.cron.d/anacron
root@Deb10:/# rm /usr/share/doc/cron/README.anacron
root@Deb10:/# rm /usr/share/augeas/lenses/dist/anacron.aug
root@Deb10:/# rm /usr/share/augeas/lenses/dist/tests/test_anacron.aug
root@Deb10:/# rm /var/cache/apt/archives/anacron_2.3-28_amd64.deb
root@Deb10:/# rm /var/lib/systemd/timers/stamp-anacron.timer


Now I wait to see if anything remains... "Film at 11" tomorrow?
4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11)
MB: Gigabyte 970A-D3P
CPU: AMD FX-8350 @4000.000 MHz cache: 2048 KB
RAM: 32GB (4x8GB) Unbuffered/Unregistered
LTO-5 SAS Tape on LSI SAS9211 controller
Video: GeForce 8400 GS to VIZIO E320VA
User avatar
w4kh
 
Posts: 89
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: Cron jobs running twice e.g., once at 6:25 and again at

Postby w4kh » 2020-01-17 20:52

A tad late on my "Film at 11" but I had two runs this morning of logwatch, so I dug in deeper.
First, I edited ( crontab -e ) and remove any reference to anacron, including the "test -x .... ||" statements.
Next I commented the command lines in /etc/crontab, because /var/log/syslog was showing two executions, /etc/crontab and var/spool/cron/crontabs/root (shown here)
Code: Select all
root@Deb10:~# cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# # m h dom mon dow user        command
# 17 *  * * *   root    cd / && run-parts --report /etc/cron.hourly
# 25 6  * * *   root    cd / && run-parts --report /etc/cron.daily
# 47 6  * * 7   root    cd / && run-parts --report /etc/cron.weekly
# 52 6  1 * *   root    cd / && run-parts --report /etc/cron.monthly
# #
# 1 0 * * wed root cp -p /etc/wed_motd /etc/motd
# 1 0 * * 0-2,4-6 root cp -p /etc/old_motd /etc/motd
# #
# 20 22 5,9,13,17,22,27 * * root /usr/local/bin/B4BU
# 32 22 5,9,13,17,22,27 * * root /usr/local/bin/runBU
# 2 0 1 * * root /usr/local/bin/L_B4BU
# 12 0 1 * * root /usr/local/bin/L_runBU
# #
# # 2 0 10 * * root /usr/local/bin/TARclone

and var/spool/cron/crontabs/root
Code: Select all
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.cwzBzw/crontab installed on Thu Jan  9 08:28:57 2020)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
17 *   * * *   cd / && run-parts --report /etc/cron.hourly
25 6   * * *   cd / && run-parts --report /etc/cron.daily
47 6   * * 7   cd / && run-parts --report /etc/cron.weekly
52 6   1 * *   cd / && run-parts --report /etc/cron.monthly
#
1 0    * * 3   cp -p /etc/wed_motd /etc/motd
1 0    * * 0-2,4-6 cp -p /etc/old_motd /etc/motd
#
20 22 5,9,13,17,22,27 * * /usr/local/bin/B4BU
32 22 5,9,13,17,22,27 * * /usr/local/bin/runBU
2 0  1 * * /usr/local/bin/L_B4BU
12 0 1 * * /usr/local/bin/L_runBU
#


and, two hours later I checked /var/log/syslog and found:
Code: Select all
Jan 17 07:17:01 Deb10 CRON[17340]: (root) CMD (   test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.hourly ))
Jan 17 07:17:01 Deb10 CRON[17341]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 08:17:01 Deb10 CRON[18038]: (root) CMD (   test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.hourly ))
Jan 17 08:17:01 Deb10 CRON[18039]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 09:17:03 Deb10 CRON[18254]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 10:17:06 Deb10 CRON[18376]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 11:17:01 Deb10 CRON[18484]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 12:17:01 Deb10 CRON[18671]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 13:17:01 Deb10 CRON[25641]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 14:17:01 Deb10 CRON[28607]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jan 17 15:17:01 Deb10 CRON[328]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)


So it would appear that between 08:17 and 09:17, when I made the changes described above, I have eliminated the dual cron tasks... more Film if anything changes
4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11)
MB: Gigabyte 970A-D3P
CPU: AMD FX-8350 @4000.000 MHz cache: 2048 KB
RAM: 32GB (4x8GB) Unbuffered/Unregistered
LTO-5 SAS Tape on LSI SAS9211 controller
Video: GeForce 8400 GS to VIZIO E320VA
User avatar
w4kh
 
Posts: 89
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: Cron jobs running twice e.g., once at 6:25 and again at

Postby w4kh » 2020-01-20 23:59

I hesitate to mark this "SOLVED" because I'll need to keep an eye on things daily to make sure cron tasks are run once, when they are supposed to be run.

That said, I
Code: Select all
apt-get purge anacron

And then I manually cleaned up all the residue, including the anacron tasks that were in each of the /etc/cron.[hourly|daily|weekly|monthly|d] -- to make my task more difficult (and perhaps the apt-get purge command would have removed these without my having to, I don't know and am not curious enough to mess up my system to find out) I had pre-pended "0" to each (anacron became 0anacron), to make sure it was the first set of tasks run under each heading... now everything is running as I want, one set of cron tasks...
4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11)
MB: Gigabyte 970A-D3P
CPU: AMD FX-8350 @4000.000 MHz cache: 2048 KB
RAM: 32GB (4x8GB) Unbuffered/Unregistered
LTO-5 SAS Tape on LSI SAS9211 controller
Video: GeForce 8400 GS to VIZIO E320VA
User avatar
w4kh
 
Posts: 89
Joined: 2006-09-09 19:10
Location: Tennessee, USA

[Solved] Cron jobs running twice e.g., once at 6:25 - again

Postby w4kh » 2020-02-20 18:17

It's been a month, and no issues, so I marked it as solved...

That said, having things changes so dramatically between upgrades to Buster 10.1 (no issue) and 10.2 (suddenly anacron and crontab EACH ran) still has my curiosity up:
What changed between 10.1 and 10.2 that change the cron behavior? My inquiring mind still needs to know! :?
4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11)
MB: Gigabyte 970A-D3P
CPU: AMD FX-8350 @4000.000 MHz cache: 2048 KB
RAM: 32GB (4x8GB) Unbuffered/Unregistered
LTO-5 SAS Tape on LSI SAS9211 controller
Video: GeForce 8400 GS to VIZIO E320VA
User avatar
w4kh
 
Posts: 89
Joined: 2006-09-09 19:10
Location: Tennessee, USA

Re: [Solved] Cron jobs run twice e.g., at 6:25 again at 7:35

Postby Deb-fan » 2020-02-20 22:15

Not even remotely a cron guru type. Thought that occurs in this though is making sure service daemons are properly re/load-started but when comes to upgrading between major releases I've always abided the recommendation of doing fresh and clean vs in place. Just clearly has to carry the risk of all manner of odd borkage and config conflicts. Talking all of once every 3 or so years. Tend to do minimal netinstalls so yeah it's a pita but know what I end up with is consistent with how the latest Debian stable is intended to be. Until I start apt-pinning anyway. :P Don't even have anacron installed on Buster.

PS, and yeah tend towards using apt purge in particular when wanting to irradicate anything which believe may cause some odd config conflict but still tend to encounter directories and what not throughout the system outside of users home in reference to packages I've purged and so expected would have been removed. Are no doubt approaches/tools for keeping better track and more thoroughly cleaning such chuff. Have read about such but atm don't recall the specifics involved. A system that's been running continuously for years, through major revisions is bound to accumulate more than it's share of chuff. So I heed the clean install thing. It's no doubt considered best practice with reason.
Deb-fan
 
Posts: 526
Joined: 2012-08-14 12:27


Return to System configuration

Who is online

Users browsing this forum: No registered users and 13 guests

fashionable