Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Command pipeline not working under cron

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
rjalex
Posts: 7
Joined: 2023-05-09 11:33
Has thanked: 4 times

Command pipeline not working under cron

#1 Post by rjalex »

I am trying to automate a command line pipeline to be run daily under cron. The pipeline works well from a command prompt but does not seem to do anything when run under cron. Tried everything that I could think of but need help since I', stuck.

This is the crontab line as shown by crontab -l

Code: Select all

30 6 * * 2-7 find /sata/mongobackup -name "*memaback_archive.gz" -type f -printf "%T@\t%p\n" | sort -n | head -n -3 | cut -f 2- | xargs rm -f >> /sata/mongobackup/cron.log 2>&1
The purpose would be to keep only the last three newest backup files.

Details follow.

I have a directory /sata/mongobackup owned rwx by bob.bob which is populated by a daily backup of my mongodb by another cron job.

This is what I see this morning:

Code: Select all

bob@newisa:/sata/mongobackup$ ls -lha
total 11G
drwxrwxr-x 2 bob  bob  4.0K Jun  9 06:00 .
drwxr-xr-x 5 root root 4.0K Jun  5 17:29 ..
-rw-r--r-- 1 bob  bob  2.7G Jun  6 06:03 20230606-060001_memaback_archive.gz
-rw-r--r-- 1 bob  bob  2.7G Jun  7 06:03 20230607-060001_memaback_archive.gz
-rw-r--r-- 1 bob  bob  2.7G Jun  8 06:03 20230608-060001_memaback_archive.gz
-rw-r--r-- 1 bob  bob  2.7G Jun  9 06:03 20230609-060001_memaback_archive.gz
The command line pipeline without the last item (xargs rm -f) would produce the following:

Code: Select all

bob@newisa:/sata/mongobackup$ find /sata/mongobackup -name "*memaback_archive.gz" -type f -printf "%T@\t%p\n" | sort -n | head -n -3 | cut -f 2- 
/sata/mongobackup/20230606-060001_memaback_archive.gz
which as expected would provide the oldest file to delete to the rm command and if I run it from the command line the pipeline works just as expected.

Code: Select all

Debian 11.7
cron/stable,stable,now 3.0pl1-137 amd64 
bash/stable,stable,now 5.1-2+deb11u1 amd64 
What am I missing? Thanks in advance

User avatar
Bloom
df -h | grep > 90TiB
df -h | grep > 90TiB
Posts: 504
Joined: 2017-11-11 12:23
Been thanked: 26 times

Re: Command pipeline not working under cron

#2 Post by Bloom »

Cron expects one single command and it can't handle pipes. Use either

Code: Select all

bash -c 'find /sata/mongobackup -name "*memaback_archive.gz" -type f -printf "%T@\t%p\n" | sort -n | head -n -3 | cut -f 2- | xargs rm -f >> /sata/mongobackup/cron.log 2>&1'
or create a bash script file with this line in it. Make sure there's a she-bang line at the start

Code: Select all

#!/bin/bash
otherwise cron won't know which interpreter to use for the script. You can also start the script without she-bang if you specify the interpreter yourself:

Code: Select all

bash /path/to/your/script.sh
Note: in most environments, cron has no or very limited access to environment variables. So it can't use what is available for your normal user account. Keep that in mind. If needed, there are ways to set environment variables for cron.

rjalex
Posts: 7
Joined: 2023-05-09 11:33
Has thanked: 4 times

Re: Command pipeline not working under cron

#3 Post by rjalex »

Very interesting thanks!

The following works (and logs coorrectly) though:

Code: Select all

0 6 * * 2-7 /bin/bash /home/bob/mongobackup.sh >> /sata/mongobackup/cron.log 2>&1
so if I understand what you told me it's because it's just one command (bash) with an argument (and then the redirect of STDOUT and STDERR to a file) ?

If I did understand well my pipeline should be single comma delimited (because there are double commas inside), right? Would it be better to place the closing single comma after the xargs rm -f command and leave the redirect outside it?

I am aware of the different env in which cron starts :)

Take care

User avatar
Bloom
df -h | grep > 90TiB
df -h | grep > 90TiB
Posts: 504
Joined: 2017-11-11 12:23
Been thanked: 26 times

Re: Command pipeline not working under cron

#4 Post by Bloom »

The redirect is for the whole command starting with and following 'find', so I would leave it inside the quotes.
Starting a .sh with bash should allow a redirect, but I usually do that inside the .sh just to be sure it all works as expected. Cron is a funny beast, sometimes.

rjalex
Posts: 7
Joined: 2023-05-09 11:33
Has thanked: 4 times

Re: Command pipeline not working under cron

#5 Post by rjalex »

I will now get my first tattoo "Cron doesn't like pipes" ;)

Thanks a bunch for spotting the problem, I was really struggling in the dark!

Now all works well!

FWIW this is the final form of the crontab lines:

Code: Select all

bob@newisa:~$ crontab -l
0 6 * * 2-7 /bin/bash /home/bob/code/mema/cron/mongobackup.sh >> /home/bob/code/mema/cron/cron.log 2>&1
15 6 * * 2-7 /bin/bash /home/bob/code/mema/cron/prunebacks.sh  >> /home/bob/code/mema/cron/cron.log 2>&1
30 16 * * 2-7 /bin/bash /home/bob/code/mema/cron/runme.sh >> /home/bob/code/mema/cron/cron.log 2>&1
This is the format of one of the three bash script files:

Code: Select all

bob@newisa:~$ cat /home/bob/code/mema/cron/runme.sh
#!/bin/bash
cd /home/bob/code/mema
/home/bob/.local/bin/poetry run python  /home/bob/code/mema/src/jobs/01_get_wp_issues.py
/home/bob/.local/bin/poetry run python  /home/bob/code/mema/src/jobs/03_fetch_missing_wp_articles_numdays.py 1
/home/bob/.local/bin/poetry run python  /home/bob/code/mema/src/jobs/05_mema_link_coref.py
/home/bob/.local/bin/poetry run python  /home/bob/code/mema/src/jobs/07_reconcile.py
and these are the rights for each script:

Code: Select all

bob@newisa:~$ ls -la /home/bob/code/mema/cron/runme.sh
-rwxr-xr-x 1 bob bob 411 Jun 10 13:31 /home/bob/code/mema/cron/runme.sh

User avatar
kent_dorfman766
Posts: 535
Joined: 2022-12-16 06:34
Location: socialist states of america
Has thanked: 57 times
Been thanked: 70 times

Re: Command pipeline not working under cron

#6 Post by kent_dorfman766 »

FWIW
I always prefer to execute piped cron command in a subshell using () syntax as
(command | pipe1 < input)
or similar

Post Reply