Scheduling jobs with cron¶
On SherlockML servers, you can configure scripts to run on a regular schedule using the standard Cron utility. For example, if you have a Python script which scrapes data from a website and saves it to a CSV file in your workspace, you can use cron to run the script automatically every morning at 7 a.m., so the new data is ready when you get to your desk.
Cron is configured using a plain text file called a
crontab. To edit your
crontab in your default editor, run:
Each job is defined on a single line in the crontab file. The line must start with five numbers separated by spaces, which denote the minute, hour, day, month of the year, and day of the week the job should be run on. An asterisk for any field indicates the job should be run on every value of that field. crontab.guru is a useful tool that makes it easier to design your crontab rule.
The command to be run should follow the time fields. For example, the following
line in your crontab will run
/usr/bin/true at 4.20 p.m. every Monday:
20 16 * * 1 /opt/anaconda/envs/Python3/bin/python /project/script.py
Cron jobs don’t necessarily run in the directory where you ran
crontab -e to create the cron task. As in the above example, you should
use absolute paths both in the crontab entry and in any paths inside your
script. If you are unsure of the absolute path for an executable like
which <executable> in the terminal.
To list your existing cron jobs, use
Using cron on SherlockML servers¶
By default, Cron provides a more restricted environment than is available to a script run directly from a shell. Scripts run in Cron are not able to find programs installed into custom paths, such as Conda environments, and don’t have access to your SherlockML credentials, for example to create new reports or use sml.
Therefore, SherlockML servers contain a tool called
sml-cron which will run
a program from cron with the same environment as if you ran the script from a
shell. Programs run via
sml-cron will have the default Conda environment
activated, and access to your SherlockML credentials. Simply prefix the normal
command in your crontab file with
sml-cron. For example, to run a Python
script in your project workspace at 7 a.m. every morning, use:
0 7 * * * sml-cron /opt/anaconda/envs/Python3/bin/python /project/example.py