Command line interface

SherlockML has a command line client, sml. This lets you:

  • create, list, and terminate servers;
  • access your servers via SSH;
  • transfer large files using appropriate protocols like scp and rsync.

sml is designed to be used from a terminal on your local machine. It will work on macOS, Linux and Windows.

Installation on MacOS and Linux

You can install it using pip:

$ pip install -U sml

If you don’t have pip, follow these instructions. If you get an error along the lines of Permission denied on macOS or Linux, you may need to re-run this command with sudo.

Once you have done this, continue to the Initializing sml section.

Installation on Windows

The recommended way to use sml on Windows is via Cygwin. This has only been tested on Windows 10.

Make sure you have an installation of Cygwin with the following packages. Installing new packages normally involves restarting the Cygwin installer.

  • python3 (in the Interpreters, Python category)
  • openssh (in the Net category)
  • rsync (in the Net category)

In the Cygwin terminal, run:

$ python3 -m ensurepip
$ pip3 install -U sml

Once you have done this, continue to the Initializing sml section.

Initializing sml

The first time you use sml, you will need to log in with a client ID and secret. This is different from your usual SherlockML username and password. To create client credentials, navigate to the My Account page in SherlockML by clicking the icon showing your initial in the lower left of SherlockML, and click Generate SML credentials.

../_images/sml_credentials.png

Once you’ve created your credentials, run the following and enter the credentials when prompted:

$ sml login

Once sml is configured, you can use it to create a new server on a project, sync data up to a project, and connect to a server from a terminal on your local machine.

To see the full list of commands run:

$ sml --help

While most of the SherlockML functionality is accessible through the UI, there are some things that you can only do through the command line (for instance, bulk uploading code to your workspace).

List projects and servers

You can list your SherlockML projects with:

$ sml projects

and you can list a project’s running servers with:

$ sml server list <project>

Note

sml server list, like many other sml subcommands, takes a project as an argument. For projects with a unique name, you can simply give the project name, however for projects with non-unique names, you must pass the project ID. Project IDs are discoverable using sml projects -v.

SSH into a server

Having successfully listed your servers, you can get SSH access using:

$ sml shell <project> <server>

You can also provide sml shell with arbitrary extra arguments, which will be passed through to ssh. For example, to set up an SSH tunnel to an arbitrary network port on the server:

$ sml shell <project> <server> -L 9000:localhost:8888

When you SSH into a server, you will be in the /home/sherlock directory. You probably want to access your project files first. These are located in /project. You can jump to that directory with:

$ cd /project

Note

As with projects, sml shell and other commands that take a server as an argument can be passed either the server name or server ID. If you have servers with non-unique names, use sml server list <project> -v to discover their IDs for use in sml shell.

Create a new server

You can create new servers for a project using:

$ sml server new <options> <project>

For example:

$ sml server new --cores=8 --memory=16 --name=new-server user-playground

Full options can be listed with sml server new --help.

Note

If the name of the project is not unique among projects you have access to, you must pass the project ID rather than the project name. Project IDs are discoverable using sml projects -v.

Open the web interface of a server in your browser

You can open the web interface of a server in your browser with:

$ sml server open <project>

sml will open the web interface of one of the running servers in the project. If you wish to open the interface of a specific server, pass the server’s name or ID with --server <server>.

Note

sml server open should be run from a terminal on your local machine, not from a SherlockML server.

Copying files to and from your workspace

You can also use sml to upload and download files to and from your project on the command line. To upload a file, use sml file put:

$ sml file put <project> <local-source> <remote-destination>

Note

sml file put should be run from a terminal on your local machine, not from a SherlockML server.

Similarly, to download a file, use sml file get:

$ sml file get <project> <remote-source> <local-destination>

Note

sml file get should be run from a terminal on your local machine, not from a SherlockML server.

These commands will copy files from the source path provided to the destination path.

You need to have a server running on the project to upload files. If you have multiple servers, you can specify which server the files are routed through with the --server option.

Note

The remote path in sml file put and sml file get will be relative to the Unix home directory (/home/sherlock) on the SherlockML server. To put files directly in the project workspace, you will want to use a remote path under /project.

sml file put and sml file get use the scp command internally, and any additional command line arguments provided on the command line will be passed through to scp. Advanced users may find this useful to customise the behaviour of sml file put and sml file get.

Uploading large files to SherlockML

When uploading large files, we recommend using sml file sync-up, as this supports resuming the upload process if your connection drops. To start a new upload with resume support, use the following command:

$ sml file sync-up <project> path/on/local/machine path/on/sherlockml --partial --progress

To resume an unfinished upload:

$ sml file sync-up <project> path/on/local/machine path/on/sherlockml --append --progress

Note

sml file sync-up should be run from a terminal on your local machine, not from a SherlockML server.

Uploading and downloading a large number of files to and from SherlockML

If you have an entire directory of files to upload to SherlockML, you can do so with sml file sync-up. To upload a directory:

$ sml file sync-up <project> path/on/localmachine/ path/on/sherlockml

To download an entire directory from SherlockML to your local machine use sml file sync-down:

$ sml file sync-down <project> path/on/sherlockml/ path/on/localmachine

You can also provide sml file sync-down and sml file sync-up with arbitrary extra arguments, which will be passed through to the rsync program. For example, to upload a directory with compression and delete files in the directory on SherlockML not present on your local machine, use:

$ sml file sync-up <project> path/on/localmachine/ path/on/sherlockml --compress --delete

Note

Users unfamiliar with the rsync program, and especially the meaning of trailing slashes on the local and remote arguments, and the meaning of the --delete option, are strongly recommended to read its documentation before using sml file sync-down or sml file sync-up. Incorrect use of these features may result in data loss.

Server environments

Server Environments can be created in SherlockML and applied through the web interface or through sml. The sml environment subcommands provide the ability to list and apply environments in a project, and to view the results of applying environments.

To list the environments in a project, run:

$ sml environment list <project>

This will list all environments in the specified project.

You can apply one of these environments to an existing server using the sml environment apply command:

$ sml environment apply <project> <server> <environment>

As with other commands, each of the project, server and environment can be a name or an ID. To get the ID of server environments in a project, you can use the -v option with sml environment list.

The status of the last environment applied to a server can be checked with the sml environment status command:

$ sml environment status <project> <server>

This will summarise the steps of the environment being applied, along with the overall status of the execution. To stream the logs of the environment application as it’s being applied, use the sml environment logs command:

$ sml environment logs <project> <server>

You can also specify a particular step to stream the logs of using the --step command line option. This accepts the index of the step as given by sml environment status.