Customising your environment¶
Your home directory¶
Computer aficionados, data scientists and data engineers like customising their
environment. We get it. You’ve spent years crafting the perfect
file. Your local home directory is a trove of configuration files for your
shell, editor, and other development tools. Why would you give that up?
You don’t have to: while servers on SherlockML are ephemeral, your home directory persists between servers and between projects. That means you can use it to store your configuration files, and share your shell history across all your servers.
Custom server environments¶
If you’ve ever found yourself needing to install an obscure visualisation library or data analysis tool to work on a project, you’ll know how time consuming and repetitive this can sometimes be. Even once you’ve set everything up, replicating this on a colleague’s machine can take as much time again.
To solve this, SherlockML provides custom server environments. Custom environments let you define sets of Python packages to be installed with conda and pip, and system packages to be installed with APT, all from your browser. When you create a new server, choose an environment to apply and when the server starts you’ll find all the packages you specified already installed. Even better: any of the team members in your project can also choose to apply your environment to their server, meaning you’ll never need to email them a list of arcane shell commands to run.
For advanced users, in addition to Python and system packages, custom
environments also let you specify Bash scripts to be run when your
environment is applied to a server, allowing arbitrary customisation. Note that
Bash scripts are executed in a randomly named temporary directory, so your
script should use absolute paths or explicitly call
cd if you require the
commands to be executed in a specific directory.
Creating a custom environment¶
Let’s walk through the example of creating a custom server environment to install the Theano and Keras machine learning libraries. To create a custom environment, go to the Environments tab and click to add your first environment.
In the modal that appears, enter a name for your environment and click CREATE ENVIRONMENT.
You’ll be taken to a page showing the environment specification, where you can expand the Python section. We choose to install Theano using Conda, and Keras using Pip, each for both Python 2 and Python 3.
The environment is automatically saved when you leave the page, and is now listed in the Environments tab.
Choosing a custom environment when creating a server¶
We can now use our new custom environment when creating a new server. Navigate to the Workspace tab and click NEW and then Server. In the modal that appears, choose the environment we just created from the ENVIRONMENT dropdown menu.
Click CREATE SERVER. Once your server has started, the packages we specified will be installed automatically. We can check this by opening a new notebook and importing them.
Community contributed environments¶
If you’ve created a custom environment that other SherlockML users can use, why not add it to the repository of community contributed environments? The documentation in the repository explains how to contribute your environment, and how to use a community environment in your own SherlockML project.