Lens Tutorial

Lens is a library for exploring data in Pandas DataFrames. It computes single column summary statistics and estimates the correlation between columns.

We wrote Lens when we realised that the initial steps of acquiring a new dataset were almost formulaic: what data type is in this column? How many null values are there? Which columns are correlated? What's the distribution of this value? Lens calculates all this for you, and provides convenient visualisation of this information.

You can use Lens to analyse new datasets as well as using it to compare how DataFrames change over time.

Using lens

To start using Lens you need to import the library:

In [1]:
import lens

Lens has two key functions; lens.summarise for generating a Lens Summary from a DataFrame and lens.explore for visualising the results of a summary.

For this tutorial we are going to use Lens to analyse the Room Occupancy dataset provided in the Machine Learning Repository of UC Irvine. It includes ambient information about a room such as Temperature, Humidity, Light, CO2 and whether it was occupied. The goal is to predict occupancy based on the room measurements.

To read it into pandas use:

In [2]:
import pandas as pd
df = pd.read_csv('http://asi-datasets.s3.amazonaws.com/room_occupancy/room_occupancy.csv')

# Split a numerical variable to have additional categorical variables
df['Humidity_cat'] = pd.cut(df['Humidity'], 5,
                            labels=['low', 'medium-low', 'medium',
                                    'medium-high', 'high']).astype('str')
In [3]:
print('Number of rows in dataset: {}'.format(len(df.index)))
df.head()
Number of rows in dataset: 8143
Out[3]:
date Temperature Humidity Light CO2 HumidityRatio Occupancy Humidity_cat
1 2015-02-04 17:51:00 23.18 27.2720 426.0 721.25 0.004793 1 medium
2 2015-02-04 17:51:59 23.15 27.2675 429.5 714.00 0.004783 1 medium
3 2015-02-04 17:53:00 23.15 27.2450 426.0 713.50 0.004779 1 medium
4 2015-02-04 17:54:00 23.15 27.2000 426.0 708.25 0.004772 1 medium
5 2015-02-04 17:55:00 23.10 27.2000 426.0 704.50 0.004757 1 medium

Creating the summary

When you have a DataFrame that you'd like to analyse the first thing to do is to create a Lens Summary object.

In [4]:
ls = lens.summarise(df, scheduler='sync')

The summarise function takes a DataFrame and returns a Lens Summary object. The time this takes to run is dependent on both the number of rows and the number of columns in the DataFrame. It will use all cores available on the machine, so you might want to use a SherlockML instance with more cores to speed up the computation of the summary. There are additional optional parameters that can be passed in. Details of these can be found in the summarise API docs.

Given that creating the summary is computationally intensive, Lens provides a way to save this summary to a JSON file on disk and recover a saved summary through the to_json and from_json methods of lens.summary. This allows to store it for future analysis or to share it with collaborators:

In [5]:
# Saving to JSON
ls.to_json('room_occupancy_lens_summary.json')

# Reading from a file
ls_from_json = lens.Summary.from_json('room_occupancy_lens_summary.json')

The LensSummary object contains the information computed from the dataset and provides methods to access both column-wise and whole dataset information. It is designed to be used programatically, and information about the methods can be accessed in the LensSummary API docs.

In [6]:
print(ls.columns)
['date', 'Temperature', 'Humidity', 'Light', 'CO2', 'HumidityRatio', 'Occupancy', 'Humidity_cat']

Create explorer

Lens provides a function that converts a Lens Summary into an Explorer object. This can be used to see the summary information in tabular form and to display plots.

In [7]:
explorer = lens.explore(ls)

Coming back to our room occupancy dataset, the first thing that we'd like to know is a high-level overview of the data.

Describe

To show a general description of the DataFrame call the describe function. This is similar to Pandas' DataFrame.describe but also shows information for non-numeric columns.

In [8]:
explorer.describe()
Out[8]:
dateTemperatureHumidityLightCO2HumidityRatioOccupancyHumidity_cat
descNonenumericnumericnumericnumericnumericcategoricalcategorical
dtypeobjectfloat64float64float64float64float64int64object
notnulls81438143814381438143814381438143
nulls00000000
unique814326513258892282358325

We can see that our dataset has 8143 rows and all the rows are complete. This is a very clean dataset! It also tells us the columns and their types, including a desc field that explains how Lens will treat this column.

Column details

To see type-specific column details, use the column_details method. Used on a numeric column such as Temperature, it provides summary statistics for the data in that column, including minimun, maximum, mean, median, and standard deviation.

In [9]:
explorer.column_details('Temperature')
Out[9]:
Temperature
descnumeric
dtypefloat64
min19.0
max23.18
mean20.6190836403
median20.39
std1.01691644111
sum167901.198083
IQR1.69

We saw in the ouput of explorer.describe() that Occupancy, our target variable, is a categorical column with two unique values. With explorer.column_details we can obtain a frequency table for these two categories - empty (0) or occupied (1):

In [10]:
explorer.column_details('Occupancy')
Out[10]:

desc: categorical, dtype: int64

itemfrequency
06414
11729

Correlation

As a first step in exploring the relationships between the columns we can look at the correlation coefficients. explorer.correlation() returns a Spearman rank-order correlation coefficient matrix in tabular form.

In [11]:
explorer.correlation()
Out[11]:
HumidityHumidityRatioTemperatureCO2LightOccupancy
Humidity1.00.9403266291569079-0.193389233341625470.223518899301086680.00706530868193160940.1292350876247768
HumidityRatio0.94032662915690791.00.104765098214187470.37835692291681820.169057127901900970.25583595029547784
Temperature-0.193389233341625470.104765098214187471.00.63690608170847720.56523725594606390.5328303325204367
CO20.223518899301086680.37835692291681820.63690608170847721.00.47310301262234730.6566512850978677
Light0.00706530868193160940.169057127901900970.56523725594606390.47310301262234731.00.8046454034169337
Occupancy0.12923508762477680.255835950295477840.53283033252043670.65665128509786770.80464540341693371.0

However, parsing a correlation table becomes difficult when there are many columns in the dataset. To get a better overview, we can plot the correlation matrix as a heatmap, which immediately highlights a group of columns correlated with Occupancy: Temperature, Light, and CO2.

In [12]:
explorer.correlation_plot()