plot_diff()
The function plot_diff() explores the difference of column distributions and statistics across multiple datasets.
Next, we demonstrate the functionality of plot_diff()
dataprep.eda supports Pandas and Dask dataframes. Here, we will load the house prices datasets for both training and testing into a Pandas dataframe.
dataprep.eda
[1]:
from dataprep.datasets import load_dataset import numpy as np df1 = load_dataset("house_prices_train") df1 = df1.replace(" ?", np.NaN) df2 = load_dataset("house_prices_test") df2 = df2.replace(" ?", np.NaN)
plot_diff([df1, df2])
We start by calling plot_diff([df1, df2]) which computes dataset-level statistics, a histogram for each numerical column, and a bar chart for each categorical column across two dataframes. The number of bins in the histogram can be specified with the parameter bins, and the number of categories in the bar chart can be specified with the parameter ngroups. If a column contains missing values, the percent of missing values is shown in the title and ignored when generating the plots.
bins
ngroups
[2]:
from dataprep.eda import plot_diff plot_diff([df1, df2])
Sometimes we want to give our datasets some better names, this can be specified with the parameter diff.label.
diff.label
[3]:
plot_diff([df1, df2], config={"diff.label": ["train", "test"]})
By default, we use the first dataset as our baseline to compute the distributions and statistics. If this baseline is not properly set, we can specify this parameter with diff.baseline.
diff.baseline
The baseline starts with index 0 instead of 1 which is in the default label parameter.
0
1
[4]:
plot_diff([df1, df2], config={"diff.baseline": 1})
By default, we will show a comparison of histogram for a numerical column. You can change it to a density plot using diff.density parameter.
diff.density
[5]:
plot_diff([df1, df2], config = {"diff.density": True})