In this project I used the K-means algorithm to cluster grocery items based on their transaction data.

Items that are often purchases together can be placed in the same aisle or aisles closer to each other, increasing sales!

This used to be done by human experts, which would require many years of experience in the industry to narrow things down. However, with the rise of big data and machine learning, why not let AI do all the trick and hard work for you?

Ok let’s get started. The first step is to gather the dataset. Here I downloaded the Instacart dataset from Kaggle, and filtered out items that don’t have a sufficient purchase history yet in the dataset (being purchased fewer than 100 times), because they may not contain enough information to be correctly classified(i.e. they may end up forming weird 1-item categories).

# Load dependencies
import pandas as pd
from scipy import sparse
from sklearn import metrics
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import seaborn as sns

pd.set_option('display.max_rows', 30)

# Load the Instacart data and cluster items that have been purchased at least 100 times
df = pd.read_csv('data/order_products__train.csv')
df = df.drop(['add_to_cart_order', 'reordered'], axis=1)
df = df.groupby('product_id').filter(lambda x:len(x) >= 100)

df_products = pd.read_csv('data/products.csv')
df_aisles = pd.read_csv('data/aisles.csv')

The main table is pretty straightforward, classical order_id - product_id pairs.

df.head()

Now let’s transform the order data into a matrix where:

• each row is an order
• each column is a product
• each value 1 indicates a purchase

df['filler'] = 1
df = df.pivot(index='order_id', columns='product_id', values='filler').fillna(0)

df.head()

print('Number of Transactions:', df.shape[0])
print('Number of Items:', df.shape[1])

Number of Transactions: 125956
Number of Items: 2457

Cool, now that everything has been set up, let’s transform the data into a similarity matrix. (here I used the cosine similarity, but other metrics should be fine).

Essentially, items that are often purchased together will have a similarity of 1, whereas those that have never been purchased together will have a similarity of 0.

# Save memories by converting it to a sparse matrix
data = df.to_numpy()
data_sparse = sparse.csr_matrix(data)
data_clustering = metrics.pairwise.cosine_similarity(data_sparse.T)
data_clustering = sparse.csr_matrix(data_clustering)

Then, time to apply the K-means algorithm using the similarity matrix we just obtained!

How do we choose K though? It’s usually more subject to the business requirements.

However, here what we can do is to pick a bunch of Ks and look at the inertia plot.

A range between 100 and 200 would probably be a good start since we don’t want to make things too complicated. (which works out to be around 10+ types of item per aisle)

cluster_inertia = []
for i in range(100, 220, 20):
    model = KMeans(n_clusters=i)
    model.fit(data_clustering)
    cluster_inertia.append(model.inertia_)

plt.plot(range(100, 220, 20), cluster_inertia)
plt.xlabel('Number of Clusters')
plt.ylabel('Inertia')
plt.show()

According to the plot, 200 seems to be a good choice since the line keeps going down.

Let’s plug it in!

kmeans = KMeans(n_clusters=200)

clusters = kmeans.fit_predict(data_clustering)
final_clusters = pd.DataFrame({'cluster':clusters,
                               'product_id':df.columns})
df_cluster = final_clusters.sort_values('cluster')
df_cluster = pd.merge(df_cluster, df_products, how='left', on='product_id')
df_cluster = df_cluster.drop(['aisle_id', 'department_id'], axis=1)

df_cluster

Get some basic stats regarding our clusters.

sns.distplot(df_cluster['cluster'].value_counts())

/Users/ree/PycharmProjects/Personal/grocery-item-clustering/venv/lib/python3.8/site-packages/seaborn/distributions.py:2619: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)





<AxesSubplot:xlabel='cluster', ylabel='Density'>

The group counts seem quite reasonable. Let’s inspect some of the categories manually!

df_cluster.query('cluster == 1')

df_cluster.query('cluster == 2')

df_cluster.query('cluster == 3')

Voila!

• For cluster 1 I cannot seem to find an obvious category, but apparently they are often bought together!
• Cluster 2 seems to be healthy/organic food
• Cluster 3 seems to be sweet, breakfast and coffee stuff