AI Shopbox Startup

Through ongoing discussions with retailers, we uncovered some common issues with existing e-commerce recommender systems, namely:
•Expensive and time consuming implementations,
•Large amounts of manual works to implemented and maintain,
•Difficulty in scaling to large product catalogues,
•Lacking ability to provide quality recommendations with little data.

The core AI technology developed by Shopox is a number of recommender systems. We are constantly refining them and adding new features as we receive feedback. We have primarily developed hybrid recommender systems, which can utilise user interaction and product data. These systems are capable of:
•Providing high quality product recommendations,
•Providing context specific recommendations depending on the user’s history and the current page they are viewing,
•Being tolerant of noisy and missing data,
•Responding quickly to changes in product and behavioural data,
•Being flexible enough to accommodate a variety of ecommerce platforms,
•Operating across any retail domain,
•Accommodating specific recommendation constraints for example based on brands or stock levels,
•Scaling effectively as new retailers are onboarded.

We offer product recommendations in a number of ways depending on the user’s context. One form of recommendations are provided using the current product being viewed. These are recommendations based on “similar products” and “frequently bought with this product”. Another system recommends products based on a user’s viewing activity. We also provide Amazon style homepage grids which recommend products based on many criteria including a user’s viewing history, trending products and top ordered products. To create the product recommendations we use product data, order data and user interaction data. Recommendations are formed by first constructing a series of similarity matrices based on specific fields in the data. For example, product descriptions are cleaned, tokenized, and vectorized, after which a similarity metric is applied to create a product by product similarity matrix.

Another example of a similarity matrix we produce uses domain specific taxonomies to identify keywords which are important to particular domains. When two products use the same taxonomic phrases they will score higher in that taxonomic similarity matrix. The similarity matrices are combined in a weighted way to produce an overall similarity matrix. We then perform a ranking on a per product basis and write the resulting product matches to a data store. The manner in which matrices are weighted enables high levels of flexibility in adapting to the volume and quality of a customer’s data. If a customer does not have high quality product category data, but has high quality description data, we can weight the description such that it has higher importance. Furthermore, this system enables us to easily incorporate new NLP techniques. If we identified a suitable transformer model to create document embeddings, we could construct a similarity matrix and combine this with the existing ones. We experimented with many different approaches before settling on this current configuration, which has proven to provide high quality recommendations in a flexible and scalable way.

The main programming language used is Python. For data preprocessing and cleaning we use beautifulsoup and NLTK. Matrices are built and manipulated using scipy and numpy. Many of the operations performed with these libraries can take advantage of highly optimised code.

The model creation process runs at least once per day per customer. Our ETL processes and recommender models are deployed using Google Cloud to spin up virtual machines which run Docker containers. Docker images are built every time we push to the Git repository. This setup of continuous integration and continuous delivery, enables reliable and rapid development.

Product data is stored in an SQL database, while user interaction data is stored in BigQuery. BigQuery can query high volumes of data exceptionally fast, which is ideal for the large volume of user interaction data.

Some of the main challenges originated from the ETL (Extract Transform Load) process which had to accommodate a wide variety of data formats and qualities. This involved creating a standard extraction procedure, which is extended if required for clients. Another issue we encountered was that the similarity matrices can become very large, and initially some virtual machines ran out of memory. We solved this by performing batch computations of portions of the similarity matrices and writing out recommendations to a datastore as they become available. Our modelling system has scaled to operate with product catalogues sizes greater than 200,000 items, which gives us a competitive advantage.

Our system enables fast and accurate model development, producing high quality recommendations from the moment we go live with a customer. For e-commerce platforms with consistent data feeds, we can complete data ingestion and model development in approximately an hour. This sharply contrasts our competitors who can take months to complete software implementations for recommender systems. As more data is gathered through user interactions, our recommendations become better, and these are automatically applied during the daily model building processes. The modelling system has scaled well, enabling the rebuilding of 55 models per day.

In the following, we are referring to Shopbox users as those people who use our recommendations, and non-Shopbox users as people who do not use our recommendations. To date we have seen success across many metrics including:
•Sales Conversion – We see a 100% to 200% increase in conversion for Shopbox over non Shopbox users.
•Average Order Value – We see a 10% to 45% increase in order values with Shopbox users.
•Repeat buyers and returning users – Shopbox users represent an increase of 20% in repeat buyers.
•More time spent on site – Shopbox users spend on average 400% longer on clients’ websites.
•Reduced bounce rate – For one large customer, Shopbox users had a 17% bounce rate, while non-Shopbox had 57%

In summary, Shopbox is helping retailers’ to be competitive with large ecommerce giants. Our competitive advantages are: speed of implementation, quality of recommendations both from the point of going live and over time, and our capacity to handle large volumes of product and customer data. We enable a more engaging customer experience, facilitating a more effective search of a retailer’s product space.