AI UniCOOK

UniCOOK

ADAPT Centre TCD

Nominated Award:
Best Application of AI in an Academic Research Body

Website of Company (or Linkedin profile of Person):
www.uni-cook.com

 

UniCOOK – Cooking up Visuals
 
We use machine learning to transform digital recipes into a format understood by all.
 
The UniCOOK Artificial Intelligence engine uses machine learning to analyse digital recipes, automatically building a storyboard designed to be inclusive for all. UniCOOK also automates the linking of ingredient terms to products in a retailer’s database easing the process of automatic shopping list population and online shopping integration, resulting in greater sales.
 
The start-up was born out of the ADAPT centre in Trinity College and brings together the multiple thematic areas covered by the centre. These include natural language processing, image processing, multimodal transformation and ethical considerations. Our goal was to take a multi-disciplinary approach and build something truly disruptive both societally and technologically.
 
Our first use case was Atul, an Indian living in Dublin. Atul discovered that he loved Irish stew and thought that his mother would too. He found a great recipe but it was in English. Atul’s mother can’t speak English. He wonders, “What if I could send that recipe to my mother in India and she could know exactly how to cook it without translation?” And that was how the idea of the storyboard was born!
 
We quickly realised at UniCOOK that the manual curation of recipes to storyboard is a very lengthy, time consuming and expensive process. If we could automate all of the curation process or most of it, we would produce a useful product.
 
The process of curation requires taking relatively unstructured text and making sense of it by automatically identifying and tagging all of the important entities in the text. Once an entity is identified, it becomes part of the knowledge graph needed to create the storyboard. All of this content transformation can be utilised or sold on. For instance, using only the first 2 of our 5 AI technologies we can provide needed solutions for supermarkets – the transformation of recipe webpages into a shopping basket is something that many supermarkets wish to do.
 
Any recipe can be automatically transformed into a storyboard using our pipeline of different AI technologies. The first deep-network determines the location of ingredients, utensils, method, descriptions and titles from unstructured text and classifies each of them. The output from this is an ingredient list that populates the supermarket’s online shopping basket.
 
Using the second deep-network the classified sentences are split up into their constituent parts to obtain the measurements, verbs, quantity types, amounts and ingredients. The directions/method of the recipe. These are then further analysed by a relational entity deep-network to produce a sequential set of instructions that can be presented to the user as a storyboard with/without a mixture of text.

Reason for Nomination:

The state of play                                                                                                                                                                          1 Million + recipes are in existence, tabulated and sorted on a Database called 1M+.

Retailers are looking at ways to increase online shopping.

Consumers are looking for more exciting meals.

Food waste contributes greatly to the world’s CO2 emmissions.

The problem
It is extremely time consuming for a supermarket to add recipe ingredients to a customer’s online shopping basket manually. It is commercially incumbent on them to do so though. Using shoppable recipes turnover increases by between 3 and 5%.

We supply a full life cycle platform from home to shop to plate. The customer of a supermarket can use any recipes to create a shopping basket. Using our meal planning product these recipes can be combined to produce a more sustainable shopping list (reduced wastage). Once delivered the customer prepares the meals using an automatically generated storyboard with a reduced cognitive load.

The solution
The UniCOOK Artificial Intelligence engine uses machine learning to analyse digital recipes, automatically building a storyboard designed to be inclusive for all. UniCOOK also automates the linking of ingredient terms to products in a retailer’s database easing the process of automatic shopping list population and online shopping integration, resulting in greater sales.

How did we do it?
Everything is built on deep neural networks with a small amount of engineering to increase percentage performance of the deep-nets. Any exceptional cases found, are factored back into the dataset to produce a balanced retraining. Sentence types are predicted with 99.5% reliability. The ingredients, method, measurements and so forth are predicted with 98.6% reliability. The relationship between different AIs is under development. The images are being gathered to match the ingredient entities. The mappings to build the storyboards will be produced automatically as knowledge graphs and processed to produce the final results.

Which models?
We used a mixture of transformer models, recurrent neural networks and image processing libraries. The personalised user models will be deep-nets. No rule based systems or standard older search engines will be used by the technology.

How difficult was it?
The relational entity model is very difficult and taking some time to create pristine data for. With very clean data however the first models are running way beyond state of the art. Finding the best models did take some time to find. We have used over 10 models to obtain the best 2 models at the beginning of our pipeline.

Our Challenges?
One major challenge was simply how to present the recipe to the user. This is ongoing by doing continuous and iterative user experience design and testing. If this is the top down approach for our solution then the machine learning is a bottom-up approach that meets the UX in the middle. We have worked with cookery/culinary schools, dyslexic groups, nutritionists and supermarket chains to arrive at best and most intuitive layout.

Curating newly found recipes to extract ingredients and populate shopping baskets is time consuming and expensive. UniCOOK does all of this automatically using only the first two machine learning techniques in our full pipeline.

The Results
The results so far are at a combined 99%+ result, but we can do better using some software engineering to inform us based on the sequential nature of sentences and paragraph structures.

The Impact:

Economic impact:
Already just using the first two AI models combined together with some software engineering we are in a position to produce a fully automated pipeline for recipe to shopping basket generation including layouts that match the requirements of a supermarket chain. There is an exact match between our problem statement and our solution. In the first demonstrations of the technology we could conclusively demonstrate that the AI was doing all the “heavy lifting” for us that would otherwise need to be done by the employees thus saving valuable resources.

Societal impact:
If food waste were a country it would produce the 3rd highest rate of greenhouse gas emissions. There are two main ways to avoid this, buy in such a way that you reduce waste and secondly store the food in a way that it lasts its longest. The first item we are immediately addressing with our meal planning offering.

Until recently recipes have only appeared in books and on websites in a static form. UniCOOK is able to render them in a way that differently abled people can understand. For example; people with intellectual disabilities, dyslexics, people with failing eyesight and people for whom English is their second language.

Technological/Scientific impact:
We pride ourselves on building hybrid systems that to our knowledge have not existed before. In the area of food research many have moved on to research what is on a plate or in a fridge based on image recognition. However, this is inhibited by occlusion and a large amount of inference. We are taking concrete machine learning across multiple disciplines to produce a robust hybrid solution that will work. With 55 years of research between our two main researchers we are convinced that we can produce something truly disruptive. We have a number of competitors in the food and drink industry but are encouraged that none of them are approaching the problem in the same way.

Additional Information:

We have trail partners in:
Two cookery schools
A large supermarket chain
A nutritionist
A healthy-eating website
A large food image database company

And are pursuing an image based curation process (by-product of our work) with:         A national newspaper

We believe that this technology could be of use to a publisher online/offline in illustrating articles that are profiling multiple people/things. For example the contenders for the seats in a constituency or the players in an upcoming sports competition.

Our technology can extract the names and automatically select an image from the normal photo libraries/databases that the publisher uses. These images could be selected for the layout for either an online/offline publication or could be hyperlinks in an online publication.

We believe that this could be developed as an extension to page layout software that could dramatically increase the efficiency of the graphic designers working for these publishing companies.

At the end of the pipeline we plan to build out micro and macro nutrient information associated with our recipes. This will enable us to produce personalised meal plans for a wide cohort of people; athletes, people with specific dietary needs; Health Issues, Coeliacs, Vegetarians/Vegans and people with ethical issues around what they eat.

The first 3 AI technologies give us an educational platform for dyslexic, aging and visually impaired people. Our image processing AI gives us faster ways of cleaning our images such as background removal and generative adversarial networks (GAN), and then the 5th AI technique can give us a fully personalised platform to shop, eat healthily, etc. Of the multiple strands that our AI technologies can be applied to, we are addressing what we believe are the largest markets first. With the plan eventually to have something that is completely visual and of great educational value that produces appealing visual content whilst providing the written accompanying text, if needed.

Cookbook hold a vast amount of intellectual property in recipes. UniCOOK unlocks this value by automating the process of presenting these recipes to a new audience. To our knowledge nobody is capable of automatically transforming these books into a digital, visual and highly-configurable representation. We believe that our AI products will be able to give publishers new outlets for their as yet to be digitised intellectual property.

Our ultimate long-term goal is to solve what we call the “Boeing problem”. This takes an aircraft manual and an exploded 3D model of an aircraft and automatically renders the steps needed to be taken in order to perform required scheduled maintenance tasks. Show, in 3D, how to fix parts of the plane, by automatically rendering the tool connection to the aircraft parts. While solving this problem is difficult it provides us with the template needed for all applications. There are a number of other related applications to be investigated, anything that resembles a ‘structured recipe’ in a domain with a well-defined taxonomy.


Files:

https://unicook.adaptcentre.ie/all-demos

https://aiawards.ie/wp-content/uploads/ninja-forms/4/UniCOOK-1-Pager-1.pdf