Vinicius Avellar Moreira
Nominated Award: Best Application of AI In a Student Project
Linedin of Person: https://www.linkedin.com/in/amvinicius/
Student Profile: Vinicius is a self-driven, enthusiastic and well-accomplished software developer with 10+ years experience. He gained solid background through completing a B.Sc. in Computer Science from FAESA – Faculdades Integradas Espírito-Santenses – Vitória – ES – Brazil – https://www.faesa.br/. Since completing his degree Vinicius has had the opportunity to work w ith several different technologies, recently focusing on Angular, TypeScript, Spring Boot and Java. Thanks to familiarity with Object-Oriented Programming and Design Patterns, Vinicius has been able to write efficient, maintainable and testable code. Vinicius also has experience with JavaScript, TypeScript, Java, Kotlin, HTML, CSS, Oracle PL/SQL, PHP, C++, bash, and loves to try and learn new ones.
Vinicius has been living in Ireland for more than four years now and is a Senior Software Engineer at Fidelity Investments.
In November 2020, Vinicius joined our Certificate in Foundations of Artificial Intelligence which Krisolis deliver in partnership with Women in AI, TU Dublin and Technology Ireland ICT Skillnet. The programme is certified by TU Dublin and participants earn a 10 ECTS CPD Award at NFQ Level 8. The programme is delivered over 12 weeks with 5 hours of class each week and students: • Learn about AI and its real-world applications
• Learn to Program in Python
• Develop machine learning solutions
• Create effective data visualisations
• Become versed in the ethical considerations of AI
• Apply all the above in your own project supported by 121 mentoring
Vinicius came to the programme with a passing knowledge of AI and no experience with Python. Over the 12-week programme Vinicius exceeded all expectations of what we expected from a student. Not only did he excel on the core material such as python, machine learning and data visualisation he also gained experience in NLP and Reinforcement Learning. And it was the topic of reinforcement learning that really peaked Vinicius interest and when choosing his project on week 3 he chooses a project that would involve reinforcement learning. Although it was not a topic covered in the programme Vinicius was happy to work independently under the mentorship of Dr. Brian Mac Namee and went on to implement a reinforcement learning solution. Given Vinicius starting point in terms of knowledge the learning progressing that he showed throughout the program was exceptional and was best demonstrated in the project that he completed as part of the programme.
It is for this project that we are recommending Vinicius for the Application of AI in a Student Project.
Reason for Nomination:
Here we will give details of the project:
Project Objective:
This project’s objective is to utilize AI to create a bot or “agent” capable of playing a card game called Bisca, without using any coded rules or historical data. The bot will be trained against a simple rule-based player and it is expected to perform well once playing against a human opponent.
Bisca (https://en.wikipedia.org/wiki/Bisca_(card_game) is a turn-based card game, where the main objective is to finish a match with more points than your opponent. Rules are noted below, but some modifications will be made to ease the learning process.
Solution:
Considering that there is no historic data available for this game online, Reinforcement Learning was used to train the agent by interacting with a coded game environment and p laying against a simple rule-based opponent.
The solution used in this project was based on the Monte Carlo Control algorithm, described by Ri chard S Sutton & Andrew G. Barto. 2018. Reinforcement Learning: An Introduction, Second Edition. MIT Press, Cambridge. This algorithm was chosen because it is easy to implement and understand. Considering the complexity of the game environment, some rules were removed or changed to make training and test simpler as it will require less processing: The game trump will be fixed as 2 of clubs. Additional rules noted in the game description will not be considered. The game engine, the agent and the training algorithm were all coded in python, using numpy and pandas as support libraries.
Data Sample: For Reinforcement Learning problems, state modelling is one of the most important steps in preparation for training, for this project the game trump, table cards and player cards were used to compose the state. Each card the agent has in hand for a specific state will be considered an action it can take. Considering the full deck, 44 cards and the state definition described above, there are 3,494,610 possible states to be considered for training.
The model was trained and the first thing noticed during development was that the randomness of the game makes it difficult to train the agent for every situation. The conditions for training change on each episode and states are different after each interaction, there is no clear transition from one state to another. This happens because the first step in the game is to shuffle the deck so that all players have an equal chance to win. Because of that randomness, training the agent on every possible state will also require a great number of episodes. It was considered generating all possible permutations of the cards deck, but it would require hours of processing, as there are 2.433E+18 different permutations for a deck containing 20 cards. Without generating permutations beforehand, it is most likely that not all states will be visited during training and will not have an optimal action value.
Even with that condition, the agent is capable of learning and win most of the time when tested outside the learning environment. When the training started with the agent winning less than 45% of the games, and after 20,000 episodes it had already increased by 20%. At the end of the training, the winning rate was close to 70% of the games.
Conclusions:
The agent performs well against the rule -based player, but that is not enough for it to perform equally well against human players. The game itself does not have a clear state sequence since the cards are shuffled at the start and it is not possible to know what card the opponent will play next. For that reason, the training strategy used considers the rewards that the agent gets at each turn, each state, more than the result at the end of the game, so the AI will learn the best action s for each state, but it does not necessarily mean that it will pick the best actions to win the game. In most cases, it wins against the same rule-based, but when playing against a human opponent it will face scenarios that were not visited before.
Regardless of the results, the AI can still be fun to play against when implemented as an opponent in the game, it will not perform incredibly well, but will pose some challenge to a human player.
Additional Information:
I hope the excerpts from Vinicius’s project submissions give you a flavour for the exceptional level of skill that Vinicius obtained in our short 12 programme. The programme was designed to introduce students to the fundamentals of AI with the hope that after 12 week they be able to implement some basic machine learning models, never did we expect that a student with no python skills and no previous machine learning knowledge would reach the skill level displayed by Vinicius. I would love to tell you that it was obviously our wonderful teaching that made the difference but the reality is that Vinicius worked incredibly hard and in the most part independently to deliver a wonderful project that is truely deserving of your consideration to be shortlisted for this award.