Curriculum learning improves compositionality of reinforcement learning agent across concept classes
The compositional structure afforded by language allows humans to decompose complex phrases and map them to novel visual concepts, demonstrating flexible intelligence. Although there have been several algorithms that can demonstrate compositionality, they do not give us insights on how humans learn...
Saved in:
Main Author: | |
---|---|
Other Authors: | |
Format: | Final Year Project |
Language: | English |
Published: |
Nanyang Technological University
2024
|
Subjects: | |
Online Access: | https://hdl.handle.net/10356/176294 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Nanyang Technological University |
Language: | English |
id |
sg-ntu-dr.10356-176294 |
---|---|
record_format |
dspace |
spelling |
sg-ntu-dr.10356-1762942024-05-17T15:44:45Z Curriculum learning improves compositionality of reinforcement learning agent across concept classes Lin, Zijun Wen Bihan School of Electrical and Electronic Engineering Cheston Tan bihan.wen@ntu.edu.sg Computer and Information Science Reinforcement learning The compositional structure afforded by language allows humans to decompose complex phrases and map them to novel visual concepts, demonstrating flexible intelligence. Although there have been several algorithms that can demonstrate compositionality, they do not give us insights on how humans learn to compose concept classes to ground visual cues. To study this multi-modal learning problem, we created a 3-dimensional environment, where a reinforcement learning agent has to navigate to a location specified by a natural language phrase (instruction). The instruction is composed of nouns, attributes and additionally, determiners or prepositions. This visual grounding task increases the compositional complexity for reinforcement learning agents, as navigating to the blue cubes above some red spheres will not be rewarded when the instruction is to navigate to “some blue cubes below the red sphere”. We first demonstrate that reinforcement learning agents can ground determiner concepts to visual scenes but struggle to ground the more complex preposition concepts. Secondly, we show that curriculum learning, a strategy employed by humans, improves concept learning efficiency by reducing the total number of training episodes needed to achieve a certain performance criterion by 15% in determiner environment. Moreover, it enables the agents to learn the preposition concepts. Lastly, we establish that agents trained on determiner or preposition concepts can decompose held-out test instructions, and also rapidly map their navigation policies to unseen visual object combinations. Various text encoders are also being compared to see whether they could facilitate the agents’ training. To conclude, our results clarify that multi-modal reinforcement learning agents can achieve compositional understanding of complex concept classes, and demonstrate the effectiveness of human-like learning strategies to improve the learning efficiency for artificial systems. Bachelor's degree 2024-05-15T07:20:41Z 2024-05-15T07:20:41Z 2024 Final Year Project (FYP) Lin, Z. (2024). Curriculum learning improves compositionality of reinforcement learning agent across concept classes. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/176294 https://hdl.handle.net/10356/176294 en application/pdf Nanyang Technological University |
institution |
Nanyang Technological University |
building |
NTU Library |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
NTU Library |
collection |
DR-NTU |
language |
English |
topic |
Computer and Information Science Reinforcement learning |
spellingShingle |
Computer and Information Science Reinforcement learning Lin, Zijun Curriculum learning improves compositionality of reinforcement learning agent across concept classes |
description |
The compositional structure afforded by language allows humans to decompose complex phrases and map them to novel visual concepts, demonstrating flexible intelligence. Although there have been several algorithms that can demonstrate compositionality, they do not give us insights on how humans learn to compose concept classes to ground visual cues. To study this multi-modal learning problem, we created a 3-dimensional environment, where a reinforcement learning agent has to navigate to a location specified by a natural language phrase (instruction). The instruction is composed of nouns, attributes and additionally, determiners or prepositions. This visual grounding task increases the compositional complexity for reinforcement learning agents, as navigating to the blue cubes above some red spheres will not be rewarded when the instruction is to navigate to “some blue cubes below the red sphere”. We first demonstrate that reinforcement learning agents can ground determiner concepts to visual scenes but struggle to ground the more complex preposition concepts. Secondly, we show that curriculum learning, a strategy employed by humans, improves concept learning efficiency by reducing the total number of training episodes needed to achieve a certain performance criterion by 15% in determiner environment. Moreover, it enables the agents to learn the preposition concepts. Lastly, we establish that agents trained on determiner or preposition concepts can decompose held-out test instructions, and also rapidly map their navigation policies to unseen visual object combinations. Various text encoders are also being compared to see whether they could facilitate the agents’ training. To conclude, our results clarify that multi-modal reinforcement learning agents can achieve compositional understanding of complex concept classes, and demonstrate the effectiveness of human-like learning strategies to improve the learning efficiency for artificial systems. |
author2 |
Wen Bihan |
author_facet |
Wen Bihan Lin, Zijun |
format |
Final Year Project |
author |
Lin, Zijun |
author_sort |
Lin, Zijun |
title |
Curriculum learning improves compositionality of reinforcement learning agent across concept classes |
title_short |
Curriculum learning improves compositionality of reinforcement learning agent across concept classes |
title_full |
Curriculum learning improves compositionality of reinforcement learning agent across concept classes |
title_fullStr |
Curriculum learning improves compositionality of reinforcement learning agent across concept classes |
title_full_unstemmed |
Curriculum learning improves compositionality of reinforcement learning agent across concept classes |
title_sort |
curriculum learning improves compositionality of reinforcement learning agent across concept classes |
publisher |
Nanyang Technological University |
publishDate |
2024 |
url |
https://hdl.handle.net/10356/176294 |
_version_ |
1800916403997376512 |