The action of reading, understanding and combining words to create meaningful phrases comes naturally to most people. Still, the processes that govern semantic composition in the human brain are not well understood. In this thesis, we explore semantics (word meaning) and semantic composition (combining the meaning of multiple words) using two data sources: a large text corpus, and brain recordings of people reading adjective noun phrases. We show that these two very different data sources are both consistent, in that they contain overlapping information, and are complementary, in that they contain non-overlapping, but still congruent information. These disparate data sources can be used together to further the study of semantics and semantic composition as grounded in the brain, or more abstractly as represented in patterns of word usage in corpora.

This thesis is supported by three experiments. Firstly, we extend a matrix factorization algorithm to learn an interpretable semantic space that respects the composition of words into noun phrases. We use the interpretability of the model to explore semantic composition as captured in the statistics of word usage in a large text corpus. Secondly, we build a joint model of semantics in corpora and the brain, which fuses brain imaging data with corpus data into one model of semantics. When compared to models that use only a single data source, we find that this joint model excels at a variety of tasks, from matching human judgements of semantics to predicting words from brain activity.

Thirdly, we explore semantic composition in the brain through a new brain image dataset, collected with Magnetoencephalography while subjects read adjective-noun phrases. We learn several functions of the brain data that are capable of predicting semantic properties of the adjective, noun, and phrase. From the performance of these functions, we build a theory for the semantic composition of adjective noun phrases in the brain. This thesis asks a fundamentally different question than those asked in previous studies of adjective noun composition: where in the brain and when in time is phrasal meaning located? The answer to this question paints a unique picture of composition in the brain that is congruent with previous findings, but also sheds new light onto the neural processes governing semantic composition.

Together, these contributions show that brain imaging data and corpus data can be used in concert to build better models of semantics. These more successful models provide a new understanding semantic composition, both in the brain and in a more abstract sense. Furthermore, this thesis demonstrates how machine learning techniques can be used to analyze and understand complicated data, like the neural activity captured in brain images.

116 pages

Thesis Committee:
Tom Mitchell (Chair)
Marcel Just
Byron Yu
Mirella Lapata (University of Edinburgh)

Tom M. Mitchell, Head, Machine Learning Department
Andrew W. Moore, Dean, School of Computer Science