Exploring Explicit and Implicit Feature Spaces in Natural Language Processing Using Self-Enrichment and Vector Space Analysis
Abstract
Machine Learning in Natural Language Processing (NLP) deals directly with distributed representations of words and sentences. Words are transformed into vectors of real values, called embeddings, and used as the inputs to machine learning models. These architectures are then used to solve NLP tasks such as Sentiment Analysis and Natural Language Inference. While solving these tasks many models will create word embeddings and sentence embeddings as outputs. We are interested in how we can transform and analyze these output embeddings and modify our models, to both improve the task result and give us an understanding of the spaces. To this end we introduce the notion of explicit features, the actual values of the embeddings, and implicit features, information encoded into the space of vectors by solving the task, and hypothesis on an idealized spaces, where implicit features directly create the explicit features by means of basic linear algebra and set theory. To test if our output spaces are similar to our ideal space we vary the model and, motivated by Transformer architectures, introduce the notion of Self-Enriching layers. We also create idealized spaces, and run task experiments to see if the patterns of results can give us insight into the output spaces, as well we run transfer learning experiments to see what kinds of information are being represented by our models. Finally, we run direct analysis of the vectors of the word and sentence outputs for comparison.