Text processing using neural networks

Yujia Sun

Doctoral thesis

Abstract:

Natural language processing is a key technology in the field of artificial intelligence. It involves the two basic tasks of natural language understanding and natural language generation. The primary core of solving the above tasks is to obtain text semantics. Text semantic analysis enables computers to simulate humans to understand the deep semantics of natural language and identify the true meaning contained in information by building a model. Obtaining the true semantics of text helps to improve the processing effect of various natural language processing downstream tasks, such as machine translation, question answering systems, and chatbots. Natural language text is composed of words, sentences and paragraphs (in that order). Word-level semantic analysis is concerned with the sense of words, the quality of which directly affects the quality of subsequent text semantics at each level. Sentences are the simplest sequence of semantic units, and sentence-level semantics analysis focuses on the semantics expressed by the entire sentence. Paragraph semantic analysis achieves the purpose of understanding paragraph semantics. Currently, while the performance of semantic analysis models based on Deep Neural Network has made significant progress, many shortcomings remain. This thesis proposes the Deep Neural Network-based model for sentence semantic understanding, word sense understanding and text sequence generation from the perspective of different research tasks to address the difficulties in text semantic analysis. The research contents and contributions are summarized as follows: First, the mainstream use of recurrent neural networks cannot directly model the latent structural information of sentences. To better determine the sense of ambiguous words, this thesis proposes a model that uses a two-layer bi-directional long short-term memory neural network and attention mechanism. Second, static word embedding models cannot manage polysemy. Contextual word embedding models can do so, however, their performance is limited in application scenarios with high real-time requirements. Accordingly, this thesis proposes using a word sense induction task to construct word sense embeddings for polysemous words. Third, the current mainstream encoder-decoder model based on the attention mechanism does not explicitly perform a preliminary screening of the information in the source text before summary generation. This results in the input to the decoder containing a large amount of information irrelevant to summary generation as well as exposure bias and out-of-vocabulary words in the generation of sequences. To address this problem, this thesis proposes an abstractive text summarization model based on a hierarchical attention mechanism and multi-objective reinforcement learning. In summary, this thesis conducts in-depth research on semantic analysis, and proposes solutions to problems in word sense disambiguation, word sense embeddings, and abstractive text summarization tasks. The feasibility and validity were verified through extensive experiments on their respective corresponding publicly-available standard datasets, and also provide support for other related research in the field of natural language processing.

Abstract:

Natural language processing is a key technology in the field of artificial intelligence. It involves the two basic tasks of natural language understanding and natural language generation. The primary core of solving the above tasks is to obtain text semantics. Text semantic analysis enables computers to simulate humans to understand the deep semantics of natural language and identify the true meaning contained in information by building a model. Obtaining the true semantics of text helps to improve the processing effect of various natural language processing downstream tasks, such as machine translation, question answering systems, and chatbots. Natural language text is composed of words, sentences and paragraphs (in that order). Word-level semantic analysis is concerned with the sense of words, the quality of which directly affects the quality of subsequent text semantics at each level. Sentences are the simplest sequence of semantic units, and sentence-level semantics analysis focuses on the semantics expressed by the entire sentence. Paragraph semantic analysis achieves the purpose of understanding paragraph semantics. Currently, while the performance of semantic analysis models based on Deep Neural Network has made significant progress, many shortcomings remain. This thesis proposes the Deep Neural Network-based model for sentence semantic understanding, word sense understanding and text sequence generation from the perspective of different research tasks to address the difficulties in text semantic analysis. The research contents and contributions are summarized as follows: First, the mainstream use of recurrent neural networks cannot directly model the latent structural information of sentences. To better determine the sense of ambiguous words, this thesis proposes a model that uses a two-layer bi-directional long short-term memory neural network and attention mechanism. Second, static word embedding models cannot manage polysemy. Contextual word embedding models can do so, however, their performance is limited in application scenarios with high real-time requirements. Accordingly, this thesis proposes using a word sense induction task to construct word sense embeddings for polysemous words. Third, the current mainstream encoder-decoder model based on the attention mechanism does not explicitly perform a preliminary screening of the information in the source text before summary generation. This results in the input to the decoder containing a large amount of information irrelevant to summary generation as well as exposure bias and out-of-vocabulary words in the generation of sequences. To address this problem, this thesis proposes an abstractive text summarization model based on a hierarchical attention mechanism and multi-objective reinforcement learning. In summary, this thesis conducts in-depth research on semantic analysis, and proposes solutions to problems in word sense disambiguation, word sense embeddings, and abstractive text summarization tasks. The feasibility and validity were verified through extensive experiments on their respective corresponding publicly-available standard datasets, and also provide support for other related research in the field of natural language processing.

Keywords

Deep Neural Network, Natural Language Processing, Semantic Analysis, Semantic Understanding, Text Generation

Keywords

Deep Neural Network, Natural Language Processing, Semantic Analysis, Semantic Understanding, Text Generation

Language used: English

Date on which the thesis was submitted / produced: 11. 7. 2023

Thesis defence

• Date of defence: 9. 1. 2024
• Supervisor: Jan Platoš
• Reader: Pavel Kordík, Zuzana Komínková Opatková, Jiří Dvorský

VSB – Technical University of Ostrava

Fakulta elektrotechniky a informatiky

Doctoral programme / field:
Informatika, komunikační technologie a aplikovaná matematika / Informatika