TY  - JOUR
AU  - Hourrane, Oumaima 
AU  - Benlahmar, El Habib 
PY  - 2022
TI  - Topic-Transformer for Document-Level Language Understanding
JF  - Journal of Computer Science
VL  - 18
IS  - 1
DO  - 10.3844/jcssp.2022.18.25
UR  - https://thescipub.com/abstract/jcssp.2022.18.25
AB  - As long as natural language processing applications are considered prediction problems with insufficient context, usually referred to as a single sentence or paragraph, this does not reveal how humans perceive natural language. When reading a text, humans are sensitive to much more context, such as the rest or other relevant documents. This study focuses on simultaneously capturing syntax and global semantics from a text, thus acquiring document-level understanding. Accordingly, we introduce a Topic-Transformer that combines the benefits of a neural topic model that captures global semantic information and a transformer-based language model, which can capture the local structure of texts both semantically and syntactically. Experiments on various datasets confirm that our model has a lower perplexity metric compared to standard transformer architecture and the recent topic-guided language models and generates topics that are conceivably coherent compared to those of regular Latent Dirichlet Allocation (LDA) topic model.