1. Saha, A., Gittens, A., Ni, J., Hassanzadeh, O., Yener, B., & Srinivas, K. (2022). SPOCK @ Causal News Corpus 2022: Cause-Effect-Signal Span Detection Using Span-Based and Sequence Tagging Models. Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Socio-Political Events from Text (CASE), 133–137.
   PDF BIB ABSTRACT

   @inproceedings{saha2022spock2,
     abbr = {CASE2022},
     title = {SPOCK @ Causal News Corpus 2022: Cause-Effect-Signal Span Detection Using Span-Based and Sequence Tagging Models},
     author = {Saha, Anik and Gittens, Alex and Ni, Jian and Hassanzadeh, Oktie and Yener, Bulent and Srinivas, Kavitha},
     booktitle = {Proceedings of the 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE)},
     pages = {133--137},
     file = {saha2022spock2.pdf},
     year = {2022}
   }
   

   Understanding causal relationship is an importance part of natural language processing. We address the causal information extraction problem with different neural models built on top of pre-trained transformer-based language models for identifying Cause, Effect and Signal spans, from news data sets. We use the Causal News Corpus subtask 2 training data set to train span-based and sequence tagging models. Our span-based model based on pre-trained BERT base weights achieves an F1 score of 47.48 on the test set with an accuracy score of 36.87 and obtained 3rd place in the Causal News Corpus 2022 shared task.

2. Saha, A., Ni, J., Hassanzadeh, O., Gittens, A., Srinivas, K., & Yener, B. (2022). SPOCK at FinCausal 2022: Causal Information Extraction Using Span-Based and Sequence Tagging Models. Proceedings of the 4th Financial Narrative Processing Workshop@ LREC2022, 108–111.
   PDF BIB ABSTRACT

   @inproceedings{saha2022spock,
     abbr = {FinCausal2022},
     title = {SPOCK at FinCausal 2022: Causal Information Extraction Using Span-Based and Sequence Tagging Models},
     author = {Saha, Anik and Ni, Jian and Hassanzadeh, Oktie and Gittens, Alex and Srinivas, Kavitha and Yener, Bulent},
     booktitle = {Proceedings of the 4th Financial Narrative Processing Workshop@ LREC2022},
     pages = {108--111},
     file = {saha2022spock.pdf},
     year = {2022}
   }
   

   Causal information extraction is an important task in natural language processing, particularly in finance domain. In this work, we develop several information extraction models using pre-trained transformer-based language models for identifying cause and effect text spans from financial documents. We use FinCausal 2021 and 2022 data sets to train span-based and sequence tagging models. Our ensemble of sequence tagging models based on the RoBERTa-Large pre-trained language model achieves an F1 score of 94.70 with Exact Match score of 85.85 and obtains the 1st place in the FinCausal 2022 competition.

3. Saha, A., Finegan-Dollak, C., & Verma, A. (2021). Position Masking for Improved Layout-Aware Document Understanding. Document Intelligence Workshop at KDD 2021.
   PDF BIB ABSTRACT

   @inproceedings{saha2021position,
     abbr = {DI@KDD},
     title = {Position Masking for Improved Layout-Aware Document Understanding},
     author = {Saha, Anik and Finegan-Dollak, Catherine and Verma, Ashish},
     booktitle = {Document Intelligence Workshop at KDD 2021},
     arxiv = {2109.00442},
     pdf = {https://document-intelligence.github.io/DI-2021/files/di-2021_final_21.pdf},
     slides = {https://document-intelligence.github.io/DI-2021/files/di-2021_slides_21.pptx},
     video = {https://youtu.be/2XqrIHexte0},
     file = {saha2021position.pdf},
     year = {2021}
   }
   

   Natural language processing for document scans and PDFs has the potential to enormously improve the efficiency of business processes. Layout-aware word embeddings such as LayoutLM have shown promise for classification of and information extraction from such documents. This paper proposes a new pre-training task called position masking that can improve performance of layout-aware word embeddings that incorporate 2-D position embeddings. We compare models pre-trained with only language masking against models pre-trained with both language masking and position masking, and we find that position masking improves performance by over 5% on a form understanding task.

Preprints

1. Saha, A., Hassanzadeh, O., Gittens, A., Ni, J., Srinivas, K., & Yener, B. (2023). A Cross-Domain Evaluation of Approaches for Causal Knowledge Extraction. In arXiv preprint arXiv:2308.03891.
   PDF BIB ABSTRACT

   @unpublished{saha2023crossdomain,
     title = {A Cross-Domain Evaluation of Approaches for Causal Knowledge Extraction},
     author = {Saha, Anik and Hassanzadeh, Oktie and Gittens, Alex and Ni, Jian and Srinivas, Kavitha and Yener, Bulent},
     journal = {arXiv preprint arXiv:2308.03891},
     year = {2023},
     eprint = {2308.03891},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
     file = {saha2023crossdomain.pdf}
   }
   

   Causal knowledge extraction is the task of extracting relevant causes and effects from text by detecting the causal relation. Although this task is important for language understanding and knowledge discovery, recent works in this domain have largely focused on binary classification of a text segment as causal or non-causal. In this regard, we perform a thorough analysis of three sequence tagging models for causal knowledge extraction and compare it with a span based approach to causality extraction. Our experiments show that embeddings from pre-trained language models (e.g. BERT) provide a significant performance boost on this task compared to previous state-of-the-art models with complex architectures. We observe that span based models perform better than simple sequence tagging models based on BERT across all 4 data sets from diverse domains with different types of cause-effect phrases.

2. Saha, A., Hassanzadeh, O., Gittens, A., Ni, J., Srinivas, K., & Yener, B. (2023). Improving Neural Ranking Models with Traditional IR Methods. In arXiv preprint arXiv:2308.15027.
   PDF BIB ABSTRACT

   @unpublished{saha2023improving,
     title = {Improving Neural Ranking Models with Traditional IR Methods},
     author = {Saha, Anik and Hassanzadeh, Oktie and Gittens, Alex and Ni, Jian and Srinivas, Kavitha and Yener, Bulent},
     journal = {arXiv preprint arXiv:2308.15027},
     year = {2023},
     eprint = {2308.15027},
     archiveprefix = {arXiv},
     primaryclass = {cs.IR},
     file = {saha2023improving.pdf}
   }
   

   Neural ranking methods based on large transformer models have recently gained significant attention in the information retrieval community, and have been adopted by major commercial solutions. Nevertheless, they are computationally expensive to create, and require a great deal of labeled data for specialized corpora. In this paper, we explore a low resource alternative which is a bag-of-embedding model for document retrieval and find that it is competitive with large transformer models fine tuned on information retrieval tasks. Our results show that a simple combination of TF-IDF, a traditional keyword matching method, with a shallow embedding model provides a low cost path to compete well with the performance of complex neural ranking models on 3 datasets. Furthermore, adding TF-IDF measures improves the performance of large-scale fine tuned models on these tasks.

3. Saha, A., Gittens, A., & Yener, B. (2023). Word Sense Induction with Knowledge Distillation from BERT. In arXiv preprint arXiv:2109.00442.
   PDF BIB ABSTRACT

   @unpublished{saha2023word,
     title = {Word Sense Induction with Knowledge Distillation from BERT},
     author = {Saha, Anik and Gittens, Alex and Yener, Bulent},
     journal = {arXiv preprint arXiv:2109.00442},
     year = {2023},
     eprint = {2304.10642},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
     file = {saha2023word.pdf}
   }
   

   Pre-trained contextual language models are ubiquitously employed for language understanding tasks, but are unsuitable for resource-constrained systems. Noncontextual word embeddings are an efficient alternative in these settings. Such methods typically use one vector to encode multiple different meanings of a word, and incur errors due to polysemy. This paper proposes a two-stage method to distill multiple word senses from a pre-trained language model (BERT) by using attention over the senses of a word in a context and transferring this sense information to fit multi-sense embeddings in a skip-gram-like framework. We demonstrate an effective approach to training the sense disambiguation mechanism in our model with a distribution over word senses extracted from the output layer embeddings of BERT. Experiments on the contextual word similarity and sense induction tasks show that this method is superior to or competitive with state-of-the-art multi-sense embeddings on multiple benchmark data sets, and experiments with an embedding-based topic model (ETM) demonstrates the benefits of using this multi-sense embedding in a downstream application.