Currently looking for a new PhD student to work on representation learning in LLMs. We will use our nonparametric variational information bottleneck to induce more interpretable abstract structured representations of language.

Short Biography

James Henderson is a Senior Researcher at the Idiap Research institute, where he heads the Natural Language Understanding group. He is currently an Action Editor for the journal Transactions of the Association for Computational Linguistics (TACL).

Previously he was a Chargé de Cours in the Department of Computer Science of the University of Geneva, where he participated in running the interdisciplinary research group Computational Learning and Computational Linguistics. Before moving to Idiap, Dr Henderson was the Principal Scientist of the Parsing and Semantics group at Xerox Research Centre Europe, in Grenoble, which is now Naver Labs Europe. Before that he was a Maître d'Enseignement et de Recherche (MER) in the Department of Computer Science of the University of Geneva. Earlier he was a Research Fellow in the Institute for Communicating and Collaborative Systems at the University of Edinburgh , a Maître-Assistant in the Department of Computer Science at the University of Geneva, and a Lecturer in the then Department of Computer Science at the University of Exeter, UK.

 

Education

Dr Henderson received his PhD and MSc from the University of Pennsylvania, and his BSc from the Massachusetts Institute of Technology, all in computer science.

 

Research

Dr Henderson investigates machine learning methods for natural language processing tasks. He is well known for his pioneering work on recurrent neural networks (deep learning) for syntactic and semantic-role parsing (structured prediction). Current research topics include representation learning for the semantics of language, graph-to-graph deep learning models (see our big-picture paper), entity induction (see my ACL 2020 theme paper), and variational-Bayesian attention-based representation learning (see our ICLR 2023 paper). Recent work was presented at the CALCULUS Symposium in the talk "Bayesian Language Understanding with Nonparametric Variational Transformers". See the publications below and the description of the Natural Language Understanding group for other topics.

 

Publications

Fabio Fehr, and James Henderson. Nonparametric Variational Regularisation of Pretrained Transformers. COLM, 2024.

Andrei Coman, Christos Theodoropoulos, Marie-Francine Moens, and James Henderson. GADePo: Graph-assisted declarative pooling transformers for document-level relation extraction. Workshop on Knowledge Augmented Methods for NLP, at ACL, 2024.

Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, and Arman Cohan. TESS: Text-to-text selfconditioned simplex diffusion ACL, 2024.

Melika Behjati, Fabio Fehr, and James Henderson. Learning to Abstract with Nonparametric Variational Information Bottleneck. EMNLP Findings, 2023.

James Henderson, Alireza Mohammadshahi, Andrei C. Coman, and Lesly Miculicich. Transformers as Graph-to-Graph Models. Big Picture Workshop, at EMNLP, 2023.

Melika Behjati and James Henderson. Inducing meaningful units from character sequences with dynamic capacity slot attention. TMLR Journal, 2023. [PDF]

Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, and James Henderson. HyperMixer: An MLP-based low cost alternative to transformers. ACL, 2023.

Alireza Mohammadshahi and James Henderson. Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling. RepL4NLP, at ACL, Toronto, Canada, 2023.

James Henderson and Fabio Fehr. A VAE for Transformers with Nonparametric Variational Information Bottleneck. ICLR, 2023.  [PDF]  [video]

Florian Mai and James Henderson. Bag-of-vectors autoencoders for unsupervised conditional text generation. AACL & IJCNLP 2022, Online, 2022.

Andreas Marfurt and James Henderson. Unsupervised token-level hallucination detection from summary generation by-products. Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), Abu Dhabi, 2022.

Andreas Marfurt, Ashley Thornton, David Sylvan, Lonneke van der Plas, and James Henderson. A corpus and evaluation for predicting semi-structured human annotations. Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), Abu Dhabi, 2022.

Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. Prompt-free and efficient few-shot learning with language models. ACL 2022. Dublin, 2022.

Lesly Miculicich and James Henderson. Graph Refinement for Coreference Resolution. Findings of ACL 2022. Dublin, 2022.

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient Low-Rank Hypercomplex Adapter Layers. NeurIPS 2021, Online, 2021.

Andreas Marfurt and James Henderson. Sentence-level Planning for Especially Abstractive Summarization. Third Workshop on New Frontiers in Summarization, Dominican Republic, 2021.

Christos Theodoropoulos, James Henderson, Andrei Catalin Coman, Marie-Francine Moens. Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning. CoNLL 2021, Dominican Republic, 2021.

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks. ACL 2021, Online, 2021.

Haozhou Wang, James Henderson, Paola Merlo. Multi-Adversarial Learning for Cross-Lingual Word Embeddings. NAACL 2021, Online, 2021.

Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. Variational Information Bottleneck for Effective Low-Resource Fine-Tuning. ICLR 2021, Online 2021.

Alireza Mohammadshahi and James Henderson. Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement. Transactions of the Association for Computational Linguistics (TACL), 9:120–138, 2021.

Lesly Miculicich and James Henderson. Partially-Supervised Mention Detection. CRAC 2020, Online, 2020.

Florian Mai, Nikolaos Pappas, Ivan Montero, Noah A. Smith, James Henderson. Plug and Play Autoencoders for Conditional Text Generation. EMNLP 2020, Online, 2020.

Alireza Mohammadshahi and James Henderson. Graph-to-graph Transformer for Transition-Based Dependency Parsing. Findings of ACL: EMNLP 2020, Online, 2020.

James Henderson, The Unstoppable Rise of Computational Linguistics in Deep Learning. ACL 2020 (theme track), Online, 2020.

Rabeeh Karimi Mahabadi, Yonatan Belinkov and James Henderson. End-to-End Bias Mitigation by Modelling Biases in Corpora. ACL 2020, Online, 2020.

Haozhou Wang, James Henderson and Paola Merlo. Weakly-Supervised Concept-based Adversarial Learning for Cross-lingual Word Embeddings. EMNLP 2019, Hong Kong, China, 2019.

Nikolaos Pappas, James Henderson. GILE: A Generalized Input-Label Embedding for Text Classification. Transactions of the Association for Computational Linguistics (TACL), 7:139–155, 2019.

Nikolaos Pappas, James Henderson. Deep Residual Output Layers for Neural Language Generation. ICML, 2019.

Diana Nicoleta Popa, Julien Perez, James Henderson, and Eric Gaussier. Implicit discourse relation classification with syntax-aware contextualized word representations. FLAIRS-32, Sarasota, USA, 2019.

Xiao Pu, Nikolaos Pappas, James Henderson, and Andrei Popescu-Belis. Integrating Weakly Supervised Word Sense Disambiguation into Neural Machine Translation. Transactions of the Association for Computational Linguistics (TACL), 6:635–649, 2018.

Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas and James Henderson. Document-Level Neural Machine Translation with Hierarchical Attention Networks. EMNLP 2018, Brussels, Belgium, 2018.

Nikolaos Pappas, Lesly Miculicich and James Henderson. Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation. WMT 2018, Brussels, Belgium, 2018.

James Henderson. Learning Word Embeddings for Hyponymy with Entailment-Based Distributional Semantics. ArXiv e-prints, arXiv:1710.02437 [cs.CL], 2017.

Diana Nicoleta Popa and James Henderson. Bag-of-Vector Embeddings of Dependency Graphs for Semantic Induction. ArXiv e-prints, arXiv:1710.00205 [cs.CL], 2017.

Christophe Moor, Paola Merlo, James Henderson, and Haozhou Wang CLCL (Geneva) DINN Parser: a Neural Network Dependency Parser Ten Years Later. CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, Vancouver, Canada, 2017.

James Henderson and Diana Nicoleta Popa. A Vector Space for Distributional Semantics for Entailment. ACL 2016, Berlin, Germany, 2016.  [poster]

Nikhil Garg and James Henderson. A Bayesian Model of Multilingual Unsupervised Semantic Role Induction. ArXiv e-prints, arXiv:1603.01514 [cs.CL], 2016.

Majid Yazdani, Meghdad Farahmand, and James Henderson. Learning Semantic Composition to Detect Non-compositionality of Multiword Expressions. EMNLP 2015, Lisbon, Portugal, 2015.

Majid Yazdani and James Henderson. A Model of Zero-Shot Learning of Spoken Language Understanding. EMNLP 2015, Lisbon, Portugal, 2015.

Will Radford, Xavier Carreras and James Henderson. Named entity recognition with document-specific KB tag gazetteers. EMNLP 2015, Lisbon, Portugal, 2015.

M.Yazdani, J.Henderson. Incremental Recurrent Neural Network Dependency Parser with Search-based Discriminative Training. CoNLL 2015, Beijing, China, 2015.

A.Gesmundo, J.Henderson, Undirected Machine Translation with Discriminative Reinforcement Learning. EACL 2014, Gothenburg, Sweden, 2014.

J.Henderson, P.Merlo, I.Titov, and G.Musillo, Multilingual Joint Parsing of Syntactic and Semantic Dependencies with a Latent Variable Model. Computational Linguistics, December 2013, Vol. 39, No. 4.

J.Lang and J.Henderson, Graph-Based Seed Set Expansion For Relation Extraction Using Random Walk Hitting Times. NAACL 2013, Atlanta, Georgia, USA, 2013.

Helen Hastie, Marie-Aude Aufaure, Panos Alexopoulos, Heriberto Cuayáhuitl, Nina Dethlefs, Milica Gasic, James Henderson, Oliver Lemon, Xingkun Liu, Peter Mika, Nesrine Ben Mustapha, Verena Rieser, Blaise Thomson, Pirros Tsiakoulis, Yves Vanrompay, Boris Villazon-Terrazas, Steve Young, Demonstration of the Parlance system: a data-driven, incremental, spoken dialogue system for interactive search. SIGDIAL 2013.

N.Garg and J.Henderson, Unsupervised Semantic Role Induction with Global Role Ordering. ACL 2012, Jeju Island, Korea, 2012.

A.Gesmundo, G.Satta, and J.Henderson, Heuristic Cube Pruning in Linear Time. ACL 2012, Jeju Island, Korea, 2012.

J.Henderson. Bayesian Network Automata for Modelling Unbounded Structures. IWPT 2011, Dublin, Ireland, 2011.

A.Gesmundo and J.Henderson. Heuristic Search for Non-Bottom-Up Tree Structure Prediction. EMNLP 2011, Edinburgh, UK, 2011.

N.Garg and J.Henderson. Temporal Restricted Boltzmann Machines for Dependency Parsing. ACL 2011, Portland, Oregon, 2011.

L.van der Plas, P.Merlo, and J.Henderson. Scaling up Cross-Lingual Semantic Annotation Transfer. ACL 2011, Portland, Oregon, 2011.

J.Henderson and I.Titov. Incremental Sigmoid Belief Networks for Grammar Learning. Journal of Machine Learning Research, 11(Dec):3541-3570, 2010.

A.Gesmundo and J.Henderson. Faster Cube Pruning. International Workshop on Spoken Language Translation (IWSLT), Paris, France, 2010.

J.Henderson. Artificial Neural Networks. A.Clark, C.Fox, and S.Lappin, editors, Handbook of Computational Linguistics and Natural Language Processing (Blackwell), 2010.

I.Titov and J.Henderson. A Latent Variable Model for Generative Dependency Parsing. In H. Bunt, P. Merlo and J. Nivre, editors, Trends in Parsing Technology. Text, Speech and Language Technology Series (Springer) 2010.

A.Gesmundo, J.Henderson, P.Merlo, and I.Titov. A Latent Variable Model of Synchronous Syntactic-Semantic Parsing for Multiple Languages. CoNLL 2009 Shared Task, Conf. on Computational Natural Language Learning (CoNLL-09), Boulder, CO, 2009.

L.van der Plas, J.Henderson, and P.Merlo. Domain Adaptation with Artificial Data for Semantic Parsing of Speech. NAACL, Boulder, CO, 2009.

I.Titov, J.Henderson, P.Merlo, and G.Musillo. Online Graph Planarisation for Synchronous Parsing of Semantic and Syntactic Dependencies. IJCAI-09, Pasadena, California, USA, 2009.

K.Georgila, O.Lemon, J.Henderson, and J.D.Moore. Automatic annotation of context and speech acts for dialogue corpora. Natural Language Engineering, 15(03):315-353, 2009.

J.Henderson, O.Lemon, K.Georgila. Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets. Computational Linguistics, 34(4):487-511, 2008.

J.Henderson, P.Merlo, G.Musillo, I.Titov A Latent Variable Model of Synchronous Parsing for Syntactic and Semantic Dependencies. CoNLL 2008 Shared Task, Conf. on Computational Natural Language Learning (CoNLL-08), Manchester, UK, 2008.

J.Henderson and O.Lemon. Mixture model POMDPs for efficient handling of uncertainty in dialogue management. ACL'08, Columbus, Ohio, 2008.

I.Titov and J.Henderson. Incremental Bayesian Networks for Structure Prediction. ICML 2007, Corvallis, OR, USA , 2007.

I.Titov and J.Henderson. Constituent Parsing with Incremental Sigmoid Belief Networks. ACL 2007, Prague, Czech Republic, 2007.

I.Titov and J.Henderson. A Latent Variable Model for Generative Dependency Parsing. IWPT 2007, Prague, Czech Republic, 2007.

I.Titov and J.Henderson. Fast and Robust Multilingual Dependency Parsing with a Generative Latent Variable Model. EMNLP-CoNLL 2007, Prague, Czech Republic, 2007. (CoNLL Shared Task, 3rd result out of 23)

I.Titov and J.Henderson. Loss minimization in parse reranking. EMNLP 2006, Sydney, Australia, 2006.

I.Titov and J.Henderson. Porting statistical parsers with data-defined kernels. CoNLL 2006, New York, NY, USA, 2006.

I.Titov and J.Henderson. Bayes Risk Minimization in Natural Language Parsing. Technical report. University of Geneva, 2006. [Send email to Ivan Titov if you would like the implementation]

J.Henderson and I.Titov. Data-defined kernels for parse reranking derived from probabilistic models. ACL 2005, Ann Arbor, MI, USA, 2005.

I.Titov and J.Henderson. Deriving kernels from MLP probability estimators for large categorization problems. IJCNN 2005, Montreal, Quebec, Canada, 2005.

J.Henderson, O.Lemon, and K.Georgila. Hybrid reinforcement/supervised learning for dialogue policies from COMMUNICATOR data. IJCAI workshop on Knowledge and Reasoning in Practical Dialogue Systems, Edinburgh, UK, 2005.

K.Georgila, J.Henderson, and O.Lemon. Learning User Simulations for Information State Update Dialogue Systems. INTERSPEECH - EUROSPEECH 2005, Lisbon, Portugal, 2005.

K.Georgila, O.Lemon, and J.Henderson. Automatic annotation of COMMUNICATOR dialogue data for learning dialogue strategies and user simulations. SEMDIAL: DIALOR 2005, Nancy, France, 2005.

J.Henderson. Discriminative training of a neural network statistical parser. ACL 2004, Barcelona, Spain, 2004.

J.Henderson. Lookahead in deterministic left-corner parsing. Workshop on Incremental Parsing: Bringing Engineering and Cognition Together, Barcelona, Spain, 2004.

J.Henderson. Estimating Probabilities for Unbounded Categorization Problems. Neurocomputing, 57:77--86, 2004.

J.Henderson. A neural network parser that handles sparse data. In H. Bunt, J. Carroll, and G. Satta, editors, New Developments in Parsing Technology. Kluwer, Boston/Dordrecht/London, 2004.

J.Henderson. Inducing History Representations for Broad Coverage Statistical Parsing. HLT-NAACL 2003, pages 103-110, Edmonton, Canada, 2003.

J.Henderson. Structural Bias in Inducing Representations for Probabilistic Natural Language Parsing. ICANN/ICONIP 2003, Istanbul, Turkey, 2003.

J.Henderson. Generative Versus Discriminative Models for Statistical Left-Corner Parsing. IWPT 2003, pages 115-126, Nancy, France, 2003.

J.Henderson. Neural network probability estimation for broad coverage parsing. EACL 2003, pages 131-138, Budapest, Hungary, 2003.

P.Lane and J.Henderson. Towards effective parsing with neural networks: Inherent generalization and bounded resource effects. Applied Intelligence, 19(1):83-99, 2003.(web page)

J.Henderson, P.Merlo, I.Petroff, and G.Schneider. Using Syntactic Analysis to Increase Efficiency in Visualizing Text Collections. COLING 2002, pages 335-341, Taipei, Taiwan, 2002.

J.Henderson, P.Merlo, I.Petroff, and G.Schneider. Using NLP to Efficiently Visualize Text Collections with SOMs. NLIS 2002, Aix-en-Provence, France, 2002.

J.Henderson. Estimating Probabilities for Unbounded Categorization Problems. ESANN 2002, pages 383-388, Bruges, Belgium, 2002.

P.Lane and J.Henderson. Incremental Syntactic Parsing of Natural Language Corpora with Simple Synchrony Networks. IEEE Transactions on Knowledge and Data Engineering, 13(2), 2001.

J.Henderson. Segmenting State into Entities and its Implication for Learning. In S.Wermter, J.Austin, and D.Willshaw, editors, Emergent Neural Computational Architectures based on Neuroscience, pages 227-236. Springer-Verlag, Heidelberg, Germany, 2001.

J.Henderson. Estimating a Probabilistic Grammar Using a Neural Network 1st workshop on Robust Methods in Analysis of Natural Language Data (ROMAND 2000), Lausanne, Switzerland, 2000.

J.Henderson. Segmenting State into Entities and its Implication for Learning International Workshop on Current Computational Architectures Integrating Neural Networks and Neuroscience (EmerNet 2000), Durham, UK, 2000.

J.Henderson. A Neural Network Parser that Handles Sparse Data. IWPT 2000, pages 123-134, Trento, Italy, 2000.

J.Henderson. Constituency, Context, and Connectionism in Syntactic Parsing. In M.Crocker, M.Pickering, and C.Clifton, editors, Architectures and Mechanisms for Language Processing, pages 189--209. Cambridge University Press, Cambridge UK, 2000.

J.Henderson and P.Lane. A Connectionist Architecture for Learning to Parse. COLING-ACL`98, pages 531-537, University of Montreal, Canada, 1998.

P.Lane and J.Henderson. Simple Synchrony Networks: Learning to Parse Natural Language with Temporal Synchrony Variable Binding. ICANN`98, pages 615-620, Skövde, Sweden, 1998.

J.Henderson. A Connectionist Architecture with Inherent Systematicity. CogSci`96, pages 574--579, La Jolla, CA, 1996.

J.Henderson. Connectionist Syntactic Parsing Using Temporal Variable Binding. Journal of Psycholinguistic Research, 23(5):353--379, 1994.

J.Henderson. Description Based Parsing in a Connectionist Network. PhD thesis, University of Pennsylvania, Philadelphia, PA. Technical Report MS-CIS-94-46, 1994.

J.Henderson. A Connectionist Parser for Structure Unification Grammar. ACL`92, Newark, DE, 1992.

J.Henderson. A Structural Interpretation of Combinatory Categorial Grammar. Technical Report MS-CIS-92-49, University of Pennsylvania, Philadelphia, PA, 1992.

J.Henderson. Structure Unification Grammar: A Unifying Framework For Investigating Natural Language. Masters thesis, University of Pennsylvania, Philadelphia, PA. Technical Report MS-CIS-90-94, 1990.

 

Curriculum vitae

---

Tel: +41277217729
Office: 107.1
Contact

© Idiap Research Institute 2017-