Jack Hessel

Research Scientist @ AI2

contact: jackh@allenai.org

CV (as of 7/2022).

I am on github, twitter, google scholar, and semantic scholar.

I am currently a Research Scientist at AI2. Previously, I earned a PhD in Computer Science at Cornell University.

I work mostly on natural language processing, machine learning, and computer vision. If you're looking for me, I look something like this (facial hair subject to change):

Publications (in reverse chronological order)

• The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning.
  Jack Hessel*, Jena D Hwang*, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi.
  ECCV 2022 (camera ready forthcoming)
  dataset/code/leaderboard; press: venturebeat
• Reframing Human-AI Collaboration for Generating Free-Text Explanations.
  Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi.
  NAACL 2022
  code
• Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer.
  Yanpeng Zhao, Jack Hessel, Youngjae Yu, Ximing Lu, Rowan Zellers, and Yejin Choi.
  NAACL 2022
  code
• Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
  Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi
  NAACL 2022
  code, explainer video
• MERLOT Reserve: Neural Script Knowledge through Sound, Language, and Vision
  Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, Yejin Choi
  CVPR 2022
  code, project page, Press: Venturebeat, Press: GeekWire, with interview from Rowan!
  
• MERLOT: Multimodal Neural Script Knowledge Models
  Rowan Zellers*, Ximing Lu*, Jack Hessel*, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.
  NeurIPS 2021; Oral presentation selection
  code, project page, Press: The Batch, Press: Venturebeat
  
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning
  Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi
  EMNLP 2021
  code, talk
  
• How effective is BERT without word ordering? Implications for language understanding and data privacy.
  Jack Hessel and Alexandra Schofield
  ACL 2021
  code, talk
  
• Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!
  Jack Hessel and Lillian Lee
  EMNLP 2020
  code, talk
  
• Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube
  Jack Hessel, Zhenhai Zhu, Bo Pang, and Radu Soricut
  EMNLP 2020
  data, talk
  
• Domain-Specific Lexical Grounding in Noisy Visual-Textual Documents
  Gregory Yauney, Jack Hessel, and David Mimno
  EMNLP 2020
  code
  
• Learning from Multimodal Web Data
  PhD Thesis
  Cornell University 2020
  
• Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents
  Jack Hessel, Lillian Lee, and David Mimno
  EMNLP 2019
  code/data, poster, project page
• A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions
  Jack Hessel, Bo Pang, Zhenhai Zhu, and Radu Soricut
  CoNLL 2019
  talk slides
• Something's Brewing! Early Prediction of Controversy-causing Posts from Discussion Features
  Jack Hessel and Lillian Lee
  NAACL 2019
  data
• Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets
  Jack Hessel, David Mimno, and Lillian Lee
  NAACL 2018
  code/data
• Cats and Captions vs. Creators and the Clock: Comparing Multimodal Content to Context in Predicting Relative Popularity
  Jack Hessel, Lillian Lee, David Mimno
  WWW 2017
  code/data, slides, replication
• Aligning Images and Text in a Digital Library
  Jack Hessel and David Mimno
  Computer Vision in Digital Humanities Workshop at DH 2017 (extended abstract)
• Science, AskScience and BadScience: On the Coexistience of Highly Related Communities
  Jack Hessel, Chenhao Tan, and Lillian Lee
  ICWSM 2016
  data, slides, replication
• What do Vegans do in their Spare Time? Latent Interest Detection in Multi-Community Networks
  Jack Hessel, Alexandra Schofield, Lillian Lee, David Mimno
  Workshop on Networks in the Social and Information Sciences at NeurIPS 2015
  project page
• Image Representations and New Domains in Neural Image Captioning
  Jack Hessel, Nicolas Savva, and Kimberly J. Wilber
  Workshop on Vision and Language at EMNLP 2015
  slides
• Using Reproductive Altruism to Evolve Multicellularity in Digital Organisms
  Jack Hessel and Sherri Goings
  ECAL 2013

Industry Experience

• I was a postdoc on the Mosaic team at AI2 from October 2020 through June 2021.
• I spent the summer of 2019 working with Google's natural language understanding team.
• I spent the summer of 2018 working with Google's natural language understanding team.
• I spent the summer of 2017 working with Facebook's core data science team.
• I spent the summer of 2016 working with the Twitter Cortex team in NYC.

Things I've Taught

1. I was a visiting faculty member at Carleton College in Spring, 2019; I taught two classes: Natural Language Processing and Discrete Math.
2. INFO/CS 4300 Language and Information, Cornell University, SP 2016
3. CS4786 Machine Learning for Data Science, Cornell University, SP 2015 for which I received an outstanding TA award!
4. CS4620 Introduction to Computer Graphics, Cornell University, FA 2014
5. Introduction to Computer Science/Graph Theory, Carleton College, FA 2011

Invited Talks

• Oct 2020: University of Pittsburgh: "(at least) Two Conceptions of Visual-Textual Grounding" (at the NLP seminar, and for Prof. Malihe Alikhani's CS3730 class) video
• April 2020: Allen Institute for AI: "The Promise and the Perils of Learning Grounding from Visual-Textual Web Data"; video
• April 2020: UNC Chapel Hill: "The Promise and the Perils of Learning Grounding from Visual-Textual Web Data"
• October 2019: Rutgers University: "Multimodal Grounding from User-generated Web Content."
• October 2019: SRI International: "Multimodal Grounding from User-generated Web Content."
• March 2018: University of Pittsburgh: I was the digital humanities research network graduate speaker series speaker. "Grounding Images from a Digital Library in their Textual Contexts."

Service/Guest Lectures/Other Activites

1. Program Committees: ACL 2016, 2017, 2018*, 2019, 2020*, 2021*; NAACL 2018, 2019*, 2021; EMNLP 2017, 2018*, 2019*, 2020; EACL 2017; AAAI 2017; ICWSM 2018; CoNLL 2019*, 2020, 2021; ICML 2020*. For the starred conferences, I was acknowledged as a top reviewer!
2. Workshop Program Committees: Black in AI @ NeurIPS 2017, 2018, 2019, 2020; Student Research Workshop @ NAACL 2018, EACL 2021, ACL 2020, 2021 ; Workshop on Noisy User-generated Text (W-NUT) @ EMNLP 2018, 2019, 2020.
3. Two guest lectures (and one "magic" trick) for Spring 2016/2017 CS4300 about my favorite machine learning algorithm: the truncated SVD!
4. Spoke at Carleton College's Networks Reading Group where I discussed the role of altruism on Kickstarter.
5. Spring Cornell CS4740 Guest Lecture on Topic Models with Xanda Schofield
6. Helped with Cornell's annual Expand Your Horizons conference, where some friends and I taught a class on programming (2015, 2016)

Other Projects

• I wrote a TreeLSTM in tensorflow 2; this is a neural network whose topology changes based on each input example.
• I wrote a factorization machine layer in pytorch; for speed reasons, the forward and backward passes are written in cython.
• I implemented Monroe et al.'s "Fightin' Words" algorithm for robustly comparing word frequencies in two corpora. This implementation has been used in several publications, e.g., this and this
• As part of an NSF REU, I contributed to the implementation of a really (really) fast SVM solver in Kilian Weinberger's lab at Washington University, St. Louis. [now at Cornell!] see Parallel Support Vector Machines in Practice by Tyree, S., Gardner, J. R., Weinberger, K. Q., Agrawal, K., & Tran, J. (2014).
• "A Comparative Analysis of Popular Phylogenetic Reconstruction Algorithms." Undergraduate thesis project/best paper award at MICS 2014: joint work with Evan Albright, Nao Hiranuma, Cody Wang, and Sherri Goings.

More

I grew up in beautiful Portola Valley, California. I earned a B.A. from Carleton College in 2014, studying computer science and mathematics/statistics. During my time in Northfield, I played ice hockey, hosted a radio show, and was lucky enough to meet and work with a lot of great people. I even returned to Carleton briefly in 2019, this time, as a visiting faculty member! I'm a die hard San Jose Sharks fan, avid consumer (and very occasionally a producer) of electronic music, and, most of the time, a vegetarian (I am no longer a vegan!). During graduate school at Cornell, I was a member of Stewart Little Coop, a cooperative living community of 15 people, I played ice hockey in the Ithaca Hockey Association (and, during summer internships in CA, in the San Jose Adult Hockey League).