JavaScript is not available.

| Language models, AI-assisted writing, Human-AI interaction | #NLP #HCI

California, USAminalee.infoJoined March 2018

280 Following

1,581 Followers

Mina Lee (on the academic job market)’s Tweets

Pinned Tweet

Language models (LMs) are already deployed in many real-world applications and used to interact with users 👩‍🦰, but these models are primarily evaluated non-interactively. How can we evaluate LMs interactively and why is it important? (1/8)

321

Daniel Jiang

@danielrjiang

Jan 27

New paper with Yijia Wang on “fast-slow MDPs” — MDPs with high frequency decisions, but slowly varying information are also influential. Modeling all states is often computational intractable, so slow states (e.g. weather) are often *omitted* from the decision model in practice.

Alex Tamkin - Tips for New Researchers

Alex Tamkin

@AlexTamkin

Jan 20

I wrote up some things I wish I'd known as an undergrad starting out in research! A few highlights below:

alextamkin.com

Tips for Success as a New Researcher September 2022 This guide is a compilation of tips I wish I knew when I was starting out doing computer science research as an undergraduate student at Stanford....

275

Dr. Susan D'Agostino

@susan_dagostino

Academic experts offer advice on ChatGPT

#ChatGPT advice #academia can use now To harness the potential and avert the risks of

@OpenAI

’s new chat bot, academics should think a few years out, invite #students into the conversation and—most of all—experiment, not panic.

@insidehighered

#GPT

insidehighered.com

To harness the potential and avert the risks of OpenAI’s new chat bot, academics should think a few years out, invite students into the conversation and—most of all—experiment, not panic.

128

Tuhin Chakrabarty

@TuhinChakr

Love it. Especially interactive and collaborative generation as a subarea in the generation track 🤩

Quote Tweet

ACL 2023

@aclmeeting

Curious about our paper-reviewer matching process? Here is a blogpost explaining the keywords-based workflow to assign reviewers to papers #ACL2023 2023.aclweb.org/blog/reviewer-

AI and education: guidance for policy-makers

While thinking about the topic, I came across "AI and education: guidance for policy-makers" by UNESCO (unesdoc.unesco.org/ark:/48223/pf0), which includes benefit-risk assessments and policy recommendations for using AI in education. What other reading resources would you recommend?

unesdoc.unesco.org

book

Academic experts offer advice on ChatGPT

In a classroom setting, how can we harness the potential while minimizing the risks of large language models such as ChatGPT? 👩‍🏫👨‍🎓 Check out this article by Inside Higher Ed:

insidehighered.com

To harness the potential and avert the risks of OpenAI’s new chat bot, academics should think a few years out, invite students into the conversation and—most of all—experiment, not panic.

Percy Liang

Jan 11

I have 6 fantastic students and post-docs who are on the academic job market this year. Here is a short thread summarizing their work along with one representative paper:

529

ACL 2023 Policy on AI Writing Assistance

ACL 2023

@aclmeeting

Jan 10

Planning to use your favorite AI assistant to polish your ACL submission? Checkout ACL 2023's policy on AI Writing Assistance. 2023.aclweb.org/blog/ACL-2023- #ACL2023NLP #ACL2023Toronto #ChatGPT

2023.aclweb.org

Text generation models have been long available, and they are powering many existing tools assisting with input or the linguistic form of the text, like predictive keyboards or language checkers....

Jan 4

When building AI writing support tools, what are the most important factors to consider? How can we make them safe and inclusive? How will they change the way we write? Join the conversation at our workshop on Intelligent and Interactive Writing Assistants! #CHI2023 ✍️👩‍💻

Quote Tweet

Kenneth Huang

@windx0303

Jan 4

The 2nd #In2Writing workshop will be at #CHI2023! This year we invite

2-page position papers

that portray thoughts on writing assistants (see CFP). Submit a paper to join the in-person event!

Submission Deadline: 2/23

Workshop Date: 4/23 (Sun)

in2writing.glitch.me

Show this thread

Ines Gerard-Ursin,

, Tony Lee,

, and

. Work from

. (8/8)

Evaluating Human-Language Model Interaction

TL;DR: Standard (non-interactive) benchmark performance is an imperfect proxy for evaluating human-LM interaction. For more details: arxiv.org/abs/2212.09746 Joint work with

, (cont.) (7/8)

Many real-world applications of language models (LMs), such as code autocomplete and writing assistance, involve human-LM interaction. However, the main LM benchmarks are non-interactive in that a...

Lastly, we observe a discrepancy between metrics based on third-party and first-person perspectives, suggesting that what users find helpful in interacting with and improving these models over time might be not captured by standard benchmarking practices. (6/8)

Second, users sometimes perceived LMs to be more helpful than they are. For instance, users were easily deceived by misinformation from LMs, especially when they were fluent. We further observe that short prompts tend to exacerbate misinformation and toxicity. (5/8)

From 1000+ human-LM interactions, we first find that non-interactive performance and interactive performance can diverge. For example, a model with the worst accuracy achieved the best performance as an interactive LM assistant in certain contexts. (4/8)

We design five tasks, ranging from goal-oriented to open-ended, to capture a variety of different interactions. Then, we evaluate three variants of OpenAI’s GPT-3 and AI21’s Jurassic-1 by constructing interactive systems that connect LMs and users via an interface. (3/8)

We develop a framework, HALIE, that captures (i) the interactive process, not only the final output; (ii) the first-person subjective experience, not just a third-party assessment; and (iii) notions of preference beyond quality. (2/8)

Flurry of apps pushes the limits of co-learning and co-making alongside AI

Dec 16, 2022

CoAuthor is featured in the Stanford Graduate School of Education's article on "A New Class of AI Tools" 🤖✍️💡

medium.com

A New Class of AI Tools

185

Percy Liang

Nov 17, 2022

Language models are becoming the foundation of language technologies, but when do they work or don’t work? In a new CRFM paper, we propose Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of LMs. Holistic evaluation includes three elements:

208

787

Kawin Ethayarajh

@ethayarajh

Nov 16, 2022

'The Authenticity Gap in Human Evaluation' (EMNLP 2022): arxiv.org/abs/2205.11930 We find that the standard way of doing human evaluation is wrong in most cases, leading to incorrect conclusions like GPT3-davinci being no better than GPT3-curie at story generation.

124

Pang Wei Koh

@PangWeiKoh

Nov 14, 2022

I'm honored to be named as one of MIT TR's Asia-Pacific Innovators under 35. Really grateful to all of my collaborators and mentors, who were invaluable to the work that the award recognizes -- thank you! mittrchina.com/news/detail/11

204

MIT 테크놀로지 리뷰 코리아, 35세 미만 젊은 혁신가상 한국 수상자 발표 - MIT Technology Review

Nov 14, 2022

I got selected as one of the Korean Innovators Under 35 by MIT Technology Review! 🇰🇷🏆 Happy and honored! Will do my best to continue to learn, grow, and do good research. ☺️

technologyreview.kr

한국을 넘어 세계로 비상할 '젊은 혁신가' 12명이 선정됐다.

221

Could AI help you to write your next paper?

Matthew Hutson

@SilverJacket

Nov 1, 2022

Could AI help you to write your next paper? My piece in this week’s

@Nature

nature.com

Nature - Large language models can draft abstracts or suggest research directions, but these artificial-intelligence tools are a work in progress.

Participant Recruitment Survey

Yewon Kim

@hai_yewon

Oct 11, 2022

[RT appreciated] We are looking for online interview participants who use AI writing assistants such as Grammarly, WordTune, and Google Translate (~1 hour). We will email selected participants and send a $25 gift card upon completion. Sign up/more info:

docs.google.com

Our team is working on a user-friendly writing assistant. As a first step, we would like to learn more about how people use AI writing tools. If you participate, we will ask you to (i) write a short...

Cori Faklaris (Mastodon handle in bio)

@heycori

Oct 7, 2022

Students in #hci who are on the job market or soon will be: sign up for the CHI-JOBS mailing list. It was very helpful to me in my search! Archive: listserv.acm.org/SCRIPTS/WA-ACM #AcademicTwitter #academicjobs #ux #cs #infosci #compsci #sigchi

113

Bogdana Rakova

@bobirakova

Sep 15, 2022

Fascinating to connect with

@MeganPHMa

, discussing some of the challenges when relying on crowdworkers in human-computer interaction studies on algorithmic bias. Do you have thoughts? 1/

Siva Rajadurai

@sivadotblog

Jul 3, 2022

I recently learned about #CoAuthor from Christopher Manning in #DataAiSummit. Some amazing work by

can be found in coauthor.stanford.edu If you are an #AI & #NLP enthusiasit, this is a great paper to read this weekend.

0:23

2.3K views

Il écrit son prochain scénario avec une I.A. Et le résultat le fascine

🙏🏻

⁩ ⁦

⁩ ⁦

⁩ ⁦

⁩ ⁦⁦

⁩ ⁦

⁩ ⁦

⁩ ⁦

⁩ ⁦

⁩ ⁦⁦

⁩ ⁦

⁩ ⁦⁦⁦

⁩

Le scénariste Simon Bouisson écrit son prochain film avec une intelligence artificielle. Une collaboration qui ne cesse pas de le surprendre. Nous aussi. Il nous raconte.

Emergent Abilities of Large Language Models

rishi

@RishiBommasani

Jun 20, 2022

We just put out a paper on emergent abilities of LMs, led by the vivacious

@_jasonwei

! arxiv.org/abs/2206.07682

@_jasonwei

@LiamFedus

have put out great commentaries that I recommend checking out! Here, I give my thoughts on emergence, scale, and scientific & social consequences.

arxiv.org

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon...

169

Ludwig Siegele

@EconoScribe

Jun 8, 2022

I asked Mina Lee, a PhD candidate in computer science at Stanford University, to build an “EconoBot”, a digital writing buddy. Read what happened buff.ly/3Hkq9UJ

katyilonka

@katyilonka

May 27, 2022

I had a great time at our Workshop on Intelligent and Interactive Writing Assistants #In2Writing yesterday at #ACL2022! 🤖👩‍🦰✏️🌟 Thanks so much to my fellow organizers

! A thread of my highlights...

May 26, 2022

CoAuthor is featured on

@StanfordHAI

News and Announcements! ✏️ Joint work with

Meet CoAuthor, an Experiment in Human-AI Collaborative Writing

hai.stanford.edu

Researchers study how humans and AI can write together by designing large interaction datasets.

Kenneth Huang and 5 others

May 25, 2022

Curious how writers write with language models / how industries build writing assistants / how researchers evaluate their performance? Workshop on Intelligent and Interactive Writing Assistants #In2Writing is happening tomorrow at #ACL2022! 🤖👩‍🦰✏️ in2writing.glitch.me

May 21, 2022

Congrats

@WanyuDu

and coauthors! Look forward to learning more about your work at the #In2Writing workshop next week! 🏆✍️

Quote Tweet

Wanyu Du

@WanyuDu

May 19, 2022

If you are interested in human-machine interactive text revision, please check out our paper, which won

the best paper award

at #In2Writing workshop at #ACL2022! We attach more details below. Paper: arxiv.org/abs/2204.03685 Demo: youtube.com/watch?v=lK08tI

Show this thread

May 5, 2022

Thank you

Qian Yang @fabulousqian@hci.social

! Check out our paper and dataset here: coauthor.stanford.edu ✍️ #CHI2022 By the way, I seem to have closed my eyes for all the photos for some reason 🫠 PC:

@joon_s_pk

Quote Tweet

May 4, 2022

Fantastic talk - @MinaLee__’s #chi2022 presentation of our Best Paper Honorable Mention paper “CoAuthor” (in collaboration with @percyliang)— A much needed discussion on the capabilities and limits of LLMs when interacting with users!

Qian Yang @fabulousqian@hci.social

Mar 29, 2022

Honorable Mention 🏅 So proud of you, Mina 👏

Quote Tweet

Jan 20, 2022

CoAuthor: Human-AI Collaborative Writing Dataset #CHI2022

CoAuthor captures rich interactions between 63 writers and GPT-3 across 1445 writing sessions Paper & dataset (replay): coauthor.stanford.edu Joint work with @percyliang @fabulousQian

Show this thread

GIF

read image description

ALT

Dongyeop Kang (DK)

@dongyeopkang

Mar 23, 2022

Human writers often make revisions "iteratively" (v1->v2->v3->..) to improve the text quality. We release a dataset that annotates the intentions behind the edits over different versions of the texts. You can also load the dataset/models with a few lines of

@huggingface

code!

Quote Tweet

Wanyu Du

@WanyuDu

Mar 22, 2022

New ACL Paper Alert

#acl2022nlp #nlpproc Iteratively revising a text is a crucial part of writing, and in our paper, we find the text quality keeps increasing as humans “iteratively” revise the text. To foster future research, we construct the IteraTeR dataset (first figure).

Show this thread

Ananya Kumar

@ananyaku

Mar 2, 2022

How should you fine-tune a large pretrained model (CLIP, SimCLR) robustly? We find that standard fine-tuning can do poorly out-of-distribution (test data ≠ fine-tuning data). Our analysis leads to a simple fix, higher accuracy on 10 datasets. arxiv.org/abs/2202.10054 (ICLR Oral)

127

646

Kenneth Huang

@windx0303

Mar 2, 2022

🚨DEADLINE EXTENSION🚨 We extended the deadline of #In2Writing workshop (at #ACL2022NLP

@aclmeeting

). The new deadline is *March 9, 2022* (Wed). The submission system is open. Send us your papers :)

Quote Tweet

Kenneth Huang

@windx0303

Dec 21, 2021

Call For Papers

1st Workshop on Intelligent & Interactive Writing Assistants #In2Writing will co-locate with #ACL2022nlp! #NLProc #HCI We welcome 4 submission types:

Paper

Extended Abstract

Demo

Cross-submission (!) Deadline: Feb 28, 2022 Site:in2writing.glitch.me

read image description

ALT