Abigail Jacobs

Excited to share this work on measurement and lessons from NLP with @sulin_blodgett @s010n @haldaume3 @hannawallach @fatconferencehttps://twitter.com/umsi/status/1220828729930285056 …

Very yes!https://twitter.com/karen_ec_levy/status/1212926225762717697 …

See also - excellent recent work by @l_eckhouse @KLdivergence et al. No free lunch.

Finally, the focus of algorithms is often on allocative harms. What can measurement help us understand about civic capacity, or representational harms? (Credit to @profjennabednar at #ChoiceUMich ) 8/

So, you already assumed a precise mathematical defn of fairness, corresponding to a theoretical understanding? Excellent! A precise operationalization comes with a strong assumption about the theoretical construct and embedded values. (Plus. earlier issues baked in upstream!) 7/

Ok ok, some constructs seem gnarly. “Privacy” or “fairness” are essentially contested—inherently ill-defined.  But to the degree to which we measure them anyways, the language of measurement to unpack our assumptions gives us a powerful framework for interrogating them. 6/

Assumptions are present in every step of the ML pipeline: data collection, feature design, target variable, task design. Making these assumptions explicit through the language of measurement and evaluating validity helps us know where to look for potential problems! 5/

Harms emerge from a mismatch between constructs (e.g., 'creditworthiness') and operationalizations (e.g., credit history), or between theoretical understandings of those constructs. (And these constructs are there whether we want to talk about it or not. “Risk,” “quality”.)  4/

The language of measurement helps us unpack assumptions in the design of computational decision-making systems — to diagnose, mitigate, and prevent harms. 3/

This draws from the social sciences to weigh in on a setting where social and technical problems are intertwined — in the fairness/accountability/transparency in ML space. (Work in progress. Feedback welcome!) 2/