Abigail Jacobs

Excited to share this paper on how the language of measurement can unite + clarify issues in the fairness, accountability, transparency in ML space. A common framework for an emerging field. Joint with @hannawallach https://arxiv.org/abs/1912.05511  1/

This draws from the social sciences to weigh in on a setting where social and technical problems are intertwined — in the fairness/accountability/transparency in ML space. (Work in progress. Feedback welcome!) 2/

The language of measurement helps us unpack assumptions in the design of computational decision-making systems — to diagnose, mitigate, and prevent harms. 3/

Harms emerge from a mismatch between constructs (e.g., 'creditworthiness') and operationalizations (e.g., credit history), or between theoretical understandings of those constructs. (And these constructs are there whether we want to talk about it or not. “Risk,” “quality”.)  4/

Assumptions are present in every step of the ML pipeline: data collection, feature design, target variable, task design. Making these assumptions explicit through the language of measurement and evaluating validity helps us know where to look for potential problems! 5/

Ok ok, some constructs seem gnarly. “Privacy” or “fairness” are essentially contested—inherently ill-defined.  But to the degree to which we measure them anyways, the language of measurement to unpack our assumptions gives us a powerful framework for interrogating them. 6/