We're still evaluating how well (or not) the system works, identifying problems, making improvements, and so on.
But here's some very preliminary analysis, by @andy_matuschak. Or, more precisely, it's me talking aloud about Andy's analysis!
-
-
That is, the marginal review time is going down, even though the time between reviews is increasing exponentially.
Show this thread -
Similarly, the accuracy is getting better with repetition, even with the time between reviews increasing exponentially. At least in this sense, people are becoming more fluent.pic.twitter.com/UeGQPdT1Di
Show this thread -
All very preliminary, of course, and there are many significant issues (esp. selection effects!) which we're working to address in other ways. Still, it's encouraging!
Show this thread -
One curious thing about this demonstrated retention vs review repetitions plot: at 5 reviews, almost everyone who sticks with it has 2.5k days of demonstrated retention. For a worst-case outcome, that seems v encouraging!pic.twitter.com/WBaeqe4bJ3
Show this thread -
Of course, "everyone who sticks with it" is a significant caveat. Maybe people who aren't going so well give up? We're investigating whether that might be the case or not.
Show this thread -
Another curious thing: the people scoring almost zero percent accuracy in the bottom left here. Again, something we're going to look into!pic.twitter.com/fz43OCYRM0
Show this thread -
I suspect, though, that it will turn out to be people using the system in an unusual way. E.g., in the body of the essay we advise people to delight in transgressive uses!pic.twitter.com/EimKoz5vXp
Show this thread -
(Someone running a citizen science project once told me they had a small stream of transgressive users - people who were completely ignoring the point of the project, and "doing things wrong" for their own delight.)
Show this thread -
Another fun tidbit: by repetition 5, almost everyone is getting 90+% of the questions right. It'd be nice to also better understand (a) what's happening for the other users; and (b) how honest people feel they are being (and actually are)?pic.twitter.com/HNm1t1WPph
Show this thread -
It just occurred to me that by strongly encouraging people to engage in transgressive uses, we're putting determined contrarians in a bind. To be transgressive they need to do exactly as we ask
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.