or they just have no idea what the baseline should be. IMO that's a lot more excusable than what "we" do.
-
agree with
@johnregehr that this seems v. hard. Past example tasks (e.g., SPEC) got heavily gamed...
For most papers I care about Idk how you'd even create a standard let alone a non-gamable one.
-
-
Many papers claim something like "We build A, 2x better than B", where both are proprietary.
-
AFAICT there's no way to verify the claim other than "know a guy who isn't invested in lying"
-
And, unfortunately, I don't really see how that can change with proprietary systems.
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.