I'm starting to think loss is harmful. Our loss has been a flat 2.2 for the past five days training GPT-2 1.5B. Yet according to human testing, it's been getting noticeably better every day. It's now good enough to amuse /r/dota2:https://www.reddit.com/r/DotA2/comments/ei505v/ive_been_working_on_an_rdota2_simulator_its_like/ …
-
Show this thread
-
Replying to @theshawwn @heghbalz
This would suggest that the weights still travel to "better" arrangement despite near constant loss. I thought that at that point it's more of a random walk.
1 reply 0 retweets 1 like -
Replying to @FlyingOctopus0 @theshawwn
Isn't it why we have eval measures rather than the loss e.g, accuracy, F1, FID, etc? To tell us how *good* the weights are? In GANs at least this is normal not to trust much the avg loss, and look at eval measures.
1 reply 0 retweets 1 like -
Replying to @heghbalz @FlyingOctopus0
Notably,
@citnaj doesn’t really pay attention to loss or eval metrics for DeOldify. The only way to test it is to look at it and see. I think this idea is upsetting to people who want to believe everything can be automated. (It upsets me.) But it seems true.3 replies 0 retweets 5 likes
2/ Generally speaking, I really think we should be skeptical of simple measures in life. What can be measured isn't necessarily what's important, though it's easy to fixate on because it's simple. The consequences for ignoring the problems with this I think are profound.
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.