The number one thing to keep in mind about machine learning is that performance is evaluated on samples from one dataset, but the model is used in production on samples that may not necessarily follow the same characteristics...
Show this thread
The finance industry has a saying for this: "past performance is no guarantee of future results". Your model scoring X on your test dataset doesn't mean it will perform at level X on the next N situations it encounters in the real world. The future may not be like the past.
-
-
So when asking the question, "would you rather use a model that was evaluated as 90% accurate, or a human that was evaluated as 80% accurate", the answer depends on whether your data is typical per the evaluation process. Humans are adaptable, models are not.
Show this thread -
If significant uncertainty is involved, go with the human. They may have inferior pattern recognition capabilities (versus models trained on enormous amounts of data), but they understand what they do, they can reason about it, and they can improvise when faced with novelty
Show this thread -
If every possible situation is known and you want to prioritize scalability and cost-reduction, go with the model. Models exist to encode and operationalize human cognition in well-understood situations.
Show this thread -
("well understood" meaning either that it can be explicitly described by a programmer, or that you can amass a dataset that densely samples the distribution of possible situations -- which must be static)
Show this thread
End of conversation
New conversation -
-
-
This is the induction problem, not just an ML problem.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
I remember going to a great talk about evaluating your model once it's in production for financial trading strategies. How do you know your model is bad or today is an outlier?
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
A fellow called David Hume has a few things to say on this matter
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.