🚂Data Engine

Notes from Andrej Karpathy talks

Data Engine HLD internally at metaforms.ai

Hypothesis:

• Unknown unknowns: Dataset is always imperfect, all scenarios are not represented well yet and can always be more diverse

• Capable base model/architecture: Improving dataset improves AI/product guarantees

Inspirations:

A Recipe for Training Neural Networkskarpathy.github.io

1. Become one with the data 2. Set up the end-to-end training/evaluation skeleton + get dumb baselines 3. Overfit 4. Regularize 5. Tune 6. Squeeze out the juice

from 6th to 15th minute

"The only sure certain way I have seen of making progress on any task is, you curate the dataset that is clean and varied and you grow it and you pay the labeling cost and I know that works.”

"Potentially nitpicky but competitive advantage in AI goes not so much to those with data but those with a data engine. And whoever can spin it fastest. Slide from Tesla to ~illustrate but concept is general”

QualEval: Qualitative Evaluation for Model Improvement

https://medium.com/swlh/about-the-long-tail-113e98ce8717