> For the complete documentation index, see [llms.txt](https://notes.siddish.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://notes.siddish.com/ai/data-engine.md).
# Data Engine
Data Engine HLD internally at metaforms.ai
Hypothesis:
* Unknown unknowns: Dataset is always imperfect, all scenarios are not represented well yet and can always be more diverse
* Capable base model/architecture: Improving dataset improves AI/product guarantees
**Inspirations**:
{% embed url="" %}
**1. Become one with the data**\
**2. Set up the end-to-end training/evaluation skeleton + get dumb baselines**\
**3. Overfit**\
**4. Regularize**\
**5. Tune**\
**6. Squeeze out the juice**
{% endembed %}
{% embed url="" %}
from 6th to 15th minute
{% endembed %}
{% embed url="" %}
"The only sure certain way I have seen of making progress on any task is, you curate the dataset that is clean and varied and you grow it and you pay the labeling cost and I know that works.”
"Potentially nitpicky but competitive advantage in AI goes not so much to those with data but those with a data engine. And whoever can spin it fastest. Slide from Tesla to \~illustrate but concept is general”
[QualEval: Qualitative Evaluation for Model Improvement](https://qualeval.org/#:~:text=QualEval%3A%20Qualitative%20Evaluation%20for%20Model%20Improvement)
{% embed url="" %}