I haven't used it but git seems like the wrong framework for managing experiments. Even programmers don't especially like git, forget about data scientists. You can't meaningfully version control the data; it's usually binary, and in an object store. The code already lives in a git repository. So what does that leave, the metadata around the experiment?
A relational database like Dolt seems like a better fit. You want to be able to query by experiment name, date, test results, and other metadata.
Let me know if I'm doing it wrong! What's your use case?
> You can't meaningfully version control the data; it's usually binary, and in an object store. The code already lives in a git repository. So what does that leave, the metadata around the experiment?
Out of the box you can't. We're taking a different approach at work (XetHub). GitHub sees pointer files but we embed rendered views and diffs (supporting more file types incrementally) using a Github app from our service.
DVC stores metadata files in your git repo and the actual files somewhere else (local, S3, etc). It handles swapping around the files so they match what you've checked out. It does a lot more but that's the relevant part for your main question.
I built an ML pipeline in pachyderm years ago, maybe 2016, and loved using it then. But my data was small-to-medium, and I stuck with CSVs because back then you couldn't get many of the benefits with a binary format.
I think it's a shame it took Airflow and similar several more years to realize "each step is a docker container" is the right way to build a dag. It's not clear to me why Pachyderm was left behind while Prefect and Dagster became serious contenders, and Airflow/Astronomer started recommending everyone use it just like Pachyderm (container per step).
I have a similar harness going for my recent experiments, except instead of hosting with huggingface I have a dataframe with pointers to the files on S3 and then just download them during local preprocessing.
Every time I see DVC mentioned I always feel like the idea was so close (and perhaps right in intuition to use git for everything) but the execution had just enough friction that I looked elsewhere. Small DX improvements really do cascade pretty far.
DVC is great for medium-scale projects in small teams, but that's where I'd stop with it. It only really makes sense for work that you're doing on your own machine, or an old-school Linux server type of setup, not something you'd use for modern-day ML work in a cloud environment.
Also I always thought the idea of using Git branches to track experiments was a bad idea. I would never want to only have one experiment "active" at a time. Even if I'm only running one process at a time, I still want to be able to look at outputs and such all side-by-side. Maybe there's some magic tooling they created that makes it workable.
DVC has `dvc exp` that doesn't require creating commits or branches. It's utilizing git custom references (technical details [1]). And it can be visualized in CLI or VS Code.
Thanks! I've been using DVC solely for tracking data, and had basically ignored all of its other features.
I'll have to take a look at this. Most/all of my projects use small or medium scale data, and I consider DVC indispensable for tracking data therein. I wouldn't mind having a good system for tracking experiment results, although admittedly I find that a spreadsheet or text file does a pretty good job for what I need to do.
How do you track code-changes between model iterations with this setup though? From looking at wandb, it seems like it does something similar to MLFlow, so it only logs meta-parameters right?
It can log anything you want model config and evaluation metrics along with git commit hash of the code that was used. You can also put pointers to datasets. Easy to compare experiments and choose a model or repeat experiments.
which things do you do with HF, and which with WandB?
HF seems like "GitHub for models and datasets", it has a cool brand and everyone in ML uses it in some capacity. But when it comes to _company_ needs like private datasets/models, experiment tracking, CI integrations, etc. it seems WandB is a superset of HF.
HF has an enterprise offering, but it seems to be de-prioritized, and I think you'd still need WandB or MLFlow for experiment tracking?
