Your model weights are versioned. Your code is in Git. But can you reproduce last Thursday's training run down to the exact byte of input data? For most ML teams, the answer is a quiet no. Data version control (DVC) closes that gap by treating datasets and pipeline artifacts as first-class citizens in your workflow. This guide shows you how to build a pragmatic DVC pipeline on top of object storage—covering snapshot strategies, storage layout, experiment tracking, and the trade-offs you will face when your data outgrows a single bucket.
Model registries capture the output, not the input. When a stakeholder asks why a churn model degraded by 12% between Q3 and Q4, the model artifact tells you nothing about the training data drift. DVC gives you a content-addressable link between a model checkpoint and the exact dataset version that produced it. This matters for audits, debugging, and even simple collaboration: without data versioning, two engineers can run the same training script and get different results simply because one pulled a newer copy of the raw data.
The core mechanism is simple: DVC stores metadata in Git (or any VCS) and stores the actual data in a remote cache, typically S3, GCS, or Azure Blob. Each dataset version is identified by a hash of its contents, and the version pointer is committed to Git. This gives you atomic, shareable, and reproducible datasets without bloating your repository.
Git LFS also stores large files outside Git, but it lacks dependency tracking between files and pipeline stages. Delta Lake provides ACID transactions and time travel on data lakes, but it ties you to a table format and a compute engine like Spark. DVC sits in between: it versions arbitrary files and directories, orchestrates pipeline steps, and works with any storage backend. For ML teams that already use Pandas, PyTorch, or TensorFlow on raw files, DVC is often the least invasive starting point.
Start with a dedicated bucket for DVC cache and artifacts. The bucket should have versioning enabled on the cloud side, even though DVC keeps its own version pointers. This adds a second layer of protection against accidental deletion. Choose a prefix layout that separates cache from outputs. A common pattern is s3://ml-data/dvc-cache/ for the content-addressable store and s3://ml-data/artifacts/ for final models and evaluation reports. You can also use separate buckets—one for raw data, one for intermediate, one for final—if your governance team requires strict separation.
Initialize DVC in your project directory and add the remote:
dvc init dvc remote add -d storage s3://ml-data/dvc-cache
Next, add dependencies and outputs to dvc.yaml. For example, your data processing stage might look like this:
stages:
process:
cmd: python src/process.py --input data/raw/ --output data/processed/
deps:
- src/process.py
- data/raw/
outs:
- data/processed/When you run dvc repro, DVC checks if dependencies changed. If they did, it executes the command and stores new outputs in the cache, updating dvc.lock with the new hashes. Commit both dvc.yaml and dvc.lock to Git. This is your reproducibility contract: any teammate who checks out that commit and runs dvc pull will get the exact same data files.
Not all data is created equal. A directory of 50,000 small JSON files hashes differently than a single 10 GB parquet file, and both demand different snapshot strategies.
If you work with partitioned datasets (e.g., year=2025/month=04/), you rarely want to version the entire directory at once. Instead, add individual partitions or a manifest file. Using a manifest is a clean pattern: run a script that reads the partition list and writes a JSON manifest with file sizes and checksums. Add that manifest as a DVC dependency, not the raw files. This avoids re-hashing thousands of files on every run. For parquet, you can also rely on file metadata—the footer stores column statistics—but DVC doesn't read that, so you must generate the manifest yourself.
Image datasets change incrementally—new images arrive, some get relabeled. Snapshotting the entire directory every time is wasteful. Instead, keep a data/images/ directory with DVC tracking the whole folder, but use a separate small file—like a CSV of image paths and labels—as the actual versioned input. Update that CSV when labels change, and DVC will detect the change and re-run downstream stages. This keeps the expensive image files in the cache while enabling fine-grained versioning of labels.
Feature stores often generate time-series features that are append-only. DVC is not ideal for continuously updating tables; it is designed for snapshots. If you need point-in-time correctness, take periodic snapshots (e.g., nightly) and version those snapshots. On average, a snapshot taken at midnight on April 15 should be good enough for most training jobs. For sub-second consistency, use a dedicated feature store with its own versioning, and have DVC point to the snapshot ID rather than the live table.
Object storage is cheap but egress bandwidth is not. DVC supports file-level deduplication and hard-link caching on local disk, allowing you to switch between different dataset versions without re-downloading. On a shared machine, you can enable a local .dvc/cache and set cache.type to reflink, copy to leverage copy-on-write filesystems like APFS or XFS. For single large files, DVC uses content addresing at the file level, so if you only change one file in a directory of a thousand, only that file is re-uploaded or re-downloaded.
However, DVC does not automatically handle incremental updates at the byte level for a single file. If your training data is one huge TFRecord file that changes slightly every day, DVC will treat the whole file as new. To work around this, split large files into shards or use a directory of smaller files. For example, instead of data/train.tfrecord, use data/shard-*.tfrecord and track the directory. In practice, teams report that sharded files reduce storage costs by up to 70% compared to monolithic files when data updates frequently.
DVC alone gives you data versioning, but pairing it with experiment tracking makes your pipeline fully auditable. When you run a training script, capture the model metrics and hyperparameters in a separate file, and add that file as an output of a stage. For example:
stages:
train:
cmd: python src/train.py --data data/processed/ --out models/model.pkl --metrics metrics.json
deps:
- src/train.py
- data/processed/
outs:
- models/model.pkl
metrics:
- metrics.jsonNow dvc exp run can execute this stage, record metrics, and associate them with the exact data version. You can compare experiments using dvc exp show, which lists metrics along with the data hash. This becomes your shield when someone asks, “Did that accuracy improvement come from the new sampling strategy or from a silent change in the data?”
Use Git tags to mark important checkpoints. For example, after a high-performing run, tag the commit with v1.2.0-churn-model. The tag points to a specific dvc.lock that hashes every dependency and output. A year later, you can clone the repo, check out the tag, and run dvc checkout to retrieve the exact data. If you use DVC with a cloud remote, dvc pull will fetch the necessary files.
Beyond reproducibility, DVC gives you a crude but effective lineage graph. Because dvc.yaml declares dependencies and outputs, you can trace which dataset version fed which model. This satisfies many GDPR and internal governance requirements. However, DVC does not automatically capture column-level lineage or transformations. If a data scientist manually edits a CSV outside a DVC stage, that change is invisible. To enforce strict lineage, run all data processing through DVC stages, and avoid manual edits to tracked files.
For column-level lineage, consider integrating a tool like Great Expectations to validate data expectations at each stage, and store the validation report as a DVC output. That way, you can prove not only which data was used, but also that it met your quality thresholds.
DVC works with Git, but data conflicts are harder to resolve than code conflicts. When two engineers branch from the same commit and both alter dvc.lock, merging can be a nightmare—the lock file is auto-generated and not human-edit-friendly. A pragmatic approach is to avoid long-lived branches for data work. Instead, use a linear main branch and feature branches for code only. When a branch is merged, run dvc repro on the main branch to ensure the data pipeline still works. If you must branch data, use dvc exp save to create experiment checkpoints rather than Git branches.
Another practical tip: use a shared remote cache (e.g., a common S3 bucket) with read-write access for your team. DVC allows multiple users to push/pull from the same remote, and file locking is not needed because each version is immutable. This avoids the classic problem of two people processing the same raw data and generating different intermediate files due to non-deterministic scripts. Always write your scripts to be deterministic (set random seeds, avoid date-time based outputs) to minimize ambiguity.
Object storage costs are driven by storage volume, GET/PUT requests, and egress. DVC's content-addressing means each unique file is stored once per version, so the total storage grows with the number of unique file versions. To minimize costs, prune old versions aggressively. DVC provides dvc gc to collect garbage from the remote cache—orphaned files no longer referenced by any commit. Set up a monthly cron job to run dvc gc --workspace --rev HEAD to keep only files needed for the current and recent commits.
However, be careful with pruning if you need to reproduce older experiments. Balance the cost of storage against the value of long-term provenance. A hybrid approach is to keep full versions for the last three months and snapshots of the final datasets for older experiments, rather than every intermediate file.
Also, consider using S3 Glacier or GCP Nearline for infrequently accessed data. DVC supports storage classes via lifecycle policies. For example, you can transition the cache bucket's objects older than 90 days to Glacier, then use a retrieval script to restore them before dvc pull. This can cut storage costs by 60% or more, at the cost of retrieval latency.
To prevent drift between what was merged and what is in production, integrate DVC into your CI/CD pipeline. For example, in GitHub Actions, after a PR is merged, run dvc repro and dvc push to update the data. You can also validate that the lock file has no uncommitted changes using dvc diff—this ensures that every model is built from tracked data.
A simple CI stage could look like this:
steps:
- name: Pull data
run: dvc pull
- name: Run pipeline
run: dvc repro
- name: Push new artifacts
run: dvc pushIf a stage fails because the data changed unexpectedly, the CI fails, forcing the team to review the change. This gate keeps data drift from silently entering your training pipeline.
One nuance: DVC does not ship a built-in scheduler for periodic data refreshes. You need to trigger it yourself via cron or a workflow engine like Airflow. For that, you can wrap DVC commands in Python or use the dvc.api to make data updates programmatically.
If your data pipeline is heavily based on Spark or Flink, you might be better served by Delta Lake or Hudi, which provide transactional upserts and time travel directly on the data lake. DVC is file-based, and it does not understand row-level changes. For thousands of small parquet files, DVC is fine, but if you need to add or update rows in a table, you'll end up rewriting entire partitions and versioning them—which can be inefficient.
Similarly, if you need seamless integration with notebooks, consider a tool like Dolt, which gives Git-like semantics to SQL tables, or even feature store with built-in versioning. DVC's command-line focus is great for reproducibility, but it assumes you are comfortable with a Git-centric workflow.
Edge cases: DVC struggles with files that are larger than 5 GB on some remotes (HTTPS has upload limits), but S3 multipart upload handles it. Another edge case is a dataset that is a single massive binary file (e.g., a 50 GB model checkpoint). DVC stores it as one object, which is fine, but note that DVC does not chunk the file internally, so re-uploading a changed version requires transferring the whole file. If your checkpoints change frequently, consider storing them on a filesystem with reflink support (e.g., ZFS) and use DVC only for the final artifact.
Start by introducing DVC on a single project where reproducibility pain is highest. For the first week, merely track the raw data and a simple processing stage. Commit dvc.yaml and dvc.lock to Git, and make sure your teammates can dvc pull and get identical data. In the second week, add a training stage and record metrics. In the third week, set up a shared remote and a CI integration. By the end of the month, you will have a working audit trail for that pipeline.
For teams that have no versioning today, even the basic setup pays for itself when someone asks “What changed?”. Do not try to adopt all features at once—start by versioning the data, then layer on pipeline stages. Pair DVC with a lightweight experiment tracker like MLflow if you want dashboards, but DVC's own metrics com
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse