CausalMix: Data Mixture as Causal Inference for Language Model Training
CausalMix treats data mixing for LLM training as a causal problem instead of a guessing game. It estimates how different data buckets change downstream scores, then picks mixtures based on those estimated effects. If you are curating giant training pools, this gives you a more principled way to choose what to oversample.
Zinan Tang, Yukun Zhang