This October 2024 paper introduces synthetic continued pretraining (synthetic CPT), a novel method designed to enhance language model knowledge acquisition from small, specialized text collections. Current large language models often struggle with data efficiency and learning niche facts from limited sources. The core of this approach is EntiGraph, a synthetic data augmentation algorithm that extracts entities and their relationships from a small corpus to generate a much larger, more diverse synthetic dataset. Experiments using the QuALITY dataset demonstrate that EntiGraph CPT significantly improves a model's ability to answer questions and summarize content about these specialized domains, outperforming direct training on raw data or simple rephrasing techniques. The authors also explore the scaling properties of EntiGraph, revealing distinct phases of knowledge acquisition.
Source:
https://arxiv.org/pdf/2409.07431
