Speaker
Description
How much exposure does a language model need before it reliably learns a new entity, and how can this process be measured experimentally on large-scale HPC infrastructure? We present the computational design of a dissertation experiment focused on entity learnability in large language models, using fictitious brands and their attributes as a controlled case. The experiment uses continued pretraining on the EuroHPC LUMI system.
The scientific objective is to measure how information about a previously unseen entity is acquired by a model as a function of exposure volume, source type, and distribution. From an HPC perspective, this also creates an orchestration problem: a large matrix of comparable training runs must be generated, scheduled, monitored, checkpointed, validated, and evaluated while preserving tightly controlled experimental conditions.
For each condition, synthetic corpora are constructed to represent different source types, including entity-owned websites, catalogues, independent editorial content, press releases, user discussions, and texts containing contradictory or denying statements. The text is generated through a frontier LLM API and then passed through a mechanical validation layer. Validated content is inserted into a fixed reference corpus at controlled token positions, allowing the total training volume, cumulative entity exposure, and exposure distribution to be varied independently.
Selected open language models then undergo continued pretraining on LUMI GPU accelerators. A central design issue is checkpoint comparability. Conventional checkpointing by training step ties measurement points to batch configuration and parallelization strategy. Instead, checkpoints are scheduled by cumulative exposure to the target entity. Measurement points are therefore defined by experimental meaning rather than by a particular compute configuration, making runs directly comparable across execution geometries.
The poster focuses on the engineering required to make such an experimental campaign reproducible and operationally manageable on shared HPC infrastructure. Corpus construction is deterministic, with cryptographically fingerprinted shards and explicit manifests describing corpus composition and entity placement. Configurations, datasets, checkpoints, and derived artifacts are fingerprinted in the same way. Saved checkpoints are automatically verified before downstream processing to detect incomplete or corrupted artifacts early.
Training throughput is calibrated empirically across micro-batch and parallelization configurations rather than estimated from nominal accelerator performance. This supports resource planning based on observed performance and enables cost comparisons across experimental cells.
The pipeline spans CUDA and ROCm environments, providing practical insights into portability across containers, libraries, numerical precision, memory behavior, and job configuration. Monitoring uses a pull-based architecture without persistent cluster-side services: jobs emit structured state and metrics that are aggregated externally for progress tracking and failure detection.
The study is preregistered on OSF, and the methodology and code are intended to be public. Beyond the specific application to brand entities, the project serves as a case study in reproducible orchestration of controlled continued-pretraining experiments on shared HPC infrastructure.