Analysis technique
We evaluated our CDR-aware masking methods (WC, CDR, or hybrid) in two complementary methods. First, we assessed region-stratified masked restoration efficiency to point out that CDR-focused coaching efficiently improves illustration of those hypervariable areas whereas sustaining framework accuracy. Whereas these masked-recovery metrics reveal that our strategy learns improved CDR representations, in addition they function proxy measures of total mannequin high quality. As such, we then assessed whether or not the CDR-focused illustration enhancements translate to practical utility within the downstream binding affinity prediction process. This analysis confirmed that CDR-optimized embeddings considerably enhance prediction of antibody–antigen binding throughout numerous targets, establishing each the validity of our coaching strategy and its sensible worth for antibody engineering.
Determine 1 summarizes the general pipeline, from VH/VL sequence inputs and region-specific masking insurance policies to embedding extraction and linear regression used all through this examine. Full particulars on datasets, masking insurance policies, mannequin coaching, analysis protocols, and statistical analyses are supplied in Part 4.
A Antibody construction and area annotation. The Y-shaped antibody includes paired heavy (VH) and lightweight (VL) variable domains that type the antigen-binding web site. Every area accommodates 4 framework areas (FR1–4) offering structural stability and three complementarity-determining areas (CDRs 1–3) that instantly contact antigens. B Two-stage curriculum studying pipelines. ESM2-3B follows an elective unpaired pretraining step, then Stage I whole-chain adaptation, and Stage II refinement with both CDR-focused or hybrid methods. The 80% and 20% ratios within the hybrid masking point out the proportion of coaching samples receiving CDR or WC masking inside every batch, respectively. ESM C follows a less complicated pipeline with both direct CDR coaching or a two-stage WC → CDR curriculum. C Embedding extraction and binding affinity prediction. Wonderful-tuned fashions encode paired VH–VL sequences into fixed-dimensional embeddings by imply pooling throughout all positions. Ridge regression with L2 regularization maps these representations to predicted dissociation constants (OkD) for antibody–antigen pairs.
Area-stratified masked restoration and trade-offs
The bottom ESM2-3B mannequin exhibits pronounced efficiency disparities between framework and CDR areas (Fig. 2A, left), with framework areas attaining 72–92% restoration whereas CDRs, significantly the hypervariable HCDR3 (35.69%) and LCDR3 (46.06%), carry out considerably worse. This gradient displays the inverse relationship between sequence conservation and practical variety in antibody recognition.
A ESM2-3B efficiency throughout areas. Left: whole-chain masking (WC-15%) take a look at exhibiting per-region top-1 restoration accuracies for base ESM2-3B, Stage I whole-chain fine-tuning (WC, epoch 5), Stage II Hybrid masking (epoch 3), and Stage II CDR-only masking (epoch 3). Proper: CDR-specific masking (CDR-50%) take a look at with frameworks unmasked. B, C Pareto throughout masking distributions. B FR(avg) vs. CDR(avg) accuracy on WC-15% take a look at throughout Stage II coaching epochs (e = 1–10) ranging from Stage I WC (epoch 5, marked by dashed crosshairs). C Joint analysis: x-axis exhibits CDR accuracy on CDR-50% take a look at (frameworks unmasked), y-axis exhibits FR accuracy on WC-15% take a look at. D ESM C (600M) efficiency comparability. Identical structure as panel A for ESM C fine-tuned fashions. E, F Unpaired pretraining ablation. E Heavy and light-chain framework and CDR macro accuracies on WC-15% take a look at. F HCDR3, LCDR3, and averaged CDR accuracies on CDR-50% take a look at. Each panels evaluate Base ESM2, Unpaired-only pretraining, Unpaired+Paired-WC sequential coaching, and Stage I WC (paired-only). Bars characterize imply accuracy; error bars characterize 95% bootstrap confidence intervals. Knowledge characterize imply of n = 2 unbiased analysis seeds (totally different random masking patterns), with confidence intervals computed from 2000 bootstrap replicates per seed, resampling from n = 20, 225 biologically unbiased take a look at sequences.
Complete-chain fine-tuning considerably improves each areas, with frameworks approaching saturation (97.57% common) and CDRs bettering by 35 proportion factors (pp) to 85.65%. HCDR3 stays most difficult (62.38%) regardless of a 75% relative acquire. This result’s in keeping with V(D)J recombination, wherein stochastic becoming a member of of V, D, and J gene segments generates considerably better sequence variety than the V–J becoming a member of that offers rise to the opposite CDRs.
Stage II refinement with CDR-focused or hybrid masking gives extra enhancements. Hybrid masking preserves framework efficiency whereas bettering CDRs (HCDR3: 62.38% → 63.09%, LCDR3: 86.96% → 87.18%), whereas CDR-only masking achieves related CDR positive aspects with minimal framework degradation (−0.10 pp). Beneath CDR-specific analysis the place frameworks present context (Fig. 2A, proper), each Stage II methods outperform Stage I baseline, with CDR-only masking attaining the best HCDR3 restoration (59.03% vs. 57.83% for Stage I).
To characterize the framework–CDR trade-off, we analyze Pareto frontiers throughout Stage II epochs (Fig. 2B, C). Beneath whole-chain analysis, the hybrid department maintains framework stability whereas bettering CDR restoration, whereas CDR-only masking ultimately degrades framework representations, decreasing the mannequin’s potential to leverage framework context for CDR predictions. Beneath CDR-specific analysis (Fig. 2C), Stage II CDR-focused coaching advances additional alongside the CDR axis when frameworks stay unmasked throughout analysis, indicating efficient CDR specialization.
ESM C (600M), regardless of having solely 20% of ESM2’s parameters, demonstrates superior base efficiency throughout all areas (Fig. 2D). Framework areas obtain 82–94% restoration, and CDRs present marked enchancment over ESM2, although HCDR3 stays difficult at 39.41%. WC fine-tuning improves each frameworks (reaching 96–99%) and CDRs (85.41% common), with HCDR3 leaping from 39.41% to 62.30%(+22.89 pp) and LCDR3 growing from 52.52% to 86.69% (+34.17 pp). Direct CDR fine-tuning (Stage I CDR) achieves related CDR efficiency (85.16% common) however degrades framework accuracy. A two-stage WC–CDR refinement yields solely marginal CDR adjustments whereas additional eroding frameworks, providing no benefit on the WC take a look at. Beneath CDR-specific analysis (Fig. 2D, proper), CDR-focused fine-tuning exhibits its meant impact. Stage I CDR surpasses the WC mannequin and performs comparably to Stage II WC → CDR, regardless of requiring solely half the coaching epochs, suggesting that direct CDR optimization is extra environment friendly than sequential refinement for this structure.
Unpaired pretraining ablation
We subsequent investigated whether or not large-scale unpaired antibody knowledge might present a useful initialization for paired fine-tuning. Earlier work18 has proven that domain-specific pretraining can enhance downstream process efficiency, and the OAS database gives billions of unpaired sequences in comparison with tens of millions in our paired dataset. We subsequently ablated this unpaired pretraining stage by evaluating fashions skilled with and with out this intermediate step.
We pretrained ESM2-3B on single-chain OAS sequences with WC-15% masking, then fine-tuned on paired sequences. Determine 2E, F evaluate masked-recovery efficiency with and with out this unpaired stage. Unpaired-only pretraining yields a considerable bounce over the launched base mannequin throughout all areas, confirming that single-chain antibody knowledge are informative.
Nonetheless, as soon as we fine-tune on paired knowledge, beginning both from the unpaired checkpoint or instantly from BASE, the fashions converge: paired-only WC (Stage I) matches or barely exceeds the unpaired → paired variant. The unpaired pretraining stage gives no measurable profit for masked restoration and didn’t enhance downstream binding affinity prediction (Desk S1), whereas including substantial compute and pipeline complexity.
Complete comparability with antibody-specific masked-recovery baselines
We in contrast our Stage II fashions with revealed antibody-specific baselines, together with IgBERT and IgT5, on masked restoration. Direct cross-paper comparisons are approximate as a result of the take a look at splits differ; nevertheless, each research use equally sized take a look at units (~20k sequences) derived from OAS, enabling indicative comparisons.
On the WC-15% take a look at (Desk 1), our fashions obtain efficiency akin to IgBERT and IgT5 throughout most areas, with constant enhancements in HCDR3 and LCDR3 in all our fashions. Given potential knowledge break up variations, we interpret these outcomes as demonstrating that CDR-focused fine-tuning from common protein fashions can match the masked-recovery efficiency of antibody-specific architectures skilled on large unpaired corpora, whereas requiring considerably much less computational assets (8 epochs vs. 46 epochs for IgBERT).
For the CDR-50% take a look at (decrease panel of Desk 1), we evaluated revealed baselines on our take a look at break up since this analysis protocol was not included of their unique work. As a result of our take a look at break up might overlap with their coaching knowledge, the reported baseline values characterize an higher certain on their efficiency underneath this analysis. Regardless of this potential benefit, our Stage II CDR fashions obtain the best efficiency throughout all CDR areas (74.87% HCDR, 89.88% LCDR), demonstrating efficient CDR specialization.
These masked-recovery outcomes motivated the downstream practical prediction analyses in Part “Embedding high quality and switch to antibody binding affinity prediction”, the place CDR-optimized representations present the strongest proof of improved antibody modeling by positive aspects in binding affinity prediction.
Embedding high quality and switch to antibody binding affinity prediction
To evaluate whether or not this region-stratified masking pipeline results in improved practical prediction, we chosen fashions for downstream embedding analysis primarily based on validation set efficiency. We selected checkpoints that: (a) achieved optimum masked CDR restoration, (b) demonstrated Pareto optimality between FR and CDR restoration, and (c) represented the earliest epoch assembly these standards to forestall overfitting to the masked language modeling goal. The chosen fashions have been ESM2 Stage II CDR epoch 3 and ESM C Stage I CDR epoch 4, hereafter known as AbCDR-ESM2 and AbCDR-ESMC, respectively.
The embedding high quality of AbCDR-ESM2 and AbCDR-ESMC was evaluated on three therapeutic antibody binding datasets (Desk 2) from FLAb8, the place every dataset measured binding affinities of single-residue mutants. We observe the analysis protocol established by18 and evaluate in opposition to their reported baseline efficiency. Our reproduced AntiBERTy and AbLang outcomes carefully matched these revealed values, confirming the reliability of our analysis protocol. We moreover benchmarked AbLang220, which was not beforehand evaluated on these datasets.
In comparison with their base counterparts, our antibody-adapted fashions persistently outperformed throughout all benchmarks; ESM2 adaptation improved R2 from 0.302 to 0.359 on D44 (+18.8%), from 0.264 to 0.298 on G6 (+12.8%), and modestly elevated Trast (0.335 → 0.350; +4.4%).
The compact AbCDR-ESMC mannequin was extremely environment friendly: focused adaptation yielded the second-highest R2 on D44 (0.345) and highest R2 on G6 (0.313), surpassing the 3B-parameter IgT5 by 16% and 25%, respectively, regardless of utilizing 5x fewer parameters and solely paired fine-tuning. This validates that domain-specific fine-tuning advantages fashions throughout totally different scales and architectures.
In comparison with different antibody-specific fashions, our strategy achieved superior efficiency with out large unpaired pretraining. Whereas IgBert, AbLang2, and IgT5 leverage billions of unpaired sequences, our tailored fashions attained 20–170% increased R2 with simply 8 epochs of paired coaching for ESM2 (5 WC + 3 CDR) or 4 epochs of direct CDR coaching for ESM C. Notably, AbLang2 achieved the best R2 on Trast (0.460), the one case wherein a broadcast mannequin surpassed our tailored fashions. Nonetheless, Trast can be the smallest dataset (n = 422) with the best cross-validation variance.
Curiously, base ESM2 already matched or exceeded IgT5 throughout datasets, and base ESM C did so on D44 and G6 (however not Trast). The constant positive aspects from CDR-focused fine-tuning, starting from ~4–20% for ESM2 and as much as 73% for ESM C on Trast, point out that strategic masking on high-quality paired knowledge gives advantages orthogonal to mannequin scale.
Generalization to combinatorial mutant prediction
To guage generalization past single-mutant binding affinities, we prolonged our analysis of AbCDR-ESM2 and AbCDR-ESMC to incorporate datasets with each single and combinatorial antibody mutants throughout three numerous antigen programs (Desk 3). We evaluated all baseline fashions utilizing the embedding extraction and ridge regression protocol described in Part “Embedding extraction and downstream heads”.
Our tailored fashions demonstrated constant enhancements throughout all datasets, with significantly putting positive aspects on the anti-Fluorescein set the place AbCDR-ESM2 improved R2 from 0.547 to 0.693, a 26.6% relative enhance in comparison with the bottom mannequin, and from 0.531, a 30.5% relative enhance in comparison with the perfect revealed mannequin, AbLang. This dataset’s concentrate on multi-site CDR mutations might significantly profit from our CDR-focused coaching, as these higher-order CDR correlations are discovered throughout 50% CDR masking.
On the large-scale anti-HR2 SARS-CoV-2 dataset (n = 71,830), AbCDR-ESM2 achieved R2 = 0.396, surpassing the perfect revealed mannequin, IgT5, by 8.8% whereas decreasing MAE from 0.865 to 0.841. ESM C confirmed extra modest positive aspects, probably as a result of its base efficiency already exceeded all revealed baselines. This sample (stronger base fashions exhibiting smaller however constant enhancements) recurred throughout datasets, suggesting that our adaptation gives orthogonal advantages to architectural advances.
For H1 Hemagglutinin, tailored AbCDR-ESM2 achieved the best R2 (0.411) however had an nearly equal Spearman correlation (0.758) to AbCDR-ESMC. The constant enhancements throughout antigens with totally different binding modes (HR2 area of SARS-CoV-2 spike protein, small-molecule fluorescein, hemagglutinin), each with respect to the bottom mannequin and main revealed fashions, reveal that the two-stage CDR-focused coaching captures generalizable patterns fairly than target-specific biases.
Complete embedding comparability
AbCDR-ESM2 and AbCDR-ESMC have been chosen as the perfect ESM2 and ESM C primarily based fashions in an unsupervised method, primarily based on superior masked CDR restoration whereas remaining Pareto optimum with respect to FR restoration. We retrospectively evaluated this choice technique by systematically evaluating downstream embedding efficiency throughout MLM methods and coaching epochs. To generate sturdy estimates of imply efficiency, 10-fold cross-validation efficiency was averaged over 40 random seeds per epoch.
The plots in Fig. 3 present modest settlement throughout datasets between the number of Stage II CDR epoch 3 for ESM2 (AbCDR-ESM2) and Stage I CDR epoch 4 for ESM C (AbCDR-ESMC) and the utmost embedding efficiency throughout epochs. These outcomes help the Pareto-based criterion as a sensible unsupervised technique for checkpoint choice. Specifically, this helps keep away from noticed embedding degeneration in later epochs of coaching, because the dilution of broad sequence-level options in favor of CDR options begins to worsen downstream embeddings.
R2 efficiency of embeddings of every MLM technique calculated per epoch by way of 10-fold cross-validation break up repeated over 40 random seeds per epoch. A–F ESM2 fashions on datasets similar to single-mutant units (D44, G6, and Trastuzumab) and combinatorial mutant units (Covid, Fluorescein, H1 Hemagglutinin). G–L ESM C fashions on datasets similar to single-mutant units (D44, G6, and Trastuzemab) and combinatorial mutant units (Covid, Fluorescein, H1 Hemagglutinin). Error bands characterize the usual error of the imply (SEM) throughout 40 random seeds.
We additionally confirmed that the good thing about CDR masking is unbiased of coaching time and mutational distance. AbCDR-ESM2 generated a relative enchancment of 0.76%, 9.65%, 9.12%, and three.98% over the perfect epoch of WC mannequin for the single-mutant units of G6, D44 and the combinatorial mutant units of anti-Fluorescein and anti-HR2 SARS-CoV-2, respectively. Equally, AbCDR-ESM2 outperforms Stage II Hybrid for 3 datasets with matching efficiency, inside customary error, for the remaining three.
For AbCDR-ESMC, we noticed a extra pronounced impact; in some circumstances, involvement of WC masking (D44 and G6) led to worse efficiency than the bottom mannequin. AbCDR-ESMC led to a relative acquire of 16.1%, 25.7%, and 1.0% over the perfect epoch of WC masking for G6, D44, and anti-Fluorescein, respectively, with matching efficiency for anti-HR2 SARS-CoV-2 and Trast.
Notably, there are dataset exceptions, as WC masking outperforms any CDR masking technique for anti-H1 HA whereas Trast exhibits no distinction in any masking technique. Nonetheless, these units are noticeably smaller (n = 1038 and n = 422), which limits confidence in whether or not the noticed imply displays the true underlying distribution for these two antibodies. Other than anti-H1 HA, CDR masking methods from both base ESM fashions, achieved superior binding prediction in comparison with conventional WC masking methods.
