domains / families / science / bionemo-kermt-add-cmim-pretrain

bionemo-kermt-add-cmim-pretrain

separate

Convert a grover_base checkpoint (encoder-only or encoder + vocab heads) into a hybrid checkpoint by adding a randomly-initialized cMIM decoder + latent_dist, then continue pretraining on the user's corpus as hybrid (vocab + contrast). Effectively kermt-continue-pretrain with a one-time ckpt-conversion step prepended.

When to use

• Extending grover_base checkpoint to hybrid
• Adding SMILES-reconstruction contrastive objective
• Upgrading encoder-only or vocab-head checkpoints
• Continuing pretraining as hybrid (vocab + contrast)

When not to use

• Continuing existing hybrid or cmim checkpoints
• Training a fresh model on a custom corpus
• Checkpoints with existing contrast heads
• Checkpoints with task FFN heads

Skill page

Source