Skip to main content

Automatically Integrate Huggingface Models

· 2 min read

Checkpoints can now be created directly from public or private Hugging Face models.

How It Works​

To integrate with Hugging Face, first create a User Access Token with these instructions. If you plan to only download data, create a read token. If you plan to upload results back to huggingface, create a write token. Once you have the token, go back to the trainML third-party key configuration page, and select Hugging Face from the Add menu under Third-Party Keys. Enter the your Hugging Face account name as the Key ID and the generated token as the Key Secret and click the check button.

Once the third party key is added, select Hugging Face as the Checkpoint source type. Use the repo name as the source uri in the format <namespace>/<repo>. By default, checkpoint creation will automatically download the default branch and remove all other branches and git history to save space. To create a checkpoint from a different branch, specify that in the Branch field.

Using the SDK​

To create a Hugging Face checkpoint using the SDK, use the following syntax.

checkpoint = await trainml.checkpoints.create(
    name="System Tests - Hugging Face Checkpoint",
    source_type="huggingface",
    source_uri="<namespace>/<repo>",
    source_options=dict(branch="branch_name")
)