Hugging Face Transformers

Installation

pip install transformers
Install the transformers library

pip install transformers[torch]
Install transformers with PyTorch

pip install transformers[tf]
Install transformers with TensorFlow

Loading Models

from transformers import AutoModel, AutoTokenizer
Import basic model and tokenizer classes

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
Load a pre-trained tokenizer

model = AutoModel.from_pretrained('bert-base-uncased')
Load a pre-trained model

pipeline = pipeline('sentiment-analysis')
Load a pre-configured pipeline for a task

Tokenization

tokenizer.tokenize('Hello world')
Tokenize text into subwords

encoded_input = tokenizer('Hello world', return_tensors='pt')
Encode text for model input (PyTorch)

encoded_input = tokenizer('Hello world', return_tensors='tf')
Encode text for model input (TensorFlow)

tokenizer.decode(token_ids)
Convert token IDs back to text

Inference

outputs = model(**encoded_input)
Run inference with a model

prediction = pipeline('Hello world')
Run inference with a pipeline

outputs = model.generate(input_ids)
Generate text with a language model

Fine-tuning

from transformers import Trainer, TrainingArguments
Import training classes

trainer = Trainer(model=model, args=training_args, train_dataset=train_dataset)
Create a trainer

trainer.train()
Start training

model.save_pretrained('./my-fine-tuned-model')
Save a fine-tuned model

tokenizer.save_pretrained('./my-fine-tuned-model')
Save the tokenizer with the model

Model Hub

from huggingface_hub import login
Import login function

login()
Login to Hugging Face Hub

model.push_to_hub('my-model')
Upload model to Hugging Face Hub

tokenizer.push_to_hub('my-model')
Upload tokenizer to Hugging Face Hub