Installation
pip install transformers
Install the transformers library
pip install transformers[torch]
Install transformers with PyTorch
pip install transformers[tf]
Install transformers with TensorFlow
Loading Models
from transformers import AutoModel, AutoTokenizer
Import basic model and tokenizer classes
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
Load a pre-trained tokenizer
model = AutoModel.from_pretrained('bert-base-uncased')
Load a pre-trained model
pipeline = pipeline('sentiment-analysis')
Load a pre-configured pipeline for a task
Tokenization
tokenizer.tokenize('Hello world')
Tokenize text into subwords
encoded_input = tokenizer('Hello world', return_tensors='pt')
Encode text for model input (PyTorch)
encoded_input = tokenizer('Hello world', return_tensors='tf')
Encode text for model input (TensorFlow)
tokenizer.decode(token_ids)
Convert token IDs back to text
Inference
outputs = model(**encoded_input)
Run inference with a model
prediction = pipeline('Hello world')
Run inference with a pipeline
outputs = model.generate(input_ids)
Generate text with a language model
Fine-tuning
from transformers import Trainer, TrainingArguments
Import training classes
trainer = Trainer(model=model, args=training_args, train_dataset=train_dataset)
Create a trainer
trainer.train()
Start training
model.save_pretrained('./my-fine-tuned-model')
Save a fine-tuned model
tokenizer.save_pretrained('./my-fine-tuned-model')
Save the tokenizer with the model
Model Hub
from huggingface_hub import login
Import login function
login()
Login to Hugging Face Hub
model.push_to_hub('my-model')
Upload model to Hugging Face Hub
tokenizer.push_to_hub('my-model')
Upload tokenizer to Hugging Face Hub