OM

A Complete Guide to Fine-Tune using Hugging Face Transformers

August 2025 · Olame Muhindo · 6 minutes read

Introduction

Natural Language Processing (NLP) has evolved rapidly with the rise of pre-trained Transformer models like BERT, RoBERTa, and GPT. Instead of training models from scratch, we can fine-tune pre-trained models on specific tasks such as sentiment analysis, text classification, or named entity recognition.

In this guide, we'll walk through a step-by-step process to fine-tune a pre-trained model using Hugging Face's transformers library.

Our case study: Sentiment Analysis on the IMDB Movie Reviews Dataset

What We'll Cover

  1. Installing dependencies
  2. Loading and preparing the IMDB dataset
  3. Preprocessing and tokenization
  4. Setting up training arguments
  5. Initializing the model and trainer
  6. Fine-tuning the model
  7. Evaluating performance
  8. Saving and reusing the fine-tuned model

Let's dive in.

Step 1: Install Dependencies

First, ensure you have the datasets and transformers libraries installed:

pip install datasets transformers

Step 2: Load and Explore the Dataset

We'll use Hugging Face's **datasets** library to load the IMDB dataset. This dataset contains 50,000 labeled movie reviews (positive/negative).

from datasets import load_dataset
 
# Load the dataset
dataset = load_dataset("imdb")
 
# Print metadata
print(dataset)

Output:

DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    unsupervised: Dataset({
        features: ['text', 'label'],
        num_rows: 50000
    })
})

Each sample contains a movie review and a label (0 = negative, 1 = positive).

Example review:

print(dataset["train"][0])
{'text': 'I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered "controversial" I really had to see this for myself.<br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br /><br />What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br /><br />I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\'t have much of a plot.',
'label': 0}

Step 3: Preprocess and Tokenize the Data

Before training, we must convert text into numerical tokens using a pre-trained tokenizer.

from transformers import AutoTokenizer
 
# Load BERT tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
 
# Define tokenization function
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)
 
# Apply tokenization
tokenized_datasets = dataset.map(tokenize_function, batched=True)

Example tokenized output:

print(tokenized_datasets["train"][0])

You'll see fields like:

  • input_ids -> tokenized word indices
  • attention_mask -> identifies real tokens vs. padding
  • token_type_ids -> (for sentence pairs)

Step 4: Define Training Arguments

We configure hyperparameters and training settings using TrainingArguments.

from transformers import TrainingArguments
 
training_args = TrainingArguments(
    output_dir="results",   # save models here
    eval_strategy="epoch",  # evaluate after each epoch
    learning_rate=2e-5,      # learning rate
    per_device_train_batch_size=4,  # reduce batch size if limited GPU
    per_device_eval_batch_size=4, # Batch size for evaluation
    num_train_epochs=1,     # for demo, set higher in practice
    weight_decay=0.01,
)

Step 5: Initialize the Model & Trainer

We'll use BERT (bert-base-uncased) for sequence classification with 2 labels (positive/negative).

from transformers import AutoModelForSequenceClassification, Trainer
 
# Load pre-trained model for classification
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
 
# Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"]
)

Step 6: Fine-Tune the Model

Now, let's fine-tune the model on IMDB reviews.

print("---------- Training started----------")
trainer.train()
print("----------Training completed----------")

Output:

Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
 ---------- Training started ----------
  0%|                                                                    | 0/6250 [00:00<?, ?it/s]/Users/galien/Documents/ai/mcp-server/venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py:684: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, then device pinned memory won't be used.
  warnings.warn(warn_msg)
  6%|███▍                                                      | 368/6250 [03:20<56:32,  1.73it/s]                                                                 {'loss': 0.453, 'grad_norm': 0.4327799081802368, 'learning_rate': 1.84032e-05, 'epoch': 0.08}
{'loss': 0.423, 'grad_norm': 52.526954650878906, 'learning_rate': 1.6803200000000002e-05, 'epoch': 0.16}
{'loss': 0.3922, 'grad_norm': 10.000938415527344, 'learning_rate': 1.52032e-05, 'epoch': 0.24}
{'loss': 0.3486, 'grad_norm': 14.510967254638672, 'learning_rate': 1.36032e-05, 'epoch': 0.32}
{'loss': 0.369, 'grad_norm': 198.06076049804688, 'learning_rate': 1.2003200000000002e-05, 'epoch': 0.4}
{'loss': 0.3768, 'grad_norm': 117.08883666992188, 'learning_rate': 1.04032e-05, 'epoch': 0.48}
{'loss': 0.342, 'grad_norm': 0.08970938622951508, 'learning_rate': 8.8032e-06, 'epoch': 0.56}
{'loss': 0.3398, 'grad_norm': 20.2169132232666, 'learning_rate': 7.2032e-06, 'epoch': 0.64}
{'loss': 0.3242, 'grad_norm': 70.76058959960938, 'learning_rate': 5.603200000000001e-06, 'epoch': 0.72}
{'loss': 0.3138, 'grad_norm': 66.68832397460938, 'learning_rate': 4.0032e-06, 'epoch': 0.8}
{'loss': 0.2953, 'grad_norm': 16.9731388092041, 'learning_rate': 2.4032e-06, 'epoch': 0.88}
{'loss': 0.2799, 'grad_norm': 14.353861808776855, 'learning_rate': 8.032000000000001e-07, 'epoch': 0.96}
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6250/6250 [58:43<00:00,  1.98it/s]/Users/galien/Documents/ai/mcp-server/venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py:684: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, then device pinned memory won't be used.
  warnings.warn(warn_msg)
{'eval_loss': 0.26994386315345764, 'eval_runtime': 738.8534, 'eval_samples_per_second': 33.836, 'eval_steps_per_second': 8.459, 'epoch': 1.0}
{'train_runtime': 4263.8654, 'train_samples_per_second': 5.863, 'train_steps_per_second': 1.466, 'train_loss': 0.3548667651367188, 'epoch': 1.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6250/6250 [1:11:03<00:00,  1.47it/s]
 ---------- Training completed ----------

Step 7: Evaluate the Model

We can now test the model's performance.

results = trainer.evaluate()
print(results)

Expected output includes:

  • eval_loss
  • eval_runtime
  • eval_samples_per_second

The lower the loss, the better the model is performing.

Output:

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6250/6250 [12:46<00:00,  8.15it/s]
{'eval_loss': 0.6899533867835999, 'eval_model_preparation_time': 0.0011, 'eval_runtime': 745.346, 'eval_samples_per_second': 33.541, 'eval_steps_per_second': 8.385}

Step 8: Save the Fine-Tuned Model

Finally, save both the model and tokenizer for future inference.

model.save_pretrained("./fine-tuned-model")
tokenizer.save_pretrained("./fine-tuned-model")

Now, you can load this model later with:

from transformers import pipeline
 
sentiment_model = pipeline("sentiment-analysis", model="./fine-tuned-model", tokenizer="./fine-tuned-model")
print(sentiment_model("The movie was absolutely fantastic!"))

Final Thoughts

In this tutorial, we:

  • Loaded the IMDB dataset using Hugging Face Datasets
  • Preprocessed and tokenized text with BERT's tokenizer
  • Fine-tuned a pre-trained Transformer for sentiment analysis
  • Evaluated and saved the model

This workflow can be adapted for other datasets and tasks like spam detection, emotion classification, or topic modeling.

Pro Tip: For better results, try:

  • Training for 3-5 epochs
  • Using a larger batch size (if GPU allows)
  • Experimenting with models like RoBERTa, DistilBERT, or DeBERTa

You now have a production-ready fine-tuned model built with Hugging Face!

A Complete Guide to Fine-Tune using Hugging Face Transformers