miniGPT

A compact, educational implementation of a Generative Pre-trained Transformer (GPT) model built from scratch in PyTorch. This project serves as a learning experiment to understand the inner workings of Transformer architectures, self-attention mechanisms, and Large Language Model (LLM) training pipelines.

Project Overview

This repository contains:

• Core Transformer Architecture: Implementation of Self-Attention, Decoder Blocks, and the full Transformer model.
• Training Pipeline: Scripts to train the model on provided text datasets using mixed precision (autocast).
• Inference: A testing script to generate text based on user prompts.
• Data Processing: Utilities to clean and prepare raw text data (e.g., books) for training.

Note: This is a learning project. The model was trained on a single NVIDIA RTX 2070 Super, which limited the model size and training duration.

Results & Limitations

• Current Model Version: model7_geminiVersion
• Training Hardware: RTX 2070 Super
• Performance:
  • The model successfully learned to generate coherent sentences and structures.
  • However, due to limited VRAM and training time, it struggles with long-term coherence and maintaining context over extended passages.
  • It is recommended to retrain on a larger dataset and more powerful hardware for better results.

Books Used for Training

The following books were used to create the dataset:

• 1984
• Animal Farm
• Eragon (Series)
• Harry Potter (Series)
• Lord of the Rings (Series)

Directory Structure

• src/: Source code for the model, training, and data processing.
• data/: Dataset used for training (excluded from repo due to size).
• models/: Saved model checkpoints (excluded from repo due to size).

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/miniGPT.git
   cd miniGPT

2. Install dependencies:

   pip install torch tokenizers

Usage

1. Prepare Dataset

If you have raw text files in data/booksOfDataset, run:

cd src
python makedataset.py

This will generate finalDataset2.txt in the data/ directory.

2. Train the Model

To start training:

cd src
python train.py

This will save checkpoints and the final model to the models/ directory.

3. Generate Text

To test the model:

cd src
python testing.py

Future Work

• Implement more efficient attention mechanisms (e.g., Flash Attention).
• Scale up the model parameters.
• Train on a more diverse dataset (e.g., OpenWebText or C4).