TALOS-V2: Implementing Transformer Architecture as Pure Hardware Circuitry on FPGA to Achieve Over 5
Luthira Abeykoon and Krish Chhajer, Electrical and Computer Engineering undergraduates at the University of Toronto, have successfully ported Karpathy's MicroGPT—a minimalist GPT implementation consisting of only 200 lines of pure Python code and<!--autointro-->
The system runs on a DE1-SoC Cyclone V, an educational-grade IntelFPGA. Key architectural decisions include:
Weight Storage: Model weights are stored in on-chip ROM using a Q4.12 fixed-point format.
Systolic Array: The repetitive Matrix-Vector Multiplication (MatVec) operations found throughout the model are implemented as a 16-channel systolic array. This single unit is time-shared to handle Q/K/V projections, the Multi-Layer Perceptron (MLP), and the Language Model (LM) head.
Overcoming the Attention Mechanism Challenge
The attention mechanism proved to be the most complex component to migrate from software to hardware. While a single line of Python code suffices for attention in a software environment, the hardware implementation required decomposing the process into eight distinct steps:
The creators emphasize that the primary goal of TALOS-V2 was not to run Large Language Models (LLMs), but rather to visualize every step of Transformer inference as tangible hardware components: memory units, counters, state machines, and Look-Up Tables (LUTs).
Comments & Questions (0)
No comments yet
Be the first to comment!