Ideas

GPU & Computer Architecture

Notes and projects on GPU programming and computer architecture: CUDA kernels, memory hierarchy, warp scheduling and simulation, and building a CPU from logic gates up.

Low-level work, starting from logic gates. The nand2tetris projects build a working computer up from NAND gates and a VM translator. The GPU course goes the other direction into CUDA: tiled kernels, a warp scheduler simulated cycle by cycle, and reading compiled SASS to find branch divergence.

Most of it comes down to the same question: what is the hardware actually doing? The answer usually involves memory hierarchy, scheduling, and divergence.

GPUCUDAcomputer-architecture
Thread Retro 4 parts Updated
Green ripples abstract art
Retro Complete 29 min read

GPU Hardware and Software

What I Learned in GPU Hardware and Software (CS 8803) - A Retrospective
CUDA Fundamentals: Tiled Matrix Multiply & Bitonic Sort
Project Complete 17 min read

CUDA Fundamentals: Tiled Matrix Multiply & Bitonic Sort

Writing real GPU kernels, exploring shared memory tiling, parallel sorting algorithms, and performance optimization on an NVIDIA H100.
FlashAttention & LLM Inference on GPUs
Project Complete 19 min read

FlashAttention & LLM Inference on GPUs

Writing a FlashAttention CUDA kernel from scratch, tiling the attention matrix to avoid materializing N×N memory, building a KV cache for token generation, and running GPT-2 with custom kernels end-to-end.
GPU Simulation: Warp Scheduling & Compute/Tensor Cores
Project Complete 17 min read

GPU Simulation: Warp Scheduling & Compute/Tensor Cores

Building a cycle-level GPU simulator from the inside, implementing GTO and CCWS warp schedulers, modeling compute instruction latencies, and extending to tensor core simulation.
Static Analysis: Detecting Branch Divergence in GPU Code
Project Complete 15 min read

Static Analysis: Detecting Branch Divergence in GPU Code

A simple if statement can halve GPU throughput, exploring how static dataflow analysis detects branch divergence by building a def-use chain over a SASS control flow graph.