Model Training

2 articles

Artificial Intelligence 03 Sep 2026 9 min read

Understand Knowledge Distillation for Smaller AI Models

A large model may produce useful predictions but still be too expensive or slow for the environment where it must run. A mobile application, an edge device, or a high-volume service can have tighter limits on memory, latency, and compute. Knowledge distillation is one way to address that gap. Instead of training a smaller model only from the original labels, we also train it to imitate information produced by a stronger teacher model. The smaller model is called the student.

Artificial Intelligence Machine Learning

Artificial Intelligence 03 Sep 2026 10 min read

Train Larger AI Models with Gradient Accumulation

Training a neural network often becomes memory-bound before it becomes compute-bound. You may want a batch of 64 examples for stable optimization, but the model, activations, optimizer state, and input tensors leave enough accelerator memory for only 8 examples at a time. Reducing the batch size to 8 may work, but it also changes the optimization process. Gradient accumulation provides another option: process several smaller microbatches, add their gradients together, and update the model only after the desired effective batch has been processed.

Artificial Intelligence Machine Learning