Technology Radar
FreeToken is an open-source inference engine designed to run large open-weight mixture-of-experts (MoE) models on consumer hardware. As these models become more capable, their full weights can exceed the memory limits of consumer GPUs. Unlike general-purpose local model runners, FreeToken specializes in oversized MoE models by adaptively combining GPU memory, CPU execution, host memory and PCIe bandwidth. It dynamically caches active experts on the GPU and swaps them based on which experts are activated. By coordinating this process, FreeToken enables developers to run much larger models on existing or older hardware. We’re starting to see MoE offloading techniques incorporated into broader tools such as llama.cpp, which is used by Ollama, but FreeToken offers a specialized approach worth assessing.
