Colibri v2.1.0 adds native GGUF support across CPU, Vulkan and CUDA engines, the project reported October 31.

The big change

  • The maintainers say compatible published GGUF weights run without another conversion or quantization, removing a preparation step when switching to Colibri.
  • A mixture-of-experts model selects only some components for an input, so its total parameter count does not tell the whole story about the work done on each request.
  • Storage and memory tradeoffs shape which models are practical on a given machine.

Why the format matters

Hugging Face’s GGUF documentation describes it as a model format containing tensors and standardized metadata. It supports several numerical representations, including quantized types. Quantization changes how weights are represented, so a published file may already reflect a particular size and precision choice.

Colibri imports compatible files while retaining their existing blocks.

Why streaming has limits

A sparsely gated mixture-of-experts paper describes a network that selects a subset of expert components for each input.

Colibri already read experts from storage as needed rather than keeping them all in memory, according to its documentation; streaming and single-GPU operation predate this release. The tradeoff is that the full model must still live on storage, and retrieving parts as needed makes storage access relevant to responsiveness.

The project’s v2.1 documentation lists 8 GB RAM minimum and 22 GB free disk for its smallest model; larger models need different resources. BIG CHANGE has not independently tested it.

Sources & further reading