awq-quantization

This skill quantizes LLMs to 8-bit or 4-bit precision using bitsandbytes, achieving 50-75% memory reduction with minimal accuracy loss. It's ideal for running larger models on limited GPU memory or accelerating inference, supporting formats like INT8, NF4, and FP4. The skill integrates with HuggingFace Transformers and enables QLoRA training and 8-bit optimizers.

This skill enables GGUF quantization for efficient model deployment on consumer hardware like CPUs and Apple Silicon. It provides flexible 2-8 bit quantization options without requiring GPU acceleration. Use it when optimizing models for local inference tools or resource-constrained environments.

SGLang is a high-performance LLM serving framework that uses RadixAttention for automatic prefix caching, enabling significantly faster structured generation. It's ideal for developers needing JSON/regex outputs, constrained decoding, or building agentic workflows with tool calls. Use it when you require up to 5× faster inference than alternatives like vLLM in scenarios with shared prefixes.

HQQ enables fast, calibration-free quantization of LLMs down to 4/3/2-bit precision without needing a dataset. It's ideal for rapid quantization workflows and deployment with vLLM or HuggingFace Transformers. Key advantages include significantly faster quantization than methods like GPTQ and support for fine-tuning quantized models.