Quick definition
Quantization reduces numerical precision to speed up inference.
- Category: Deployment
- Focus: hosting and runtime tradeoffs
- Used in: Running local models for privacy-sensitive workflows.
What it means
It can cut memory use with minimal quality loss. In deployment workflows, quantization often shapes hosting and runtime tradeoffs.
How it works
Deployment choices include cloud APIs, local inference, or hybrid setups. Each option trades off privacy, cost, and performance.
Why it matters
Deployment choices affect privacy, performance, and cost.
Common use cases
- Running local models for privacy-sensitive workflows.
- Using managed APIs for fast iteration and scaling.
- Hybrid setups that keep data local but call cloud models.
Example
Convert weights from 16-bit to 8-bit.
Pitfalls and tips
Local deployments require hardware planning and updates. Cloud deployments require governance and cost control.
In BoltAI
In BoltAI, this appears in provider, hosting, or local model settings.
