AI Reports · Amazon · 2026-10-22 · Blog / commentary
Stop guessing auto scaling targets for TTS on Amazon SageMaker AI
Teams deploying custom text-to-speech models on Amazon SageMaker AI often guess at autoscaling thresholds, and a wrong guess either wastes GPU spend or breaches a latency SLO. The post deploys Kokoro-82M as a custom container on a SageMaker AI real-time endpoint using AWS CDK stacks and derives a measured autoscaling policy from two live-endpoint measurements: Q_max, the highest concurrency one instance sustains before breaching the SLO, and T_total, how long a scale-out takes end to end. From those two numbers it derives the scale-out threshold, scale-in threshold, and both cooldowns by arithmetic, with every number coming from an actual deployment and measurement run.