AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-26
On Orthogonal Recurrent Neural Networks Quantization
Orthogonal recurrent neural networks handle long-term dependencies well but use full-precision weights and activations, which hinders deployment on compact devices. The authors propose quantized approximately orthogonal RNNs (QORNNs), learning them with quantization-aware training combined with orthogonal projections, and apply post-training quantization to activations for pure integer computation of the recurrent loop. The fully quantized models are currently the only alternatives that solve the 1000-timestep Copy-task and outperform existing quantized models on permuted MNIST and some IoT tasks, though they are less effective than gated RNNs such as LSTMs on other tasks.