This academic paper addresses the inherent challenges in training Recurrent Neural Networks (RNNs), specifically the vanishing and exploding gradient problems. The authors explore these issues from analytical, geometrical, and dynamical systems perspectives, building upon previous work. They propose and empirically validate a gradient norm clipping strategy to combat exploding gradients and a soft regularization constraint to mitigate vanishing gradients. The research demonstrates that these solutions significantly improve RNN performance on both synthetic pathological tasks requiring long-term memory and natural language processing and music prediction problems.
Source:
https://arxiv.org/pdf/1211.5063
