Advancing Neural Network Optimization and Design Through the Lens of Continuation Methods and Iterative Dynamical Systems

Pathak, Harsh

Etd

Advancing Neural Network Optimization and Design Through the Lens of Continuation Methods and Iterative Dynamical Systems

Public Deposited

Deep learning has achieved remarkable success across many domains, such as natural language and computer vision. Neural network design and optimization remain pivotal steps in shaping its performance. We study and apply the principles of continuation methods to make the standard optimizers, such as Adam and RMSProp, more efficient. According to the continuation methods, instead of directly solving a non-convex problem, it can be broken down into a series of problems with increasing complexity from convex (simple) to non-convex using only one additional parameter during neural network training. We devise efficient training optimizers by leveraging Adam to demonstrate accelerated convergence and improved generalization across multiple tasks. Despite the success of skip connections in deep neural networks like ResNet and Transformers, there is a lack of formalization in understanding and implementing these crucial architectural elements. In particular, the idea of weighted skip connections is limited in the deep learning literature. We advance this research by recasting various neural network architectures as iterative dynamical systems. This perspective led to Sequential2D, a novel two-dimensional framework for deep neural networks that leverages systematic skip-connections. Sequential2D provides a formalized approach to implementing and understanding skip connections in neural architectures. Finally, we combine the continuation methods and Sequential2D techniques to devise a parameter-efficient fine-tuning technique called Solo-Connection. This technique outperforms LoRA (state-of-the-art) with a 40% parameter reduction across many natural language generation benchmarks.

Creator