Efficient and Sustainable Neural Architecture Search

Zhao, Yiyang

Etd

Efficient and Sustainable Neural Architecture Search

Pubblico Deposited

Artificial intelligence (AI) now plays an indispensable role not only in the computer science domain but also in people's everyday life. AI solutions have greatly outperformed conventional methods in many real-world tasks and problems, such as image classification, object detection, image segmentation, and question answering. However, the design of AI models and systems still remains primarily in the hands of domain experts, which largely restricts the development and spread of AI. This thesis seeks to design an AI pipeline to automate the production line of AI, thereby removing this restriction efficiently and sustainably from three perspectives. (1) Multi-Objective Neural Architecture Search (NAS) Algorithm. With the advancement of computing power, deep neural networks have become the mainstream models in the AI research community and industry community. Neural Architecture Search has proven to be effective in automating the design of neural networks in various studies. However, previous NAS works have mainly focused on designing models with good performance metrics such as accuracy and mAP, while neglecting other important factors such as the number of parameters, FLOPS, power cost, and latency. In this thesis, we propose to design an effective multi-objective search algorithm for NAS, allowing it to consider multiple factors during the design process. (2) Efficient Network Evaluations. The vanilla NAS approach requires training all the searched/sampled neural architectures from scratch to evaluate their performance, incurring a significant amount of time and computational costs, often hundreds to thousands of GPU days. One-shot NAS addresses the high evaluation cost in vanilla NAS by employing a weight-sharing technique, which eliminates the need to retrain sampled networks from scratch. However, numerous studies have shown that one-shot NAS often results in degraded search performance due to inaccurate architecture performance predictions. In order to tackle the shortcomings of both vanilla and one-shot NAS, we propose a few-shot NAS approach that leverages multiple super-nets as proxies to accurately estimate the performance of models in a shorter amount of time. (3) Sustainable Neural Architecture Search. Energy costs and carbon emissions are major environmental concerns in NAS. Prior research reports that a single architecture search by NAS can produce as much carbon emissions as five cars' lifetimes. Despite many efforts to reduce energy costs in NAS, the carbon emissions can still vary with energy generation methods(e.g., solar vs. fuel) and locations. Therefore, we propose an adaptive carbon-aware NAS framework that reduces carbon consumption during the search while maintaining good search performance.

Creator