Effective methods for deep neural network sparsification

Glentis Georgoulakis, Athanasios; Γλεντής Γεωργουλάκης, Αθανάσιος

dc.contributor.author	Glentis Georgoulakis, Athanasios	el
dc.contributor.author	Γλεντής Γεωργουλάκης, Αθανάσιος	en
dc.date.accessioned	2024-04-22T09:41:18Z
dc.date.available	2024-04-22T09:41:18Z
dc.identifier.uri	https://dspace.lib.ntua.gr/xmlui/handle/123456789/59238
dc.identifier.uri	http://dx.doi.org/10.26240/heal.ntua.26934
dc.rights	Αναφορά Δημιουργού-Μη Εμπορική Χρήση-Όχι Παράγωγα Έργα 3.0 Ελλάδα	*
dc.rights.uri	http://creativecommons.org/licenses/by-nc-nd/3.0/gr/	*
dc.subject	DNN Compression	en
dc.subject	Sparsification	en
dc.subject	Weight Pruning	en
dc.subject	Sparse Training	en
dc.subject	Efficient Vision	en
dc.subject	Συμπίεση Νευρωνικών Δικτύων	el
dc.subject	Αραίωση Παραμέτρων	el
dc.subject	Κλάδεμα Βαρών	el
dc.subject	Αραιή Εκπαίδευση	el
dc.subject	Αποδοτική Όραση	el
dc.title	Effective methods for deep neural network sparsification	en
heal.type	bachelorThesis
heal.classification	Computer Vision	en
heal.classification	Machine Learning	en
heal.classification	Όραση Υπολογιστών	el
heal.classification	Μηχανική Μάθηση	el
heal.language	el
heal.language	en
heal.access	free
heal.recordProvider	ntua	el
heal.publicationDate	2023-10-20
heal.abstract	In recent years, Deep Neural Networks (DNNs) have significantly advanced the state-of-the-art in numerous machine learning tasks. Unfortunately, most compact devices that rely on embedded computing systems with limited resources cannot support the deployment of such powerful DNNs. This has driven considerable research efforts towards creating compact and efficient versions of these models. A prominent method among the model compression literature is neural network pruning, involving the removal of unimportant network elements with the goal of obtaining compressed, yet highly effective models. In this thesis, we focus on removing individual weights based on their magnitudes encompassing the sparsification process during the standard course of training (sparse training), therefore avoiding multi-cycle training and fine-tuning procedures. In the first part of the thesis we propose a pruning solution that tackles the problem of sparsity allocation over the different layers of the DNN. Modeling the distributions of the weights per-layer in a novel way as Gaussian or Laplace enables the method to learn the pruning thresholds through the optimization process, resulting in an effective non-uniform sparsity allocation for a requested overall sparsity target. In the second part of this work, recognizing that the Straight-Through Estimator is a crucial component of the aforementioned method and of sparse training in general, we devote our efforts into improving its effectiveness. This leads to the introduction of Feather, a novel sparse training module utilizing the powerful Straight-Through Estimator as its core, coupled with a new thresholding operator and a gradient scaling technique that enable robust state-of-the-art sparsification performance. More specifically, the thresholding operator balances the currently used ones, namely the hard and soft operators, combining their advantages, while gradient scaling controls the sparsity pattern variations, leading to a more stable training procedure. Both proposed methods are tested on the CIFAR and ImageNet datasets for image classification using various architectures, resulting into state-of-the-art performances. In particular, Feather achieves Top-1 validation accuracies on ImageNet using the ResNet-50 architecture that surpass those obtained from existing methods, including more complex and computationally demanding ones, by a considerable margin.	en
heal.advisorName	Maragos, Petros	en
heal.advisorName	Μαραγκός, Πέτρος	el
heal.committeeMemberName	Μαραγκός, Πέτρος	el
heal.committeeMemberName	Ποταμιάνος, Γεράσιμος	el
heal.committeeMemberName	Ροντογιάννης, Αθανάσιος	el
heal.academicPublisher	Εθνικό Μετσόβιο Πολυτεχνείο. Σχολή Ηλεκτρολόγων Μηχανικών και Μηχανικών Υπολογιστών. Τομέας Σημάτων, Ελέγχου και Ρομποτικής	el
heal.academicPublisherID	ntua
heal.numberOfPages	101 σ.	el
heal.fullTextAvailability	false