Frequency-aware representation Learning: Exploring the synergy of contrastive & reconstruction objectives

Βαλή, Ελευθερία Άννα; Vali, Eleftheria Anna

dc.contributor.author	Βαλή, Ελευθερία Άννα	el
dc.contributor.author	Vali, Eleftheria Anna	en
dc.date.accessioned	2025-09-12T07:16:19Z
dc.date.available	2025-09-12T07:16:19Z
dc.identifier.uri	https://dspace.lib.ntua.gr/xmlui/handle/123456789/62428
dc.identifier.uri	http://dx.doi.org/10.26240/heal.ntua.30124
dc.description
dc.rights	Αναφορά Δημιουργού-Όχι Παράγωγα Έργα 3.0 Ελλάδα	*
dc.rights.uri	http://creativecommons.org/licenses/by-nd/3.0/gr/	*
dc.subject	Deep learning	en
dc.subject	Learning dynamics	en
dc.subject	F-principle	en
dc.subject	Spectral bias	en
dc.subject	Autoencoders	en
dc.subject	Αυτό-αποκωδικοποιητές	el
dc.subject	Αυτοεπιϐλεπόµενη µάϑηση	EL
dc.subject	Μάϑηση αναπαϱαστάσεων	el
dc.subject	Μηχανική µάϑηση	el
dc.subject	Contrastive learning	en
dc.subject	Unsupervised learning	en
dc.subject	Representation learning	en
dc.title	Frequency-aware representation Learning: Exploring the synergy of contrastive & reconstruction objectives	en
heal.type	masterThesis
heal.classification	MACHINE LEARNING	en
heal.language	el
heal.language	en
heal.access	free
heal.recordProvider	ntua	el
heal.publicationDate	2025-02-21
heal.abstract	Deep learning has advanced artificial intelligence by enabling models to learn seman- tically rich data representations. Significant efforts have been made to understand the learning dynamics of deep networks, leading to the identification of the F-Principle, i.e., the tendency of neural networks to prioritize the learning of lower-frequency components, and then progressively capture higher-frequency features as training proceeds. While this phenomenon is well-documented in supervised learning, its presence and implications in unsupervised and self-supervised learning (SSL) remain less explored. This thesis investigates the role of the F-Principle in unsupervised representation learning and proposes methods to mitigate its limitations. First, we provide empirical evi- dence that reconstruction-based learning primarily captures low-frequency components, which can limit its effectiveness to downstream tasks. To address this, we introduce the Frequency Scheduler, a novel technique that systematically guides the learning process across different frequency bands, leading to more informative representations. Second, we explore the integration of a contrastive learning objective alongside the reconstruction loss, hypothesizing that it encourages the model to capture semantically-rich frequency components. While the overall frequency distribution remains unchanged, contrastive learning significantly enhances downstream performance and improves alignment and uniformity in the representation space, suggesting that such objectives help prioritize the low and mid frequencies that correlate more with downstream semantics. Conversely, we find that augmenting contrastive learning with reconstruction objectives also yields improvements in classification accuracy, highlighting the synergistic potential of these approaches. Finally, we propose a novel connection between representation learning and generative modeling, demonstrating how integrating decoders within contrastive learn- ing frameworks enables instance generation by sampling from the learned representation space.	en
heal.advisorName	Ποταμιανός,Αλέξανδρος	el
heal.committeeMemberName	Παναγάκης,Ιωάννης	el
heal.committeeMemberName	Βουλοδήμος, Αθανάσιος	el
heal.academicPublisher	Εθνικό Μετσόβιο Πολυτεχνείο. Σχολή Ηλεκτρολόγων Μηχανικών και Μηχανικών Υπολογιστών	el
heal.academicPublisherID	ntua
heal.fullTextAvailability	false