Entropic regularization in Wasserstein GANs : robustness, generalization and privacy
Abstract/Contents
- Abstract
- In recent years, GANs have emerged as a powerful method for learning distributions from data by modeling the target distribution as a function of a known distribution. The function, often referred to as the generator, is optimized to minimize a chosen distance measure between the generated and target distributions. One commonly used measure for this purpose is the so-called Wasserstein distance. However, Wasserstein distance is hard to compute and optimize, and in practice, entropic regularization techniques are used to facilitate its computation and improve numerical convergence. Introducing regularization, however, changes the problem we are trying to solve, and hence the learned solution. While the computational advantages of entropic regularization have been well-established in the recent literature, the influence of the regularization on the learned generative model and the distribution it generates has remained poorly understood. In this thesis, we study the consequences of regularizing Wasserstein GANs with entropic regularization and show three important impacts of the regularization: 1) we shed light on how entropic regularization impacts the learned GAN solution; 2) we show that it improves sample complexity by removing the curse of dimensionality; 3) we show that entropic regularization can be used to effectively train GANs from differentially privatized data. First, we mathematically derive the learned distribution of the GAN to show the effects of the regularization in a benchmark setting. Prior works focus primarily on evaluating GANs on real data, typically images, and although clearly valuable, such evaluations are often subjective due to a lack of clear baselines for benchmarking. The target distribution of images is too complex to get a precise form of the generated distribution even in the population setting. However, in a benchmark setting where the generator is linear and the target distribution is high-dimensional Gaussian, we are able to characterize the learned solution and prove that regularization leads to the robustness of the learned distribution as well as feature selection. We additionally show that debiasing the Entropy-regularized Wasserstein GAN with the autocorrective term (using Sinkhorn divergence as a loss function) leads to an unbiased solution. Second, we study the effect of regularization on the sample complexity. Without regularization it is known that Wasserstein GANs suffer from the curse of dimensionality, the number of samples needed for the empirical solution to approximate the population solution with some given error scales exponentially in the dimension of the problem. This dissertation shows that entropic regularization can resolve the curse of dimensionality enabling convergence at the parametric rate for a large class of generators and distributions, namely Lipschitz generators and sub-gaussian real distributions. These conditions are commonly satisfied in practice (for example, for images and sigmoid or tanh activations in the last layer of the generator). We present a theorem that quantifies the benefits of using Entropy-regularized Wasserstein Distance as a loss function. Our findings indicate that this regularization technique can effectively mitigate the challenges posed by high-dimensional data, offering a more robust and efficient learning process. Finally, we show that entropic regularization can extend the application of GANs to the realm of sensitive data privatized with differential privacy. In this framework, every data holder randomizes the data locally to preserve their privacy (e.g. by the addition of noise) before sending it to the data curator, to whom the randomization mechanism is known, but its random seed is not, so any adversary that gets access to the privatized data will be unable to learn too much about the user's personal information. In this thesis, we propose a framework for training GANs on such differentially privatized data, demonstrating that entropic regularization can effectively denoise the \emph{data distribution} without being able to denoise each individual sample, which would violate privacy. This unique combination allows for the mitigation of both the regularization bias and the effects of privatization noise while ensuring convergence at the parametric rate, allowing for more regularization, and enhancing the overall efficacy of the model.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2023; ©2023 |
| Publication date | 2023; 2023 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Reshetova, Daria |
|---|---|
| Degree supervisor | Ozgur, Ayfer |
| Thesis advisor | Ozgur, Ayfer |
| Thesis advisor | El Gamal, Abbas |
| Thesis advisor | Weissman, Tsachy |
| Degree committee member | El Gamal, Abbas |
| Degree committee member | Weissman, Tsachy |
| Associated with | Stanford University, School of Engineering |
| Associated with | Stanford University, Department of Electrical Engineering |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Daria Reshetova. |
|---|---|
| Note | Submitted to the Department of Electrical Engineering. |
| Thesis | Thesis Ph.D. Stanford University 2023. |
| Location | https://purl.stanford.edu/bm099mt8200 |
Access conditions
- Copyright
- © 2023 by Daria Reshetova
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...