# Copy of Generative Adversarial Networks %28GANs%29
course: Academy — 54-Next-Gen-AI-GenAI-Agents-Future-Trends
module: Academy/54-Next-Gen-AI-GenAI-Agents-Future-Trends
type: notebook
source_url: https://personal-learn.armco.dev/files/Academy/54-Next-Gen-AI-GenAI-Agents-Future-Trends/folders/Session_5_-_GANs_Folder/Copy_of_Generative_Adversarial_Networks_%28GANs%29.ipynb
---
[cell 1 code]
# IMPORTANT: RUN THIS CELL IN ORDER TO IMPORT YOUR KAGGLE DATA SOURCES
# TO THE CORRECT LOCATION (/kaggle/input) IN YOUR NOTEBOOK,
# THEN FEEL FREE TO DELETE THIS CELL.
# NOTE: THIS NOTEBOOK ENVIRONMENT DIFFERS FROM KAGGLE'S PYTHON
# ENVIRONMENT SO THERE MAY BE MISSING LIBRARIES USED BY YOUR
# NOTEBOOK.
import os
import sys
from tempfile import NamedTemporaryFile
from urllib.request import urlopen
from urllib.parse import unquote, urlparse
from urllib.error import HTTPError
from zipfile import ZipFile
import tarfile
import shutil
CHUNK_SIZE = 40960
DATA_SOURCE_MAPPING = 'mnist-handwriting:https%3A%2F%2Fstorage.googleapis.com%2Fkaggle-data-sets%2F1063065%2F1788557%2Fbundle%2Farchive.zip%3FX-Goog-Algorithm%3DGOOG4-RSA-SHA256%26X-Goog-Credential%3Dgcp-kaggle-com%2540kaggle-161607.iam.gserviceaccount.com%252F20240707%252Fauto%252Fstorage%252Fgoog4_request%26X-Goog-Date%3D20240707T095754Z%26X-Goog-Expires%3D259200%26X-Goog-SignedHeaders%3Dhost%26X-Goog-Signature%3D6df6931af002f8282da7ba97e8757122f702d26abc4cf82bdbf7ec20ce06b6e916f45aafc43a26bd889db5d34c568d7f44789deebc6621ec843c43b5496271db768fa90b143ba41503d2edbbb6170a8b2b4e931a10886cd2bf86b112b3b00a0a51b008413b8d10af5360c98e0c524c5198f488fb952726989a0134e5870a29bc856ee54463f8fdd65adb0c963948be2bd35c070e2e6b13590350db6264eded04c5d8a2be8ca3151afc1851fb98b77d8764494968e10c5d0555be0c6a6a6bed00af04a1178ca7afa24a9ee3c5c43e94ee31718ecdc367a9dcb2888143b9148bdb505b37cc16f0b3d7440c20e7c353bb9ad2368363aec2ab58ee6b82030f11ffe5,cs231n-gan:https%3A%2F%2Fstorage.googleapis.com%2Fkaggle-data-sets%2F1159532%2F1943746%2Fbundle%2Farchive.zip%3FX-Goog-Algorithm%3DGOOG4-RSA-SHA256%26X-Goog-Credential%3Dgcp-kaggle-com%2540kaggle-161607.iam.gserviceaccount.com%252F20240707%252Fauto%252Fstorage%252Fgoog4_request%26X-Goog-Date%3D20240707T095754Z%26X-Goog-Expires%3D259200%26X-Goog-SignedHeaders%3Dhost%26X-Goog-Signature%3D6840d22c57e9779bdaecccc11fe57c750b10f73166b50142fb5ca1752fadf3c357ead0091c496f5e0e268493190e12cb10124ed15bbacfc9d34b47fa0f4f4921c87aa0209e279b86f90c248fe8756f6a4b660c21ea2ca705bef1c2c29074475d1271ee8324faf292ffbd924e77d872a5379c95a5705d2deb8e28080369977723bb8cfb14c57b47770ecd859be5d98cc4fb5d8a596e4ae1f294069e75ff503a118bb59897ff18d0d1d0970c38fc42788a5d7075fd529e9c7e65597f28d16b5340f01ddfd7fd00757c3f20b397a2bf9477323753d6892e95fce6ed70897cd8f616f3f03512035d36694dd7bd5a4d08b9cf61777893b03260bbf2e974ba8504d001'
KAGGLE_INPUT_PATH='/kaggle/input'
KAGGLE_WORKING_PATH='/kaggle/working'
KAGGLE_SYMLINK='kaggle'
!umount /kaggle/input/ 2> /dev/null
shutil.rmtree('/kaggle/input', ignore_errors=True)
os.makedirs(KAGGLE_INPUT_PATH, 0o777, exist_ok=True)
os.makedirs(KAGGLE_WORKING_PATH, 0o777, exist_ok=True)
try:
os.symlink(KAGGLE_INPUT_PATH, os.path.join("..", 'input'), target_is_directory=True)
except FileExistsError:
pass
try:
os.symlink(KAGGLE_WORKING_PATH, os.path.join("..", 'working'), target_is_directory=True)
except FileExistsError:
pass
for data_source_mapping in DATA_SOURCE_MAPPING.split(','):
directory, download_url_encoded = data_source_mapping.split(':')
download_url = unquote(download_url_encoded)
filename = urlparse(download_url).path
destination_path = os.path.join(KAGGLE_INPUT_PATH, directory)
try:
with urlopen(download_url) as fileres, NamedTemporaryFile() as tfile:
total_length = fileres.headers['content-length']
print(f'Downloading {directory}, {total_length} bytes compressed')
dl = 0
data = fileres.read(CHUNK_SIZE)
while len(data) > 0:
dl += len(data)
tfile.write(data)
done = int(50 * dl / int(total_length))
sys.stdout.write(f"\r[{'=' * done}{' ' * (50-done)}] {dl} bytes downloaded")
sys.stdout.flush()
data = fileres.read(CHUNK_SIZE)
if filename.endswith('.zip'):
with ZipFile(tfile) as zfile:
zfile.extractall(destination_path)
else:
with tarfile.open(tfile.name) as tarfile:
tarfile.extractall(destination_path)
print(f'\nDownloaded and uncompressed: {directory}')
except HTTPError as e:
print(f'Failed to load (likely expired) {download_url} to path {destination_path}')
continue
except OSError as e:
print(f'Failed to load {download_url} to path {destination_path}')
continue
print('Data source import complete.')
[cell 2 markdown]
# Generative Adversarial Networks (GANs)
So far in CS231N, all the applications of neural networks that we have explored have been **discriminative models** that take an input and are trained to produce a labeled output. This has ranged from straightforward classification of image categories to sentence generation (which was still phrased as a classification problem, our labels were in vocabulary space and we’d learned a recurrence to capture multi-word labels). In this notebook, we will expand our repetoire, and build **generative models** using neural networks. Specifically, we will learn how to build models which generate novel images that resemble a set of training images.
[cell 3 markdown]
### What is a GAN?
In 2014, [Goodfellow et al.](https://arxiv.org/abs/1406.2661) presented a method for training generative models called Generative Adversarial Networks (GANs for short). In a GAN, we build two different neural networks. Our first network is a traditional classification network, called the **discriminator**. We will train the discriminator to take images, and classify them as being real (belonging to the training set) or fake (not present in the training set). Our other network, called the **generator**, will take random noise as input and transform it using a neural network to produce images. The goal of the generator is to fool the discriminator into thinking the images it produced are real.
We can think of this back and forth process of the generator ($G$) trying to fool the discriminator ($D$), and the discriminator trying to correctly classify real vs. fake as a minimax game:
$$\underset{G}{\text{minimize}}\; \underset{D}{\text{maximize}}\; \mathbb{E}_{x \sim p_\text{data}}\left[\log D(x)\right] + \mathbb{E}_{z \sim p(z)}\left[\log \left(1-D(G(z))\right)\right]$$
where $x \sim p_\text{data}$ are samples from the input data, $z \sim p(z)$ are the random noise samples, $G(z)$ are the generated images using the neural network generator $G$, and $D$ is the output of the discriminator, specifying the probability of an input being real. In [Goodfellow et al.](https://arxiv.org/abs/1406.2661), they analyze this minimax game and show how it relates to minimizing the Jensen-Shannon divergence between the training data distribution and the generated samples from $G$.
To optimize this minimax game, we will aternate between taking gradient *descent* steps on the objective for $G$, and gradient *ascent* steps on the objective for $D$:
1. update the **generator** ($G$) to minimize the probability of the __discriminator making the correct choice__.
2. update the **discriminator** ($D$) to maximize the probability of the __discriminator making the correct choice__.
While these updates are useful for analysis, they do not perform well in practice. Instead, we will use a different objective when we update the generator: maximize the probability of the **discriminator making the incorrect choice**. This small change helps to allevaiate problems with the generator gradient vanishing when the discriminator is confident. This is the standard update used in most GAN papers, and was used in the original paper from [Goodfellow et al.](https://arxiv.org/abs/1406.2661).
In this assignment, we will alternate the following updates:
1. Update the generator ($G$) to maximize the probability of the discriminator making the incorrect choice on generated data:
$$\underset{G}{\text{maximize}}\; \mathbb{E}_{z \sim p(z)}\left[\log D(G(z))\right]$$
2. Update the discriminator ($D$), to maximize the probability of the discriminator making the correct choice on real and generated data:
$$\underset{D}{\text{maximize}}\; \mathbb{E}_{x \sim p_\text{data}}\left[\log D(x)\right] + \mathbb{E}_{z \sim p(z)}\left[\log \left(1-D(G(z))\right)\right]$$
### What else is there?
Since 2014, GANs have exploded into a huge research area, with massive [workshops](https://sites.google.com/site/nips2016adversarial/), and [hundreds of new papers](https://github.com/hindupuravinash/the-gan-zoo). Compared to other approaches for generative models, they often produce the highest quality samples but are some of the most difficult and finicky models to train (see [this github repo](https://github.com/soumith/ganhacks) that contains a set of 17 hacks that are useful for getting models working). Improving the stabiilty and robustness of GAN training is an open research question, with new papers coming out every day! For a more recent tutorial on GANs, see [here](https://arxiv.org/abs/1701.00160). There is also some even more recent exciting work that changes the objective function to Wasserstein distance and yields much more stable results across model architectures: [WGAN](https://arxiv.org/abs/1701.07875), [WGAN-GP](https://arxiv.org/abs/1704.00028).
GANs are not the only way to train a generative model! For other approaches to generative modeling check out the [deep generative model chapter](http://www.deeplearningbook.org/contents/generative_models.html) of the Deep Learning [book](http://www.deeplearningbook.org). Another popular way of training neural networks as generative models is Variational Autoencoders (co-discovered [here](https://arxiv.org/abs/1312.6114) and [here](https://arxiv.org/abs/1401.4082)). Variational autoencoders combine neural networks with variational inference to train deep generative models. These models tend to be far more stable and easier to train but currently don't produce samples that are as pretty as GANs.
Example pictures of what you should expect (yours might look slightly different):
[cell 4 markdown]
## Setup
[cell 5 code]
import tensorflow as tf
import numpy as np
import os
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
# A bunch of utility functions
def show_images(images):
images = np.reshape(images, [images.shape[0], -1]) # images reshape to (batch_size, D)
sqrtn = int(np.ceil(np.sqrt(images.shape[0])))
sqrtimg = int(np.ceil(np.sqrt(images.shape[1])))
fig = plt.figure(figsize=(sqrtn, sqrtn))
gs = gridspec.GridSpec(sqrtn, sqrtn)
gs.update(wspace=0.05, hspace=0.05)
for i, img in enumerate(images):
ax = plt.subplot(gs[i])
plt.axis('off')
ax.set_xticklabels([])
ax.set_yticklabels([])
ax.set_aspect('equal')
plt.imshow(img.reshape([sqrtimg,sqrtimg]))
return
def preprocess_img(x):
return 2 * x - 1.0
def deprocess_img(x):
return (x + 1.0) / 2.0
def rel_error(x,y):
return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))
def count_params(model):
"""Count the number of parameters in the current TensorFlow graph """
param_count = np.sum([np.prod(p.shape) for p in model.weights])
return param_count
answers = np.load('../input/cs231n-gan/gan-checks-tf.npz')
NOISE_DIM = 96
[cell 6 markdown]
## Dataset
GANs are notoriously finicky with hyperparameters, and also require many training epochs. In order to make this assignment approachable without a GPU, we will be working on the MNIST dataset, which is 60,000 training and 10,000 test images. Each picture contains a centered image of white digit on black background (0 through 9). This was one of the first datasets used to train convolutional neural networks and it is fairly easy -- a standard CNN model can easily exceed 99% accuracy.
**Heads-up**: Our MNIST wrapper returns images as vectors. That is, they're size (batch, 784). If you want to treat them as images, we have to resize them to (batch,28,28) or (batch,28,28,1). They are also type np.float32 and bounded [0,1].
[cell 7 code]
class MNIST(object):
def __init__(self, batch_size, shuffle=False):
"""
Construct an iterator object over the MNIST data
Inputs:
- batch_size: Integer giving number of elements per minibatch
- shuffle: (optional) Boolean, whether to shuffle the data on each epoch
"""
train, _ = tf.keras.datasets.mnist.load_data()
X, y = train
X = X.astype(np.float32)/255
X = X.reshape((X.shape[0], -1))
self.X, self.y = X, y
self.batch_size, self.shuffle = batch_size, shuffle
def __iter__(self):
N, B = self.X.shape[0], self.batch_size
idxs = np.arange(N)
if self.shuffle:
np.random.shuffle(idxs)
return iter((self.X[i:i+B], self.y[i:i+B]) for i in range(0, N, B))
[cell 8 code]
# show a batch
mnist = MNIST(batch_size=25)
show_images(mnist.X[:25])
[cell 9 markdown]
## LeakyReLU
In the cell below, you should implement a LeakyReLU. See the [class notes](http://cs231n.github.io/neural-networks-1/) (where alpha is small number) or equation (3) in [this paper](http://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf). LeakyReLUs keep ReLU units from dying and are often used in GAN methods (as are maxout units, however those increase model size and therefore are not used in this notebook).
HINT: You should be able to use `tf.maximum`
[cell 10 code]
def leaky_relu(x, alpha=0.01):
"""Compute the leaky ReLU activation function.
Inputs:
- x: TensorFlow Tensor with arbitrary shape
- alpha: leak parameter for leaky ReLU
Returns:
TensorFlow Tensor with the same shape as x
"""
# TODO: implement leaky ReLU
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
x = tf.nn.leaky_relu(x,alpha)
return x
pass
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
[cell 11 markdown]
Test your leaky ReLU implementation. You should get errors < 1e-10
[cell 12 code]
def test_leaky_relu(x, y_true):
y = leaky_relu(tf.constant(x))
print('Maximum error: %g'%rel_error(y_true, y))
test_leaky_relu(answers['lrelu_x'], answers['lrelu_y'])
[cell 13 markdown]
## Random Noise
Generate a TensorFlow `Tensor` containing uniform noise from -1 to 1 with shape `[batch_size, dim]`.
[cell 14 code]
def sample_noise(batch_size, dim):
"""Generate random uniform noise from -1 to 1.
Inputs:
- batch_size: integer giving the batch size of noise to generate
- dim: integer giving the dimension of the noise to generate
Returns:
TensorFlow Tensor containing uniform noise in [-1, 1] with shape [batch_size, dim]
"""
# TODO: sample and return noise
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
noise = tf.random.uniform([batch_size,dim],minval = -1,maxval = 1)
return noise
pass
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
[cell 15 markdown]
Make sure noise is the correct shape and type:
[cell 16 code]
def test_sample_noise():
batch_size = 3
dim = 4
z = sample_noise(batch_size, dim)
# Check z has the correct shape
assert z.get_shape().as_list() == [batch_size, dim]
# Make sure z is a Tensor and not a numpy array
assert isinstance(z, tf.Tensor)
# Check that we get different noise for different evaluations
z1 = sample_noise(batch_size, dim)
z2 = sample_noise(batch_size, dim)
assert not np.array_equal(z1, z2)
# Check that we get the correct range
assert np.all(z1 >= -1.0) and np.all(z1 <= 1.0)
print("All tests passed!")
test_sample_noise()
[cell 17 markdown]
## Discriminator
Our first step is to build a discriminator. **Hint:** You should use the layers in `tf.keras.layers` to build the model.
All fully connected layers should include bias terms. For initialization, just use the default initializer used by the `tf.keras.layers` functions.
Architecture:
* Fully connected layer with input size 784 and output size 256
* LeakyReLU with alpha 0.01
* Fully connected layer with output size 256
* LeakyReLU with alpha 0.01
* Fully connected layer with output size 1
The output of the discriminator should thus have shape `[batch_size, 1]`, and contain real numbers corresponding to the scores that each of the `batch_size` inputs is a real image.
[cell 18 code]
def discriminator():
"""Compute discriminator score for a batch of input images.
Inputs:
- x: TensorFlow Tensor of flattened input images, shape [batch_size, 784]
Returns:
TensorFlow Tensor with shape [batch_size, 1], containing the score
for an image being real for each input image.
"""
model = tf.keras.models.Sequential([
# TODO: implement architecture
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
tf.keras.layers.InputLayer(784),
tf.keras.layers.Dense(256),
tf.keras.layers.LeakyReLU(0.01),
tf.keras.layers.Dense(256),
tf.keras.layers.LeakyReLU(0.01),
tf.keras.layers.Dense(1)
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
])
return model
[cell 19 markdown]
Test to make sure the number of parameters in the discriminator is correct:
[cell 20 code]
def test_discriminator(true_count=267009):
model = discriminator()
cur_count = count_params(model)
if cur_count != true_count:
print('Incorrect number of parameters in discriminator. {0} instead of {1}. Check your achitecture.'.format(cur_count,true_count))
else:
print('Correct number of parameters in discriminator.')
test_discriminator()
[cell 21 markdown]
## Generator
Now to build a generator. You should use the layers in `tf.keras.layers` to construct the model. All fully connected layers should include bias terms. Note that you can use the tf.nn module to access activation functions. Once again, use the default initializers for parameters.
Architecture:
* Fully connected layer with inupt size tf.shape(z)[1] (the number of noise dimensions) and output size 1024
* `ReLU`
* Fully connected layer with output size 1024
* `ReLU`
* Fully connected layer with output size 784
* `TanH` (To restrict every element of the output to be in the range [-1,1])
[cell 22 code]
def generator(noise_dim=NOISE_DIM):
"""Generate images from a random noise vector.
Inputs:
- z: TensorFlow Tensor of random noise with shape [batch_size, noise_dim]
Returns:
TensorFlow Tensor of generated images, with shape [batch_size, 784].
"""
model = tf.keras.models.Sequential([
# TODO: implement architecture
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
tf.keras.layers.InputLayer(noise_dim),
tf.keras.layers.Dense(1024),
tf.keras.layers.ReLU(),
tf.keras.layers.Dense(1024),
tf.keras.layers.ReLU(),
tf.keras.layers.Dense(784,activation = tf.nn.tanh)
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
])
return model
[cell 23 markdown]
Test to make sure the number of parameters in the generator is correct:
[cell 24 code]
def test_generator(true_count=1858320):
model = generator(4)
cur_count = count_params(model)
if cur_count != true_count:
print('Incorrect number of parameters in generator. {0} instead of {1}. Check your achitecture.'.format(cur_count,true_count))
else:
print('Correct number of parameters in generator.')
test_generator()
[cell 25 markdown]
# GAN Loss
Compute the generator and discriminator loss. The generator loss is:
$$\ell_G = -\mathbb{E}_{z \sim p(z)}\left[\log D(G(z))\right]$$
and the discriminator loss is:
$$ \ell_D = -\mathbb{E}_{x \sim p_\text{data}}\left[\log D(x)\right] - \mathbb{E}_{z \sim p(z)}\left[\log \left(1-D(G(z))\right)\right]$$
Note that these are negated from the equations presented earlier as we will be *minimizing* these losses.
**HINTS**: Use [tf.ones](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/ones) and [tf.zeros](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/zeros) to generate labels for your discriminator. Use [tf.keras.losses.BinaryCrossentropy](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/losses/BinaryCrossentropy) to help compute your loss function.
[cell 26 code]
def discriminator_loss(logits_real, logits_fake):
"""
Computes the discriminator loss described above.
Inputs:
- logits_real: Tensor of shape (N, 1) giving scores for the real data.
- logits_fake: Tensor of shape (N, 1) giving scores for the fake data.
Returns:
- loss: Tensor containing (scalar) the loss for the discriminator.
"""
loss = None
cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
real_loss = cross_entropy(tf.ones_like(logits_real), logits_real)
fake_loss = cross_entropy(tf.zeros_like(logits_fake), logits_fake)
loss = real_loss + fake_loss
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
return loss
def generator_loss(logits_fake):
"""
Computes the generator loss described above.
Inputs:
- logits_fake: PyTorch Tensor of shape (N,) giving scores for the fake data.
Returns:
- loss: PyTorch Tensor containing the (scalar) loss for the generator.
"""
loss = None
cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
loss = cross_entropy(tf.ones_like(logits_fake), logits_fake)
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
return loss
[cell 27 markdown]
Test your GAN loss. Make sure both the generator and discriminator loss are correct. You should see errors less than 1e-8.
[cell 28 code]
def test_discriminator_loss(logits_real, logits_fake, d_loss_true):
d_loss = discriminator_loss(tf.constant(logits_real),
tf.constant(logits_fake))
print("Maximum error in d_loss: %g"%rel_error(d_loss_true, d_loss))
test_discriminator_loss(answers['logits_real'], answers['logits_fake'],
answers['d_loss_true'])
[cell 29 code]
def test_generator_loss(logits_fake, g_loss_true):
g_loss = generator_loss(tf.constant(logits_fake))
print("Maximum error in g_loss: %g"%rel_error(g_loss_true, g_loss))
test_generator_loss(answers['logits_fake'], answers['g_loss_true'])
[cell 30 markdown]
# Optimizing our loss
Make an `Adam` optimizer with a 1e-3 learning rate, beta1=0.5 to mininize G_loss and D_loss separately. The trick of decreasing beta was shown to be effective in helping GANs converge in the [Improved Techniques for Training GANs](https://arxiv.org/abs/1606.03498) paper. In fact, with our current hyperparameters, if you set beta1 to the Tensorflow default of 0.9, there's a good chance your discriminator loss will go to zero and the generator will fail to learn entirely. In fact, this is a common failure mode in GANs; if your D(x) learns too fast (e.g. loss goes near zero), your G(z) is never able to learn. Often D(x) is trained with SGD with Momentum or RMSProp instead of Adam, but here we'll use Adam for both D(x) and G(z).
[cell 31 code]
# TODO: create an AdamOptimizer for D_solver and G_solver
def get_solvers(learning_rate=1e-3, beta1=0.5):
"""Create solvers for GAN training.
Inputs:
- learning_rate: learning rate to use for both solvers
- beta1: beta1 parameter for both solvers (first moment decay)
Returns:
- D_solver: instance of tf.optimizers.Adam with correct learning_rate and beta1
- G_solver: instance of tf.optimizers.Adam with correct learning_rate and beta1
"""
D_solver = None
G_solver = None
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
D_solver = tf.optimizers.Adam(learning_rate,beta1)
G_solver = tf.optimizers.Adam(learning_rate,beta1)
pass
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
return D_solver, G_solver
[cell 32 markdown]
# Training a GAN!
Well that wasn't so hard, was it? After the first epoch, you should see fuzzy outlines, clear shapes as you approach epoch 3, and decent shapes, about half of which will be sharp and clearly recognizable as we pass epoch 5. In our case, we'll simply train D(x) and G(z) with one batch each every iteration. However, papers often experiment with different schedules of training D(x) and G(z), sometimes doing one for more steps than the other, or even training each one until the loss gets "good enough" and then switching to training the other.
[cell 33 code]
# a giant helper function
def run_a_gan(D, G, D_solver, G_solver, discriminator_loss, generator_loss,\
show_every=20, print_every=20, batch_size=128, num_epochs=10, noise_size=96):
"""Train a GAN for a certain number of epochs.
Inputs:
- D: Discriminator model
- G: Generator model
- D_solver: an Optimizer for Discriminator
- G_solver: an Optimizer for Generator
- generator_loss: Generator loss
- discriminator_loss: Discriminator loss
Returns:
Nothing
"""
mnist = MNIST(batch_size=batch_size, shuffle=True)
iter_count = 0
for epoch in range(num_epochs):
for (x, _) in mnist:
with tf.GradientTape() as tape:
real_data = x
logits_real = D(preprocess_img(real_data))
g_fake_seed = sample_noise(batch_size, noise_size)
fake_images = G(g_fake_seed)
logits_fake = D(tf.reshape(fake_images, [batch_size, 784]))
d_total_error = discriminator_loss(logits_real, logits_fake)
d_gradients = tape.gradient(d_total_error, D.trainable_variables)
D_solver.apply_gradients(zip(d_gradients, D.trainable_variables))
with tf.GradientTape() as tape:
g_fake_seed = sample_noise(batch_size, noise_size)
fake_images = G(g_fake_seed)
gen_logits_fake = D(tf.reshape(fake_images, [batch_size, 784]))
g_error = generator_loss(gen_logits_fake)
g_gradients = tape.gradient(g_error, G.trainable_variables)
G_solver.apply_gradients(zip(g_gradients, G.trainable_variables))
if (iter_count % show_every == 0):
print('Epoch: {}, Iter: {}, D: {:.4}, G:{:.4}'.format(epoch, iter_count,d_total_error,g_error))
imgs_numpy = fake_images.cpu().numpy()
show_images(imgs_numpy[0:16])
plt.show()
iter_count += 1
# random noise fed into our generator
z = sample_noise(batch_size, noise_size)
# generated images
G_sample = G(z)
print('Final images')
show_images(G_sample[:16])
plt.show()
[cell 34 markdown]
#### Train your GAN! This should take about 10 minutes on a CPU, or about 2 minutes on GPU.
[cell 35 code]
# Make the discriminator
D = discriminator()
# Make the generator
G = generator()
# Use the function you wrote earlier to get optimizers for the Discriminator and the Generator
D_solver, G_solver = get_solvers()
# Run it!
run_a_gan(D, G, D_solver, G_solver, discriminator_loss, generator_loss)
[cell 36 markdown]
# Least Squares GAN
We'll now look at [Least Squares GAN](https://arxiv.org/abs/1611.04076), a newer, more stable alternative to the original GAN loss function. For this part, all we have to do is change the loss function and retrain the model. We'll implement equation (9) in the paper, with the generator loss:
$$\ell_G = \frac{1}{2}\mathbb{E}_{z \sim p(z)}\left[\left(D(G(z))-1\right)^2\right]$$
and the discriminator loss:
$$ \ell_D = \frac{1}{2}\mathbb{E}_{x \sim p_\text{data}}\left[\left(D(x)-1\right)^2\right] + \frac{1}{2}\mathbb{E}_{z \sim p(z)}\left[ \left(D(G(z))\right)^2\right]$$
**HINTS**: Instead of computing the expectation, we will be averaging over elements of the minibatch, so make sure to combine the loss by averaging instead of summing. When plugging in for $D(x)$ and $D(G(z))$ use the direct output from the discriminator (`score_real` and `score_fake`).
[cell 37 code]
def ls_discriminator_loss(scores_real, scores_fake):
"""
Compute the Least-Squares GAN loss for the discriminator.
Inputs:
- scores_real: Tensor of shape (N, 1) giving scores for the real data.
- scores_fake: Tensor of shape (N, 1) giving scores for the fake data.
Outputs:
- loss: A Tensor containing the loss.
"""
loss = None
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
loss = 0.5*tf.reduce_mean(tf.square(scores_real - 1)) + 0.5*tf.reduce_mean(tf.square(scores_fake))
pass
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
return loss
def ls_generator_loss(scores_fake):
"""
Computes the Least-Squares GAN loss for the generator.
Inputs:
- scores_fake: Tensor of shape (N, 1) giving scores for the fake data.
Outputs:
- loss: A Tensor containing the loss.
"""
loss = None
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
loss = 0.5*tf.reduce_mean((tf.square(scores_fake-1)))
pass
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
return loss
[cell 38 markdown]
Test your LSGAN loss. You should see errors less than 1e-8.
[cell 39 code]
def test_lsgan_loss(score_real, score_fake, d_loss_true, g_loss_true):
d_loss = ls_discriminator_loss(tf.constant(score_real), tf.constant(score_fake))
g_loss = ls_generator_loss(tf.constant(score_fake))
print("Maximum error in d_loss: %g"%rel_error(d_loss_true, d_loss))
print("Maximum error in g_loss: %g"%rel_error(g_loss_true, g_loss))
test_lsgan_loss(answers['logits_real'], answers['logits_fake'],
answers['d_loss_lsgan_true'], answers['g_loss_lsgan_true'])
[cell 40 markdown]
Create new training steps so we instead minimize the LSGAN loss:
[cell 41 code]
# Make the discriminator
D = discriminator()
# Make the generator
G = generator()
# Use the function you wrote earlier to get optimizers for the Discriminator and the Generator
D_solver, G_solver = get_solvers()
# Run it!
run_a_gan(D, G, D_solver, G_solver, ls_discriminator_loss, ls_generator_loss)
[cell 42 markdown]
# Deep Convolutional GANs
In the first part of the notebook, we implemented an almost direct copy of the original GAN network from Ian Goodfellow. However, this network architecture allows no real spatial reasoning. It is unable to reason about things like "sharp edges" in general because it lacks any convolutional layers. Thus, in this section, we will implement some of the ideas from [DCGAN](https://arxiv.org/abs/1511.06434), where we use convolutional networks as our discriminators and generators.
#### Discriminator
We will use a discriminator inspired by the TensorFlow MNIST classification [tutorial](https://www.tensorflow.org/get_started/mnist/pros), which is able to get above 99% accuracy on the MNIST dataset fairly quickly. *Be sure to check the dimensions of x and reshape when needed*, fully connected blocks expect [N,D] Tensors while conv2d blocks expect [N,H,W,C] Tensors. Please use `tf.keras.layers` to define the following architecture:
Architecture:
* Conv2D: 32 Filters, 5x5, Stride 1, padding 0
* Leaky ReLU(alpha=0.01)
* Max Pool 2x2, Stride 2
* Conv2D: 64 Filters, 5x5, Stride 1, padding 0
* Leaky ReLU(alpha=0.01)
* Max Pool 2x2, Stride 2
* Flatten
* Fully Connected with output size 4 x 4 x 64
* Leaky ReLU(alpha=0.01)
* Fully Connected with output size 1
Once again, please use biases for all convolutional and fully connected layers, and use the default parameter initializers. Note that a padding of 0 can be accomplished with the 'VALID' padding option.
[cell 43 code]
def discriminator():
"""Compute discriminator score for a batch of input images.
Inputs:
- x: TensorFlow Tensor of flattened input images, shape [batch_size, 784]
Returns:
TensorFlow Tensor with shape [batch_size, 1], containing the score
for an image being real for each input image.
"""
model = tf.keras.models.Sequential([
# TODO: implement architecture
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
tf.keras.layers.InputLayer(784),
tf.keras.layers.Reshape((28,28,1)),
tf.keras.layers.Conv2D(32,(5,5),strides = 1,padding = 'VALID'),
tf.keras.layers.LeakyReLU(0.01),
tf.keras.layers.MaxPool2D((2,2),strides = 2),
tf.keras.layers.Conv2D(64,(5,5),strides = 1,padding = 'VALID'),
tf.keras.layers.LeakyReLU(0.01),
tf.keras.layers.MaxPool2D((2,2),strides = 2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(4*4*64),
tf.keras.layers.LeakyReLU(0.01),
tf.keras.layers.Dense(1)
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
])
return model
model = discriminator()
test_discriminator(1102721)
[cell 44 markdown]
#### Generator
For the generator, we will copy the architecture exactly from the [InfoGAN paper](https://arxiv.org/pdf/1606.03657.pdf). See Appendix C.1 MNIST. Please use `tf.keras.layers` for your implementation. You might find the documentation for [tf.keras.layers.Conv2DTranspose](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/Conv2DTranspose) useful. The architecture is as follows.
Architecture:
* Fully connected with output size 1024
* `ReLU`
* BatchNorm
* Fully connected with output size 7 x 7 x 128
* `ReLU`
* BatchNorm
* Resize into Image Tensor of size 7, 7, 128
* Conv2D^T (transpose): 64 filters of 4x4, stride 2
* `ReLU`
* BatchNorm
* Conv2d^T (transpose): 1 filter of 4x4, stride 2
* `TanH`
Once again, use biases for the fully connected and transpose convolutional layers. Please use the default initializers for your parameters. For padding, choose the 'same' option for transpose convolutions. For Batch Normalization, assume we are always in 'training' mode.
[cell 45 code]
def generator(noise_dim=NOISE_DIM):
"""Generate images from a random noise vector.
Inputs:
- z: TensorFlow Tensor of random noise with shape [batch_size, noise_dim]
Returns:
TensorFlow Tensor of generated images, with shape [batch_size, 784].
"""
model = tf.keras.models.Sequential([
# TODO: implement architecture
# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
tf.keras.layers.InputLayer(noise_dim),
tf.keras.layers.Dense(1024,activation = 'relu'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dense(7*7*128,activation = 'relu'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Reshape((7,7,128)),
tf.keras.layers.Conv2DTranspose(filters=64,kernel_size=(4,4),strides = 2,activation = tf.nn.relu,padding = 'SAME'),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Conv2DTranspose(filters=1,kernel_size=(4,4),strides = 2,activation = tf.tanh,padding = 'SAME')
])
# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****
return model
test_generator(6595521)
[cell 46 markdown]
We have to recreate our network since we've changed our functions.
[cell 47 markdown]
### Train and evaluate a DCGAN
This is the one part of A3 that significantly benefits from using a GPU. It takes 3 minutes on a GPU for the requested five epochs. Or about 50 minutes on a dual core laptop on CPU (feel free to use 3 epochs if you do it on CPU).
[cell 48 code]
# Make the discriminator
D = discriminator()
# Make the generator
G = generator()
# Use the function you wrote earlier to get optimizers for the Discriminator and the Generator
D_solver, G_solver = get_solvers()
# Run it!
run_a_gan(D, G, D_solver, G_solver, discriminator_loss, generator_loss, num_epochs=5)