CODEWITHSUNDEEP
tensorflowmachine-learningneural-networkgenerative-adversarial-network
beinando
beinando

Reputation: 497

Tensorflow - tf.variable_scope , reuse argument for GAN

i am trying to build a GAN for a project and i really like to understand how this variable sharing in tensorflow's variable_scope is working.

For building the GAN, i have a generator network, and two discriminator networks: One discriminator gets fed with real images, and one discriminator is fed with the fake images, created by the generator. Important is that the discriminator fed with real images and the discriminator fed with fake images need to share the same weights. In order to do that, i need to share the weights.

I got a discriminator and generator definition, lets say:

def discriminator(images, reuse=False):
    with variable_scope("discriminator", reuse=reuse):

        #.... layer definitions, not important here
        #....
        logits = tf.layers.dense(X, 1)
        logits = tf.identity(logits, name="logits")
        out = tf.sigmoid(logits, name="out")
        # 14x14x64
    return logits, out

def generator(input_z, reuse=False):
    with variable_scope("generator", reuse=reuse):

        #.. not so important 
        out = tf.tanh(logits)

    return out

Now the generator and discriminator functions are getting called:

g_model = generator(input_z)
d_model_real, d_logits_real = discriminator(input_real)

#Here , reuse=True should produce the weight sharing between d_model_real, d_logits_real
#and d_model_fake and d_logits_fake.. why?
d_model_fake, d_logits_fake = discriminator(g_model, reuse=True)

Why does the reuse=True statement in the second call produce the sharing of the weights ? As far as i understand, you need to decide the reusing of the variables in the first call, so that you can use them somewhere later in the program.

I would be really happy if someone can explain that to me, i don't find a good source to this topic and it seems really confusing and complicated to me. Thank you!

Upvotes: 0

Views: 1735

Answers (1)

f4.
f4.

Reputation: 3852

Under the hood the variables are created using tf.get_variable().

This function will prefix the variable name with the scope, and check if it exists before creating a new one.

For instance if you are in the scope "fc" and call tf.get_variable("w", [10,10]), the variable name will be "fc/w:0".

Now when you do this a second time, if reuse=True, the scope will again be "fc" and get_variable will reuse the variable "fc/w:0".

However if reuse=False, you will get an error because the variable "fc/w:0" already exists, prompting you to use a different name or use reuse=True.

Example:

In [1]: import tensorflow as tf

In [2]: with tf.variable_scope("fc"):
   ...:      v = tf.get_variable("w", [10,10])
   ...:

In [3]: v
Out[3]: <tf.Variable 'fc/w:0' shape=(10, 10) dtype=float32_ref>

In [4]: with tf.variable_scope("fc"):
   ...:      v = tf.get_variable("w", [10,10])
   ...:
ValueError: Variable fc/w already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope?

In [5]: with tf.variable_scope("fc", reuse=True):
   ...:      v = tf.get_variable("w", [10,10])
   ...:

In [6]: v
Out[6]: <tf.Variable 'fc/w:0' shape=(10, 10) dtype=float32_ref>

Note that, instead of sharing weights, you could instantiate only one discriminator. You can then decide to feed it with real data or generated data using a placeholder_with_default.

Upvotes: 3

Related Questions