Adversarial Preference Loss

In my earlier post I talked about how the GAN training procedure can be modified slightly so it minimises $KL[Q|P]$ divergence. I have also talked about how GANs can be understood as

• in the D-step, performing direct log-probability-ratio estimation from data, obtaining an estimate of $\log\frac{Q(x)}{P(x)}$ then
• in the M-step using these estimates to minimise a divergence

From this perspective, training a binary classifier discriminatively is only one way to obtain estimates of the log-probability ratios in question, but I mentioned there are other ways.

This post is basically just having some fun with this idea: I show a different iterative algorithm that has similar flavour. Instead of using a binary classifier, I'm going to use binary pairwise preference learning as the adversarial problem.

Adversarial preference training

As before, we have a generator $G$ and a discriminator $D$ that try to "fight against each other". However, the discriminator now solves a different task: instead of binary classification it performs a two-alternative forced choice (2AFC) task.

2AFC Discriminator

The preference network $D$ receives two inputs, $x_1$ and $x_2$, one of which is a real datapont sampled from $P$, the other one is a synthetic one generated from $Q$ (using the generator $G$). $D$ has to guess which one of its inputs is real. Let's assume that $D$ takes the following symmetric and separable form:

$$D(x_1,x_2;\psi) = \Phi[s(x_1;\psi) - s(x_2;\psi)],$$

where $\Phi$ is the logistic sigmoid, and $s$ is a trainable function (e.g. deep neural network) which I will call the preference function.

The parameters $\psi$ of $D$ are updated using the following objective:

$$\mathcal{L}_{D}(\psi) = -\mathbb{E}_{x_1\sim P}\mathbb{E}_{x_2\sim Q} \log(D(x_1,x_2;\psi))$$

This is similar to a standard binary classification loss, but not quite the same (if you don't understand the objective, note that the symmetry $D(x_1,x_2) = 1 - D(x_2,x_1)$ and try to derive it yourself from the binary classification loss)

Bayes optimal behaviour

Let's consider what happens if we fix $Q$, and optimise $D$ until convergence. At this point we can assume that $D$ approximately implements the Bayes optimal behaviour, which in this case will be the following:

$$\Phi[s(x_1) - s(x_2)] \approx \frac{P(x_1)Q(x_2)}{P(x_1)Q(x_2) + P(x_2)Q(x_1)}$$

Working backwards from here, we get an expression for the preference function:

$$\begin{align} \Phi[s(x_1) - s(x_2)] &\approx \frac{P(x_1)Q(x_2)}{P(x_1)Q(x_2) + P(x_2)Q(x_1)} \\ s(x_1) - s(x_2) &\approx \Phi^{-1}\left[\frac{P(x_1)Q(x_2)}{P(x_1)Q(x_2) + P(x_2)Q(x_1)}\right]\\ s(x_1) - s(x_2) &\approx \log\left[\frac{\frac{P(x_1)Q(x_2)}{P(x_1)Q(x_2) + P(x_2)Q(x_1)}}{1 - \frac{P(x_1)Q(x_2)}{P(x_1)Q(x_2) + P(x_2)Q(x_1)}} \right]\\ s(x_1) - s(x_2) &\approx \log\left[\frac{P(x_1)Q(x_2)}{P(x_2)Q(x_1)} \right]\\ s(x_1) - s(x_2) &\approx \log\frac{P(x_1)}{Q(x_1)} - \log\frac{P(x_2)}{Q(x_2)}, \end{align}$$

Therefore, we can say that when $D$ converged, the preference function $s$ approximates the log-probability ratio between $P$ vs $Q$:

$$s(x) \approx \log\frac{P(x)}{Q(x)}$$

and therefore the loss for the generator becomes

$$\mathcal{L}_{G}(\theta) \approx - \mathbb{E}_{x\sim Q_{\theta}} \log\frac{P(x)}{Q(x)} = KL[Q|P].$$

Is this useful?

I don't know, maybe, most likely not. But that's besides the point I'm trying to make. I just wanted to highlight the arbitrariness of using binary classification in the GAN procedure. One can imagine a whole class of generative modelling procedures based on the same underlying idea of direct probability ratio estimation, and it is unclear (to me) which one method will work best.