When to and when not to polish a turd — an information-theoretic approach
For the last few years I’ve been working a lot on generative models for multimodal data. They have many different and exciting applications, but one thing we’re often asked about is “imputing missing data”.
For example, you collect some cheap data modality X and want to generate fancy (and expensive) data modality Y. X could be tissue images, while Y might be molecular data. This is not the most exciting application, but it’s really common when working with clinical data. People basically want to get more bang for their buck.
This could be done with non-generative predictive models, but there are some nice benefits to the generative ones (not to be discussed here).
There’s nothing wrong with this idea of transforming X to Y. We build for this, and we do this. I support this. However, a lot of people want to use this for completely wrong reasons.
🤦 The misconception
I have had so many meetings where I’ve heard people say something like “I want to transform X to Y to give me a better prediction of some given label(s)”. This is not good. This is like trying to polish a turd.
Let’s be clear, it is possible, and probably likely, that training a classifier on Y rather than X gives you better results. However, if Y is predicted solely from X, it cannot contain more information about the label than X already did. This is because we can’t magically add information by just transforming something; a fact also known as the Data Processing Inequality, an idea presented already in Claude Shannon’s 1948 paper (Appendix 7, Page 54).
The book “Elements of Information Theory” does a good job of describing the modern form in text:
[…] no clever manipulation of the data can improve the inferences that can be made from the data.
🙅 Why information can’t increase
Let us look at why. Also, I’ll be talking about this in terms of classification, but the underlying principle is not specific to classification.
We start by introducing three random variables $W,X,Y$. Let $W$ represent a label, for example a cat’s breed. $X$ is a photograph of the cat, and $Y$ is a 3D rendering generated from the 2D image $X$.
The label $W$ and image $X$ are statistically related, i.e., the distribution of image properties will depend on the breed, which we can describe through $p(x\mid w)$. Notably, $Y$ is solely constructed from $X$ through some learnable function $f$, which can be expressed as:
$$ Y=f(X) $$Therefore, once $X$ is known, knowing $W$ provides no additional information about $Y$:
$$ p(y\mid x,w)=p(y\mid x) $$The joint distribution can consequently be factorized as
$$ p(w,x,y)=p(w)p(x\mid w)p(y\mid x,w)=p(w)p(x\mid w)p(y\mid x) $$By definition, this means that $W,X,Y$ form a Markov chain,
$$ W\rightarrow X\rightarrow Y $$Definition Random variables W,X,Y are said to form a Markov chain in that order (denoted by $W \rightarrow X \rightarrow Y$) if the conditional distribution of Y depends only on X and is conditionally independent of W. Specifically, W, X, and Y form a Markov chain $W \rightarrow X \rightarrow Y$ if the joint probability mass function can be written as $p(w,x,y) = p(w)p(x|w)p(y|x)$ Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory (2nd ed.). Wiley. §2.8.
We can now consider the Mutual Information between $W$ and the pair $(X,Y)$. By the chain rule,
$$ I(W;X,Y)=I(W;X)+I(W;Y\mid X). $$Because $W$ and $Y$ are conditionally independent given $X$,
$$ I(W;Y\mid X)=0, $$and therefore
$$ I(W;X,Y)=I(W;X). $$We can expand the same quantity in the opposite order:
$$ I(W;X,Y)=I(W;Y)+I(W;X\mid Y). $$Combining the two expressions gives
$$ I(W;X)=I(W;Y)+I(W;X\mid Y), $$or
$$ I(W;Y)=I(W;X)-I(W;X\mid Y). $$Since conditional mutual information is nonnegative,
$$ I(W;X\mid Y)\ge 0, $$we obtain
This tells us that the information that the transformed variable $Y$ contains about the label $W$ can only ever be equal to or less than the information about $W$ already present in the original form $X$.
🤷 So why bother?
Ok but then why would we even bother with these models that go from one modality to another? Why are people spending countless hours on predicting gene expression from H&E images, single-cell data from bulk RNA-seq, etc.?
To me the answer is: 1) interpretability and 2) actionability. Perhaps our inferred $Y$ carries no more information about the label $W$ than $X$ did, but it might allow you to better interpret the results and derive more actionable insights.
To give you an example by looking at the task of predicting molecular information (e.g., bulk RNA-seq) from images (e.g., H&E). And maybe the label is, say, response to some treatment. That is: $X$ image, $Y$ inferred molecular data, $W$ treatment response.
We have established that the inferred molecular data cannot contain more information about treatment outcome than the image it was generated from, but looking at that data might allow us to better understand why some patients respond differently.
Perhaps it’s the activation of a certain gene program that correlates with treatment response, or one gene that is strongly down-regulated. This is very helpful for understanding the response mechanism, and to then derive an action to test our hypothesis.
We can knock that gene out in a relevant model system and see if it results in the same response as we saw among the patients. We could not have come up with a similar suggestion just by looking at the image with our eyes.
Despite not carrying any more information than the original data, the inferred data may support better hypothesis generation – and that is worth a lot.
I think that’s the end of my rant. If you made it this far, I appreciate it.