Why use -ŷ⋅log(y) – (1-ŷ)⋅log(1-y) as a loss function?

Why use - ŷ⋅log(y) - (1-ŷ)⋅log(1-y) as the loss function for training logistic regression and sigmoid outputs?

The purpose of this article is to explain the reasoning that leads to a choice of loss function. There are four areas you will want to understand:

  1. Basic Probability: interpreting numbers from 0 to 1 as probability
  2. Statistical Insight : what is the best understanding of “best”?
  3. Mathematical Tricks to Make Life Easier: things you can do to simplify the calculations
  4. Limitations of Computing Hardware: what kind of maths will existing computers get wrong?

Understanding “why this loss function?” should help you understand why not : when should you not use it, and how you might generate a correct alternative.

Background

When training a neural network, for instance in supervised learning, we typically start with a network of random matrices. We then use sample data to repeatedly adjust them until we have shaped them to give the right answers. This requires three things:

  1. Sample data. We need hundreds or thousand or millions of examples that are already labelled with the correct answer.
  2. A cost function. The cost function gives a number to express how wrong our current set of matrices are. “Training a neural network” means “minimise that wrongness by adjusting the matrices to reduce the value of the cost function.”
  3. An algorithm for how exactly you do this minimising. Back-propagation is currently a one-size-fits-all choice, with gradient descent as the simplest example. A consequence is that your cost function must be differentiable.

The example we’ll work towards is a cost function you commonly see in neural network introductions:
-1/m * ∑(for i=1 to m) y * log( ŷ(i)) + (1-y)* log( 1-ŷ(i) )
where

  • m is the number of examples in your sample data
  • y means the correct result for a given input x
  • ŷ means your network’s output for a given input x
  • is the mathematical symbol for sum, or adding up.
  • log is the mathematical logarithm function found in log tables.

1. Basic Probability: Interpreting numbers from 0 to 1 as probability

How certain are you that you are reading this article? 100% certain is taken to mean completely certain, and 0% certain to mean not even a little bit certain. In fact, certainly not. Rather than using percentages, we just use the numbers 0 for certainly not; 1for completely certain; and numbers in between for a scale of probability.

The probability of two independent events is found by multiplying the
probability of each individual event. Consider rolling some fair six sided dice. The probability of rolling a 6 is , which can also be written as 16.66667% or as 0.166667. The probability of rolling two 6s is ⅙ * ⅙, which is 1/36, which is about 2.77778% or 0.277778.

The probability of something not happening is (1 - probability of it happening). The probability of not rolling a six is 1-⅙ which is or
83.3333% or 0.83333333. It’s same as the probability of rolling one of 1,2,3,4,5. The probability of not rolling 2 sixes is (1 - 1/36) which is 97.2222% or 0.97222.

We abbreviate the expression “the probability of …” to p(…) . So if we call the die roll d, then

p( d = 6 )

is read as “the probability that die roll d is 6.”

Conditional probability is “The probability of … given … already happened”
and is written p(… | …).

So if we call our two dice rolls d1 and d2 then

p( d1+d2 = 12 | d2=6)

can be read as “The probability of d1 + d2 adding to 12 given that we already rolled d2 = 6”

Further reading: https://en.wikipedia.org/wiki/Probability#Mathematical_treatment

2. Statistical Insight : What is the best understanding of “Best”?

So you started with some data, that is some examples for which you already know the right answer. Let’s say you have m examples and call this set X.
X = { x(1),...,x(m) }

and the matching set of correct answers, or labels:
Y = {y(1),...,y(m)}.

You might think that the best setting for your network, given these examples, is:

  • “Best” is the one that most often returns the correct result y(i) for a
    given input x(i).

But then you find that your network almost never returns exactly the right answer. For instance in binary classification the right answers are integers 0 and 1; but your network returns answers like 0.997254 or 0.0023.

So next you think about ‘being closest to the correct answer’ and consider:

  • “Best” is the one that gets closest to the correct y(i)s

You realise that defining ‘closest’ for multiple points could mean any number of things. If you learnt Pythagoras’ theorem at school and you know about averages, you might think of Mean Squared Error as the correct cost function to try:

  • “Best” is the one that reduces mean squared error for your example data.

And that would work. This is how linear regression—finding the best straight line through points plotted on graph paper—is often done. (And, if the input data is normally distributed, mean squared error will give very similar results as the cost function we are deriving here).

But if you’ve studied even more statistics, or probability, or information theory, you realise that defining “closest to the correct answer” for a distribution of data which need not be normal is actually really tricky, and you might think of:

  • “Best” is the one that maximises the expected probability of the known
    correct answers.

The new idea here is “expected value.” It is worth your while to study enough statistics and probability to understand it.

And indeed, statisticians take this as often being the best meaning of “best”. It’s called the Maximum Likelihood Estimator and reasons for considering it “best” include: it is unbiased, consistent, and does not rely on any assumptions about the distribution of the data. So it’s a good default choice. Minimising the mean squared error, surprisingly, can sometimes resulted in a biased estimator.

What the Maximum Likelihood Estimator looks like depends on the detail of what problem you are trying to solve.

Specific Example: Binary Classification with a 0..1 output network

So let’s take the case of

  • A binary classification task : the correct answer is one of two possible
    labels. Think of them as two buckets.
  • A network where the output is a single number in the range from 0 to 1

A concrete example would be a cat recogniser : the task is to identify photos of cats. The input data are photographs. The sample data we start with are a set of photographs already correctly labelled as ‘cat’ and ‘not-cat’. The final layer of the network is a single sigmoid unit.

After we have seen some mathematical tricks, we will be able to write down a mathematical formula for “expected probability of the known correct
answers” for the case of binary classification using a network with a single output in the range 0 to 1.

3. Mathematical Tricks to Make Life Easier: Things you can do to simplify the calculations

1st Trick: Use 0 and 1 as the bucket labels.

You can’t easily do maths with words like ‘cat’ and ’not-cat’. Instead, let’s
use 0 and 1 as the names of our buckets. We’ll use 1 to mean ‘it’s a cat’ and 0
to mean ‘it’s not a cat’.

2nd Trick: Interpret a 0-1 output as a probability

This is a neat trick: since the network output is in the range 0 to 1, we decide
to interpret the output as “our estimate of the probability that 1 is the
correct bucket”. To clarify, we do this not because of any deep insight but just because it makes the maths simpler.

With this interpretation we can turn our statistical phrase “expected
probability of the known correct answer” into a mathematical formula:

p( ŷ(i)= y(i) | x(i) ; 𝜃 )

which you should read aloud as “The probability that our estimate y-hat-i equals the correct value y-i, given that the current input is x-i and given that the matrices of our network are currently set to 𝜃.”

We used the letter 𝜃 here as an abbreviation for “all of the current values in the all of the matrices in the network.” 𝜃 is in fact a very long list of hundreds or thousands of numbers.

We will have found the Maximum Likelihood Estimator if we can find the values for the network that maximise this probability.

Before we move on: if the output means “the probability of 1 being correct”, what about the probability of 0 being correct?

In binary classification, everything must go into either bucket 0 or bucket 1. The chance of you being in one or other of the buckets is 100%. But if you’re in 1, the chances of being in 0 are none at all; and vice versa.

If you’re in between – “I estimate the chance of this being a cat photo is 80%” — then remember that in probability, all possibilities together must add up to 100% (It’s 100% certain that a photo goes into one of the buckets). So “I estimate the chance of this being a cat photo is 80%” automatically means, “I estimate the chance of this being a not-cat photo is 20%”.

In general:

  • if output means “the probability of 1 being correct”
  • then (100% - output) or ( 1 - output ) must mean “the probability of 0 being correct”

2½th Trick: Put tricks 1 and 2 together

Here’s the clever bit. If you put tricks 1 and 2 together you can re-write our
formula from (2):

p( ŷ(i)= y(i) | x(i) ; 𝜃 )

as just

  • ŷ(i) when the correct answer is 1, and
  • 1 - ŷ(i) when the correct answer is 0.

Which is a lot simpler. I mean, really a lot simpler.

How does that work? Remember that the set of y(i)s were the correct answers for our sample data. So y(i)=1 if the correct answer is 1, and y(i)=0 if the correct answer is 0. Now trying putting these two sentences from earlier together, using the word output to substitute the 2nd line into the first:

ŷ(i) means the output given the current input is x(i) and given the matrices of our network are currently set to 𝜃.
-We interpret the output as our estimate of the probability that the correct answer is 1

And you get:

ŷ(i) is our estimate of the probability that the correct answer is 1 given
the current input is x(i) and given the matrices of our network are currently
set to 𝜃.”

But notice that when the correct answer y(i) is 1, this sentence which defines ŷ(i) is exactly what we meant by

p( ŷ(i)= y(i) | x(i) ; 𝜃 )

In the case when the correct answer is y(i) = 0 (remember the rule that “the probability of the result being 0 is (1.0 - probability of the result being 1)” that sentence in quotes is what we mean by

p( ŷ(i)) = 1-y(i) | x(i) ; 𝜃 )

With some high school algebra and our basic knowledge of probability and again the (1-…) rule, we realise that

p( ŷ(i)=1-y(i) ) is the same as p( 1-ŷ(i)=1 ) which is the same as 1- ŷ(i)

Conclusion: We have turned our definition of maximum likelihood into a pair of very simple formulae. For each example datum x(i):

  • if the correct answer is 1, then our maximum likelihood estimate is ŷ(i)
  • if the correct answer is 0, then our maximum likelihood estimate is
    1-ŷ(i)

We need just one more trick. Remember that to use this with back propagation, we need a single differentiable function. We must combine those two formulae into a single formula, and it has to be differentiable.

3rd mathematical trick: Invent a differentiable if(…,then …, else …) function

Let’s try to invent a differentiable if(…,then …, else …) function that can combine the two formulae into one:

if( y(i)=1 , then ŷ(i) , else (1 - ŷ(i))

To keep our maths simple, we want some kind of arithmetic version of this if( y=1, then ŷ(i) , else (1-ŷ(i) )) function. Similar to when we noticed that “the probability of not x is 1-x”, we will use a 1-x trick.

There are a couple of simple options. Try these two definitions:

if(y,b,c) = a*y + b*(1-y)

if(y,a,b) = a^y * b^(1-y)

Both work. We could call the first one the ‘times & add’ version of if and the second one the ’exponent & times’ version. You could invent more. The constraints are:

  1. It must return a when y=1 and return b when y=0
  2. It should have a (preferably simple!) derivative.

Both of my suggestions meet the need, but we go with the second option and define:

if(y,a,b) = a^y * b^(1-y)

This choice is not some profound insight; again, it is only because it will make the maths easier further down the page .

This new if() function combines our two separate formulae for “expected probability of the known correct answer” into a single, simple formula:

ŷ(i)^y * (1-ŷ(i))^(1-y)

Maximum Likelihood Estimator for all m examples

So far, we’ve only consider one sample datum, x(i), at a time. What about all m samples? You recall that the the probability of m independent events is the probability of each of the individual events all multiplied together. We use capital greek letter pi — ∏ — meaning product, to show lots of terms being multiplied together, and write it as:

∏ (for i=1 to m) ŷ(i)^y * (1-ŷ(i))^(1-y)

Which you read aloud as “The product, for every i from 1 to m, of y-hat-i to the power y, times one minus y-hat-i to the power 1 – y”. Or as “multiply all the m individual maximum likelihood estimates together.”

At this point, we have done enough work to get started on our back-propagation algorithm and train our network. We have worked out a cost function that will guide us to a maximum likelihood estimator for all the data we have.

Why don’t we go with it?

4. Limitations of Computing Hardware: What kind of maths will existing computers get wrong?

Well. Suppose you have a small training set of only 1,000 examples. Then this product will be 1,000 small numbers multiplied together. Suppose our typical ŷ(i) is about 0.5, then the result will be less than 2^-1000.

The smallest number that the IEEE 754-2008 standard for 64 bit floating point arithmetic can represent is 2^-1023. With only a thousand training examples we are already within a hair’s breadth of having cost function calculation underflow and round to zero! That’s before we’ve even thought about rounding errors for doing a 1,000 multiplications. If we used single precision 32 bit arithmetic—which we might want to because it’s about twice as fast—we hit the underflow problem with only about a hundred training examples.

If only we could use additions instead of multiplications. That would avoid
underflow and dramatically reduce rounding errors. If only …

Computing trick: Use log() to avoid underflow and rounding errors

Those of you old enough to have used log tables at school will recall that the
log() function neatly replaces multiplication with addition:

log(a * b * c) = log(a) + log(b) + log(c)

log also replaces exponentiation with multiplication:log(a^b) = log(a) * b. And, the logs of very small numbers are very big (bigly negative, that is). And log() is differentiable. The derivative of log(x) is 1/x.

Sounds perfect. What if, instead of using the product-of-probabilities for our cost function we could use the sum-of-logs-of-probabilities instead?

Important to the success of this trick is, that the log() function is
monotonic. That is, when something goes up or down,
log(something) goes up or down exactly in step. So when you increase or maximise log(something) then you simultaneously increase or maximisesomething. And vice versa.

What this means is, we can use logs. If we find the value of 𝜃 that improves or maximises the log of

∏ (for i=1 to m) ŷ(i)^y * (1-ŷ(i))^(1-y)

then we know for sure that that same value of 𝜃 simultaneously improves or maximises this product itself.

Let’s do it. The log of the product is the sum of the logs:

∑ (for i=1 to m) log( ŷ(i)^y * (1-ŷ(i))^(1-y) )

and remembering your high school grasp of log() you can simplify even further to,

The Maximum Likelihood Estimator is the network that maximises
∑ (for i=1 to m) y*log( ŷ(i) ) + (1-y)*log( 1-ŷ(i) )

You say maximise, I say minimise

For no good reason whatsoever, we often think of optimisation problems as a minimisation challenge, not a maximisation one. For the same reason we prefer to divide by m so that the cost function is sort of averagey.

It’s tradition. Whatever. So we stick a minus sign in front and divide by m and, ta-da:

-1/m * ∑ (for i=1 to m) y*log( ŷ(i) ) + (1-y)*log( 1-ŷ(i) )

That is how you derive the familiar formula for the cost function for optimising a sigmoid or logistic output with back propagation and gradient descent.

Recap

The ideas that lead to this formulae are:

  • Basic Probability: Interpreting numbers from 0 to 1 as probability
  • Statistical Insight : What is the best understanding of “best”?
  • Mathematical Tricks to Make Life Easier: A series of tricks you can do to simplify the calculations
  • Limitations of Computing Hardware: What kind of maths will existing computers get wrong?

So my recommendations, in priority order, are:

  1. Take a course on probability and statistics. Seriously. You can’t do good machine learning without a good grasp of statistics.
  2. Practise your high school maths
  3. Learn enough about your computer hardware and platform to know the gotchas.

SEO, Googlenerfing and Boris painting busses

I don’t know whether Boris has an SEO expert on his team but this appears to be a successful piece of search engine “optimisation” which is to say nerfing.

nerf (verb) : To weaken, especially of weapons in a computer game. After the company making soft-toy weapons.


Googling Boris bus has, since 2016, produced images of Boris Johnson and the Brexit bus, bearing the lie about the UK sending £350million per week to the EU.

Whereas when Boris talked about painting model buses in an interview on 25th June 2019, the flurry of internet comment it caused led to searches for Boris bus producing results for the interview instead.

I have no idea whether Boris does paint model buses, though I do know he tells lies. Eddie Mair’s interview of Boris is remarkable.

Boris Johnson stands in front of the bus with the lie about £350million


UK European Parliament Elections 2019 – Brexit analysis

It’s hard not to see the UK Euro elections as a Brexit Poll, Round 2. Perhaps some voters didn’t see it that way, but presumably Brexit party voters did (what else did Brexit Party mean?); and presumably the LibDem & Green swingers did too.

These are the England, Wales, Scotland results (Northern Ireland results not yet available) analysed as if it were a Brexit poll and on the assumption that 100% of Conservative voters were Pro-Brexit and that Labour is split 50/50:

Pro-Brexit          7,349,304 44%
Pro-Remain          6,791,094 41%
Split          2,398,097 14%
N/A                88,759 1%
Total        16,627,254 

“Split” covers primarily the Labour Party. Evidently there is a large part of the Labour Party that supports a People’s Vote; the YouGov poll survey (fieldwork between Dec 2018 and Jan 2019) puts the Labour Split as 71%/21%. Although I have treated a vote for Conservative as a vote for Brexit, the YouGov poll suggests that is only 69% true of Conservative voters.

Taking those two factors into account brings us closer to the picture described in the Jan 2019 YouGov survey, which has Brexit support at 40%, and Remain support at 46%.

If there were a referendum today on whether or not the UK should remain a member of the European Union, how would you vote?TotalConLabLib Dem
Remain a member of the EU46%2671%84
Leave the EU39%6921%11
Would not vote6%221
Don’t know7%453
Refused2%010


Most other parties have an explicit stated position:

Party%ageVotesBrexit
Position
The Brexit Party31.6%5,248,533->Brexit
Liberal Democrat20.3%3,367,284Remain
Labour14.1%2,347,255 Split
Green12.1%2,010,909Remain
Conservative9.1%1,511,485->Brexit
Scottish National Party3.60 594,553Remain
Plaid Cymru1.0% 163,928Remain
Change UK3.4% 571,846Remain
UKIP3.3% 549,348->Brexit
The Yorkshire Party0.3% 50,842Split
English Democrats0.2% 39,938->Brexit
UK European Union Party0.2% 33,576Remain
Animal Welfare Party0.2% 25,232Remain
Women’s Equality Party0.1% 23,766Remain
Independent Network0.1% 7,641
Socialist Party of Great Britain0% 3,505
Independents0.5% 77,613

Should a method name describe what it does or what it intends?

Bob Martin raises a good example in InformIT: Robert C. Martin’s Clean Code Tip of the Week #1: An Accidental Doppelgänger in Ruby > Duplication of two functions that do the same thing but mean different things by it.

I recently stumbled into a slightly different take on the question of should a function say what it does or what it intends?

When a  function implements business process Alpha that today consists of steps A and B (but tomorrow  may change) should you call the function DoBusinessProcessAlpha or call it DoStepsAandB?

One answer would be, if the function is in a public package which exposes business functionality then the name should probably show that it does BusinessProcessAlpha. But if it is a private, not exposed, function then the reader is probably looking for the detail, that it does StepsAandB.

The question is more awkward if Steps A and B are themselves business process functions. That is, if you asked your customer, they would understand what steps A and B mean.

I suppose in that case you could always call it DoAlphaAsStepsAandB().