Probabilities tell us how likely particular outcomes are to occur. In this way a probability is the long-run relative frequency of an outcome. We can similarly define the long-run average for a random variable. This expected value is what we would expect to get as we average more and more outcomes of the variable. It can be calculated easily from the random variable’s probability function.

Expected Value

A classic application of expected values is in analysing a game of chance like Keno. The table below shows the probability function for the simple one-number game introduced in Chapter 8.

Probability function for Keno winnings

So if you play this Keno game once then you will either win $3 or win $0. But what would you expect to win if you played the game over and over again? For example, playing the game 10 times you might get lucky and win in 4 of them. This gives total winnings of $12, an average of $1.20 per game. Now if you played 100 times or 1000 times, what would you expect this average to be?

You can calculate this by using the probabilities as long term relative frequencies. The probability of winning $3 is $\frac{1}{4}$ so in the long run we expect to win $3 for every 4 games, an average of $0.75 per game. Here the other option doesn’t give us anything but the reasoning would be the same. We get $0 with probability $\frac{3}{4}$ so we expect to win $3\times$0 = $0$ every 4 games, an average of $0.00 per game. The total from the two possibilities is $0.75 per game. We call this the expected value of the random variable $X$, written $\text{E}(X)$. In general we calculate it by
$$\text{E}(X)=\sum _{x}P(X=x)x.$$
Here we calculated
$$\text{E}(X)=\frac{3}{4}\times 0+\frac{1}{4}\times 3=0.75.$$
In the long term we will get $0.75 for each game we play. Unfortunately Keno costs $1 per game and so in the long term we will actually lose $0.25 per game. What makes gambling exciting is the variability in the results, rather than the long term result. We will define the standard deviation of a random variable later in this chapter.

Population Mean

In the sciences we are usually not directly interested in games of Keno. Our main focus has been on the idea of sampling from a population. It turns out in this case that the expected value has a simple interpretation.

Suppose a finite population has $N$ individuals with values for some variable of $x_1,x_2,\dots ,x_N$. Let the random variable $X$ be the process of picking an individual at random and recording the value of the variable for that individual. If this is a simple random sample then each individual has the same probability of being chosen and since all the probabilities must add up to 1 this probability must be $1/N$. The expected value of $X$ is thus
$$\begin{array}{rcl}\text{E}(X)& =& \frac{1}{N}{x}_1+\frac{1}{N}{x}_2+\cdots +\frac{1}{N}{x}_N\\ & =& \frac{x_1+x_2+\cdots +x_N}{N}.\end{array}$$
This should look familiar – it is the formula for the sample mean but instead of just a sample we have calculated it for the whole population. Thus for sampling from a finite population the expected value is just the population mean. Instead of $\text{E}(X)$ we will sometimes write ${\mu }_X$ to reflect this.

Law of Large Numbers

The law of large numbers simply states that, as the number of trials increases, sample proportions get closer to probabilities and the sample mean gets closer to the expected value.

A common misinterpretation of this idea is that, for example, if you toss a coin five times and you get heads each time then the next toss is more likely to give tails, to balance things up and get closer to the probability of 0.5. Of course this is not true. If the tosses are independent then the sixth toss is still a 50-50 chance of heads or tails. We could get a hundred tails in a row and that would not matter because after a million more tosses the hundred tails would only be a small wrinkle. The law of large numbers only talks about the long-term, not short-term, behaviour. For a nice example of improbable coin tossing, see the beginning of the film Rosencrantz and Guildenstern Are Dead (1990), based on the play by Tom Stoppard.

The phrase “law of large numbers” was first introduced in 1837 by Siméon Poisson, a French mathematician who we will meet again in Chapter 11.

Variance

An expected value is the long-run mean of a random variable so it is natural to quantify variability using squared deviations about this mean, just as we did for samples. Essentially we want the expected squared deviation. In the Keno example if we win $3 then the squared deviation from the mean is $(3-0.75{)}^2=5.0625$. This happens with probability $\frac{1}{4}$ so the long-term contribution to the squared deviation is $\frac{1}{4}\times 5.0625=1.2656$. Similarly, if we win $0 then the squared deviation is $(0-0.75{)}^2=0.5625$. This happens with probability $\frac{3}{4}$ so on average it contributes $\frac{3}{4}\times 0.5625=0.4219$ to the squared deviation. Adding these gives the long-run squared deviation, the variance of $X$,
$$\text{var}(X)=1.2656+0.4219=1.6875.$$
In general,
$$\text{var}(X)=\sum _{x}P(X=x)(x–{\mu }_X{)}^2.$$
The units of the variance are squared so as for samples we take the square root to get back to the original units, giving the standard deviation
$$\text{sd}(X)=\sqrt{\sum _{x}P(X=x)(x–{\mu }_X{)}^2}.$$
For the Keno example, $\text{sd}(X)=\mathrm{\$}1.30$. This is more than the expected value of $0.75 and that is what makes the gambling so exciting; the average return is low but there is a lot of variability.

Note that another way of writing the formula for variance is
$$\text{var}(X)=\text{E}((X–{\mu }_X{)}^2),$$
so variance really is the expected long-run squared deviation from the mean.

Population Standard Deviation

As before, suppose a finite population has $N$ individuals with values for some variable of $x_1,x_2,\dots ,x_N$. Let the random variable $X$ be the process of picking an individual at random and recording the value of the variable for that individual. Each individual’s probability of being chosen is $1/N$. The standard deviation of $X$ is thus
$$\begin{array}{rcl}\text{sd}(X)& =& \sqrt{\frac{1}{N}(x_1-{\mu }_X{)}^2+\frac{1}{N}(x_2-{\mu }_X{)}^2+\cdots +\frac{1}{N}(x_N–{\mu }_X{)}^2}\\ & =& \sqrt{\frac{\sum (x_j-{\mu }_X{)}^2}{N}}.\end{array}$$
This is the population standard deviation and we often write ${\sigma }_X$ instead of $\text{sd}(X)$. It is a little different from the sample standard deviation formula since we divide by $N$ instead of $N-1$, giving a slightly smaller measure of variability. This is reasonable since we know the mean we are calculating squared deviations around and so we can be more certain of the result. This is mainly of theoretical interest since for data analysis applications it is unlikely we know the population mean.

Precision

To give an example of how we will think about population means and standard deviations, the simple game of Keno was played 100 times with a computer (so we didn’t lose any real money). We won 28 of the games, a total of $84 in winnings. The sample mean for each game was $0.84 and the sample standard deviation was $1.35.

For this particular example we know the population mean and standard deviation. In dealing with variables for human or animal populations it will be unlikely that we know these parameters. Instead we use our sample statistics to estimate these unknown values. The sample proportion of wins, 0.28, is pretty close to the population probability 0.25. The sample mean $0.84 is close to the population mean of $0.75. The sample standard deviation $1.35 is close to the population standard deviation of $1.30.

Here we know the population values and so we can see how precise our sample statistics are. If we don’t know the population values then how can we be sure our sample statistics are as close as we would like to the unknown values? Estimating the precision of sample statistics is a vital part of statistical analysis and will be discussed starting in Chapter 13.

Continuous Random Variables

So far we have motivated and defined the expected value and standard deviation of a discrete random variable. These were the sums of observations and squared deviations, respectively, weighted by their probability of outcome. Since all the examples given involved finite numbers of outcomes, these values could be calculated directly by using the sums.

For continuous variables this is not possible since there are an uncountable number of outcomes and the probability of any individual outcome is 0. However, calculus provides a notion of sum, the integral, which can be used to make analogous definitions for continuous variables.

Suppose a continuous random variable $X$ comes from a distribution with probability density function $f(x)$. Here $f(x)$ plays the role of the probability of $X=x$ occurring. Even though that probability is actually 0, we can use $f(x)$ to weight the outcome $x$ for calculating the average
$$\text{E}(X)={\mu }_X={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}f(x)xdx,$$
the expected value of $X$. The integral sign, $\int$, is an elongated ‘S’ for ‘sum’, reflecting its relationship with the discrete sum we’ve seen before.

The variance is then
$$\text{var}(X)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}f(x)(x–{\mu }_X{)}^{2}dx,$$
giving $\text{sd}(X)=\sqrt{\text{var}(X)}$.

These formulas can be difficult to work out by hand, especially if you haven’t done much calculus before. However, many software packages and graphics calculators can now be used to do these calculations exactly, with algebra, or approximately.

Combining Variables

We are frequently interested in the behaviour of combinations of random variables. For example, obtaining a sample mean involves the sum of $n$ random outcomes and we would like to be able to describe the resulting variability of the statistic itself. We will not prove any of the formulas below but most of them are intuitive.

Shifting and Scaling

Suppose $X$ is the random variable representing the winnings in the game of Keno in the first section of this chapter. Suppose also that the casino has a promotion where they double all your winnings. That is, your winnings now come from the random variable $Y=2X$, with possible outcomes of $0 and $6. You could work out $\text{E}(Y)$ from scratch but it is easy to see that if your winnings all double then your expected value will also double. Here $\text{E}(Y)=2\times \mathrm{\$}0.75=\mathrm{\$}1.50$. Similarly, the standard deviation will also double, with $\text{sd}(Y)=2\times \mathrm{\$}1.30=\mathrm{\$}2.60$. In general, if $a$ is any number then
$$\text{E}(aX)=a\text{E}(X)\text{ and }\text{sd}(aX)=|a|\text{sd}(X).$$
We put the absolute value signs around $a$ because standard deviation can never be negative. For example, if we changed $3 to -$3 in the Keno game, so if you win you pay the casino $3 instead of them paying you, then the new expected value is -$0.75. That is, in the long-run you pay the casino $0.75. However the standard deviation is still $1.30. Even though the direction of payments has changed there is no change in the amount of variability.

Now suppose the Casino gives us an extra $2 each time we play, so our winnings come from $Y=X+2$. Since the mean of $X$ was $0.75 then the mean of $Y$ is $2.75, very generous. But what is $\text{sd}(Y)$? This is still the same as $\text{sd}(X)$. Shifting the distribution does not change how spread out it is. In general, if $b$ is any number then
$$\text{E}(X+b)=\text{E}(X)+b\text{ and }\text{sd}(X+b)=\text{sd}(X).$$

Two Random Variables

Suppose we now look at our total winnings from two games of Keno. This is not the same as the $Y=2X$ example above since that only involved one random outcome, the $X$. Instead we now have $Y=X_1+X_2$, where $X_1$ is the winnings on the first game and $X_2$ is the winnings on the second game.

We expect to win $0.75 on the first game and $0.75 on the second game, so in total we expect to win $1.50. In general,
$$\text{E}(X_1+X_2)=\text{E}(X_1)+\text{E}(X_2).$$
Expected values generally work in this simple intuitive way, giving the same answer as $2\times \mathrm{\$}0.75$. However the standard deviation of $Y$ is not $2\times \mathrm{\$}1.30=\mathrm{\$}2.60$. It turns out that we cannot add standard deviations together directly but we can add together squared deviations.

That is,
$$\text{var}(X_1+X_2)=\text{var}(X_1)+\text{var}(X_2).$$
For two games of Keno we have
$$\text{var}(X_1+X_2)={1.30}^2+{1.30}^2=3.38,$$
so $\text{sd}(X_1+X_2)=\sqrt{3.38}=\mathrm{\$}1.84$. This is actually less than $2.60 so adding up several measurements gives a random variable with less variability than the sum of the variabilities. This is a key observation and is what makes replication an important part of science. It is also what allows a casino to run, giving reliable returns when averaged over thousands or millions of games.

There are two important subtleties to be aware of when adding variance. Firstly, variance involves a squaring process and so anything negative becomes positive. Thus
$$\text{var}(X_1–X_2)=\text{var}(X_1)+\text{var}(X_2).$$
If you have two sources of variability then when you combine them you will always have more variability, even if you are subtracting them. This will be a common use of variance, looking at the difference between two populations.

Secondly, suppose we take a random sample of 10 people and let $X$ be the proportion of males and $Y$ be the proportion of females. The kind of data we might get when doing this are given in the table below.

Sample data of 10 people

There is clearly variability in $X$ and in $Y$ from sample to sample, so $\text{var}(X)>0$ and $\text{var}(Y)>0$, but what is $\text{var}(X+Y)$? It will be 0 since $X+Y$ is always 1. This is not what our formula above predicts and the reason is that this $X$ and $Y$ are not independent. In fact we can only add variances when the two random variables are independent, so the original equation should be qualified as
$$\text{var}(X_1+X_2)=\text{var}(X_1)+\text{var}(X_2),\text{ if }{X}_1\text{ and }{X}_2\text{ are independent}.$$
When we come to compare populations using statistical analysis we will make extensive use of this formula. Thus our samples will always have to be independent.