Probabilities & Estimation

The mean of a random sample is called a point estimate for the mean of the whole population. A point estimate from a larger sample is generally more reliable than one from a small sample, since it's less affected by chance variation between individual items — it's not sensible to draw firm conclusions from a sample of size 1.

If you repeatedly take random samples from a population and calculate the mean of each, those sample means themselves form a distribution — one that clusters more tightly around the true population mean than individual values do. For samples of size $n$ from $N(\mu, \sigma^2)$, the sample mean $\bar{X}$ has distribution: $$\bar{X} \sim N\left(\mu, \frac{\sigma^2}{n}\right)$$ The standard deviation of this distribution, $\dfrac{\sigma}{\sqrt{n}}$, is called the standard error of the mean — notice that it shrinks as the sample size $n$ grows.

A point estimate alone doesn't say how reliable it is. A confidence interval gives a range of plausible values together with a stated level of confidence: $$\bar{x} \pm z \times \frac{\sigma}{\sqrt{n}}$$ The most common values of $z$: 90% confidence uses $z=1.64$, 95% uses $z=1.96$, and 99% uses $z=2.58$. A higher confidence level always gives a wider interval, and a larger sample size always gives a narrower one.