Probability

Random variable x set of outcomes $S = {x_{1}, x_{2}, \dots, x_{N}}$ outcomes can be discrete or continuous. $S_{c o i n} = {h, t}, S_{V_{x}} = {V_{x}, | V_{x} | < \infty}$ .
Event is any subset of outcomes $E \subset S$ and is assigned a probability $p (E)$ .
- I.e. $p_{d i c e} ({1}) = \frac{1}{6}$ , $p_{d i c e} ({1, 3}) = \frac{1}{3}$ .
Union of Events consider events $A$ ( $B$ ) where sum of 2 dice is divisible by 3 (4). $A \cup B$ , A or B, is the set of outcomes where the sum of 2 dice is divisible by either 3 or 4.
Intersection of Events $A \cap B = A B$ (not sure about the last notation, but it was written in class), A and B, would be where the sum of 2 dice is divisible by both 3 and 4, i.e. divisible by 12.
Disjoint of Events $A \cap B = \emptyset$ (disconnected)
Complement $E_{c} = S ∖ E$

Axioms

$\forall E \subseteq S . p (E) \geq 0$
$p (S) = 1$
$p (A \cup B) = p (A) + p (B)$ if $A$ and $B$ are disjoint events
$p (E_{c}) = 1 - p (E)$
Objective probabilities: $p (A) = lim_{N \to \infty} \frac{N_{A}}{N}$ observation
Frequentist $\leftrightarrow$ Bayesian statistics
Subjective probabilities: theoretical estimates for the probabilities using a model
Computing probabilities: $S$ discrete and finite, $S = {x_{1}, x_{2}, \dots, x_{N}}$ , assume $p ({x_{i}}) = \frac{1}{N}$ , $1 < i < N$ . Then, $p (E) = \frac{# o u t c o m e s i n E}{# o u t c o m e s i n S}$ .

Combinatorics

Exercises

Question 1

How many distinct ways you can arrange the 24 letters of the alphabet? W=24!

N distinct objects can be arranged in $W = N!$ ways.

Letters in ’WHAT’: W=4!

Question 2

Letters in ’CHEESE’: W=6!/3! = Number of ways to arrange it/number of ways to arrange the non-distinct letters

Question 3

Letters in ’FREEZER’: W=7!/(3!2!1!1!).

3! from the Es, 2! from the Rs, 1! from the F, 1! from the Z.

Multinomial coefficient gives the number of arrangements W of N objectects from k distinct categories each of which appears $N_{j}$ times, where $N = \sum_{j = 1}^{k} N_{j}$ . $W = \frac{N!}{N_{1}! \dots N_{k}!} = \frac{N!}{\prod_{j} N_{j}!}$ .

We get a special case for $k = 2$ , the binomial coefficient. $W = \frac{N!}{N_{1}! N_{2}!} = \frac{N!}{N_{1}! (N - N_{1})!} = N$ choose $N_{1}$ . $(\binom{W = N}{M})$

Question 4

For $N$ coin tosses, how many are tails?

$N_{A} :=$ Outcome $A$ in $N$ trials.

We want $p_{N} (N_{T})$ .

For event $E$ : $N_{T} = 5$ , $N = 12$ . Then, $W_{E}$ is 12 choose 5. $W_{S} = (2!)^{12}$ . $P (5) = \frac{(\binom{12}{5})}{2^{1} 2}$ .

So, $P (N_{T}) = \frac{W_{E}}{W_{S}} = \frac{\frac{N!}{N_{T}! (N - N_{T})!}}{(2!)^{N}} = \frac{1}{(p_{T})^{N_{T}} (1 - p_{T})^{N - N_{T}}} ((\binom{N}{N_{T}}))$ .

Binomial Distribution

2 outcomes with $p_{a}, p_{b} = 1 - p_{a}$ in $N$ trials, $p_{N} (N_{a}) = (p_{a})^{N_{a}} (1 - p_{a})^{N - N_{a}} (\binom{N}{N_{a}})$ .

Multinomial Distribution

$N$ trials with $k$ outcomes with probabilities $p_{1}, \dots, p_{k}$ . The probability of finding $N_{1}, N_{2}, \dots, N_{k}$ outcomes $j$ with $N = \sum_{j = 1}^{k} N_{j}$ . $p_{N} (N_{1}, N_{2}, \dots, N_{k}) = N! \prod_{j = 1}^{k} \frac{1}{N_{j}!} p_{j}^{N_{j}}$ .

Stirling’s Approximation

$l n N! = N \ln N - N$

$N! = \exp (N \ln N - N) = \frac{N^{N}}{\exp (N)} = {(\frac{N}{e})}^{N}$ .

Stirling’s: $N! = \sqrt{2 π N} {(\frac{N}{e})}^{N}$ .

How? $N! = \int x^{n} \exp (- x) d x = Γ (n + 1)$ .

$(N + 1 / 2) \ln (N + 1 / 2) - (N + 1 / 2) - (1 / 2 \ln 1 / 2 - 1 / 2) - (N \ln N - N)$ $(N + 1 / 2) (\ln N + \ln (1 + 1 / (2 N))) - N - 1 / 2 - 1 / 2 \ln 1 / 2 + 1 / 2 - N \ln N + N$ $1 / 2 \ln N + (N + 1 / 2) \ln (1 + 1 / (2 N)) - 1 / 2 \ln 1 / 2$ $\ln \sqrt{N} + 0 - 1 / 2 \ln 1 / 2$ Thus, the correction is of the order $\sqrt{N}$ , which is what Stirling’s formula has.

Ways

$W = \frac{N!}{\prod_{j = 1}^{k} N_{j}!}$ , $\sum_{j = 1}^{k} N_{j} = N$ . $p_{j} = N_{j} / N$ .

Approximating

Ways

$W \approx= \sqrt{2 π N} {(\frac{N}{e})}^{N} \frac{1}{\prod_{i} N_{i}!} = \frac{1}{\prod_{j} p_{j}^{N_{j}}}$

$\ln W = - \sum_{j} N_{j} \ln p_{j}$

$\frac{\ln W}{N} = - \sum_{j} p_{j} \ln p_{j} = \frac{S}{N k_{B}} \Rightarrow S = k_{B} \ln W = - k_{B} \sum_{j} p_{j} \ln p_{j}$ . Thus, we recovered our typical entropy! This makes sense since Entropy is related to the number of ways a system can be arranged.

The entropy of a fair dice is: $S = k_{B} \ln 6$ .

No knowledge implies fiar dice maximizes the entropy.

Lets say we have a maximally unfair dice $p_{6} = 1, p_{i \neq 6} = 0$ . Then the entropy is minimized. Assume we have no missing information.

Lets say someone says, $⟨ N ⟩_{d i c e} = 3.5$ . Then we would assume it is a fair dice.

If someone says $⟨ N ⟩_{d i c e} = 3.0$ we would assume it is an unfair dice (weighted lower). There is a distribution of probabilities for the individual dice probabilities to be:

$p_{3} = 1$ , $p_{i \neq 3} = 0$
$p_{2} = p_{4} = \frac{1}{2}$ , $p_{i \neq 2, 4} = 0$

Thus, there is an increase in missing information.