Probability Theory

1. Probability Space

1.1 Abstract Setup

Recap (Probability space): A probability space is a triple (Ω,F,P∗)(\Omega, \sigmaF, \P*), where:
  • Ω\Omega is a non-empty set
  • F\sigmaF is a σ\sigma-algebra on Ω\Omega
  • P∗\P* is a probability measure on (Ω,F)(\Omega, \sigmaF)

We can think of an element ω∈Ω\omega \in \Omega as randomly chosen in Ω\Omega, i.e. the outcome of a random experiment. For an event A∈FA \in \sigmaF, the quantity P ⁣(A)\probP{\evA} represent the probability that this random element lies in A\evA.

Proposition 1.1 (Stability under countable intersections): If F\sigmaF is a σ\sigma-algebra, then whenever A1,A2,…∈F\evA_1, \evA_2, \ldots \in \sigmaF, one has ⋂n∈NAn∈F\bigcap_{n \in \N} \evA_n \in \sigmaF.
Proposition 1.2 (Complement rule): For every A∈F\evA \in \sigmaF, one has P ⁣(Ac)=1−P ⁣(A)\probP{\evA^c} = 1 - \probP{\evA}.
Proposition 1.3 (Continuity of probability measures): Let (An)n∈N\sequence{\evA} be a sequence of events. Let ∀n∈N:An⊂An+1\forall n \in \N: \evA_n \subset \evA_{n+1}, then P\P is continuous from below, i.e. P(⋃n∈NAn)=lim⁡n→∞P ⁣(An) \probP*{\bigcup_{n \in \N} \evA_n} = \liminfty \probP{\evA_n} Let ∀n∈N:An+1⊂An\forall n \in \N: \evA_{n+1} \subset \evA_n, then P\P is continuous from above, i.e. P(⋂n∈NAn)=lim⁡n→∞P ⁣(An) \probP*{\bigcap_{n \in \N} \evA_n} = \liminfty \probP{\evA_n}
Proposition 1.4 (Union bound): For any sequence of events (An)n∈N\sequence{\evA}, one has P(⋃n∈NAn)≤∑n∈NP ⁣(An) \probP*{\bigcup_{n \in \N} \evA_n} \leq \sum_{n \in \N} \probP{\evA_n}

From now on and until the end of the chapter we fix an arbitrary probability space (Ω,F,P∗)(\Omega, \sigmaF, \P*).

1.2 Generated σ\sigma-Algebra

Let C={Ai | i∈I}\setC = \set{A_i \mid i \in \setI} be a collection of events. The σ\sigma-algebra generated by C\setC is the collection of all events whose occurence is determined once we know all events in C\setC.

Definition 1.5 (Generated σ\sigma-algebra): Let C⊃P(Ω)\setC \supset \powerset(\Omega), then σ(C)\sigma(\setC) is the σ\sigma-algebra generated by defined by σ(C)=⋂C⊂FF is a σ-algebraF \sigma(\setC) = \bigcap_{\stackrel{\sigmaF\text{ is a }\sigma\text{-algebra}}{\setC \subset \sigmaF}} \sigmaF
Note:
  1. σ(C)\sigma(\setC) is the smallest σ\sigma-algebra containing C\setC.
  2. If F\sigmaF is a σ\sigma-algebra, then σ(F)=F\sigma(\sigmaF) = \sigmaF.
  3. C⊂C′  ⟹  σ(C)⊂σ(C′)\setC \subset \setC' \implies \sigma(\setC) \subset \sigma(\setC').

1.3 Borel Sets

Definition 1.6 (Borel sets): Let (E,T)(E, \setT) be a topological space where T\setT is the set of open sets of EE. The Borel σ\sigma-algebra on EE is defined by B(E)=σ(T)\borelB(E) = \sigma(\setT).

Note that the sets contained in B(E)\borelB(E) are called Borel sets.

Proposition 1.7 (Borel sets of reals): For the real numbers R\R equipped with the standard topology generated by open intervals (a,b](a,b], we have B(R)=σ({(a,b] | a,b∈R}) \borelB(\R) = \sigma(\set{(a,b] \mid a,b \in \R})

Note that equivalently B(R)=σ({(a,b) | a,b∈R})=σ({(−∞,a] | a∈R})=σ({(−∞,a) | a∈R})\begin{align*} \borelB(\R) &= \sigma(\set{(a,b) \mid a,b \in \R}) \\ &= \sigma(\set{(-\infty,a] \mid a \in \R}) \\ &= \sigma(\set{(-\infty,a) \mid a \in \R}) \end{align*}

1.4 π\pi- and λ\lambda-Systems

Definition 1.8 (λ\lambda-system): A family of sets D⊂P(Ω)\setD \subset \powerset(\Omega) is called a Dynkin or λ\lambda-sytem if
  • Ω∈D\Omega \in \setD
  • A∈D  ⟹  Ac∈DA \in \setD \implies A^c \in \setD
  • A1,A2,…∈DA_1, A_2, \ldots \in \setD pairwise disjoint   ⟹  ⋃n∈NAn∈D\implies \bigcup_{n \in \N} A_n \in \setD

The notion of a λ\lambda-sytem is very similar to a σ\sigma-algebra. The only difference lies in the third item, where a weaker condition is asked: it suffices to be stable under disjoint countable union. In particular it is easier to be a λ\lambda-sytem than a σ\sigma-algebra. We always have D is a σ-algebra  ⟹  D is a λ-system \setD \text{ is a }\sigma\text{-algebra} \implies \setD \text{ is a }\lambda\text{-system} The converse does not hold in general. To see this, consider Ω={1,2,3,4}\Omega = \set{1,2,3,4} and D={∅,Ω,{1,2},{3,4},{1,3},{2,4}}\setD = \set{\varnothing, \Omega, \set{1,2}, \set{3,4}, \set{1,3}, \set{2,4}}, then D\setD is a λ\lambda-system but not a σ\sigma-algebra.

Definition 1.9 (π\pi-system): A family of sets C⊂P(Ω)\setC \subset \powerset(\Omega) is called a π\pi-system if it is stable under finite intersection, i.e. A,B∈C  ⟹  A∩B∈CA, B \in \setC \implies A \cap B \in \setC.
Proposition 1.10 (π\pi-λ\lambda implies σ\sigma): Let D⊂P(Ω)\setD \subset \powerset(\Omega) be a π\pi-system. Then D is a σ-algebra  ⟺  D is a λ-system \setD \text{ is a }\sigma\text{-algebra} \iff \setD \text{ is a }\lambda\text{-system}

1.5 Dynkin Theorem

Definition 1.11 (Generated λ\lambda-system): Let C⊂P(Ω)\setC \subset \powerset(\Omega). The λ\lambda-system generated by C\setC is the family defined by λ(C)=⋂C⊂DD is a λ-sytemD \lambda(\setC) = \bigcap_{\stackrel{\setD\text{ is a }\lambda\text{-sytem}}{\setC \subset \setD}} \setD

As for generated σ\sigma-algebras, it follows from the definitions that λ(C)\lambda(\setC) is the smallest λ\lambda-sytem containing C\setC. In particular, since σ(C)\sigma(\setC) is a λ\lambda-system, we always have λ(C)⊂σ(C) \lambda(C) \subset \sigma(C) The converse inclusion does not hold in general. For example, consider Ω={1,2,3,4}\Omega = \set{1,2,3,4} and C={∅,Ω,{1,2},{3,4},{1,3},{2,4}}\setC = \set{\varnothing, \Omega, \set{1,2}, \set{3,4}, \set{1,3}, \set{2,4}}, then λ(C)=C\lambda(\setC) = \setC because C\setC is already a λ\lambda-system but σ(C)=P(Ω)\sigma(\setC) = \powerset(\Omega).

Dynkin Theorem asserts that the inclusion above is an equality when C\setC is a π\pi-system.

Theorem 1.12 (Dynkin π\pi-λ\lambda-Theorem): Let C\setC be a π\pi-sytem. Then, we have σ(C)=λ(C) \sigma(\setC) = \lambda(\setC)

Dynkin Theorem is useful as in probability one often wants to prove statements valid for all events in a generated σ\sigma-algebra, i.e. if PP is a property then we would be interested in ∀A∈σ(C):P(A)\forall \evA \in \sigma(\setC) : P(\evA). The family σ(C)\sigma(\setC) is generally difficult to access, and showing that a property is stable under general countable union may be delicate. Often, probability statements involve probability measures, and it is easier to prove that a statement is stable under disjoint countable union, i.e. ∀A∈λ(C):P(A)\forall A \in \lambda(C) : P(A) is often easier to show. If C\setC is stable under finite intersection, Dynkin Theorem allows us to deduce the validity of the statement on σ(C)\sigma(\setC).

1.6 Independence of σ\sigma-Algebras

The language of probability theory relies on measure theory, together with the concept of independence. We first recall the definition of independence for a collection of events.

Definition 1.13 (Indepence of events): Let I\setI be an arbitrary, not necessarily finite index set. A collection {Ai | i∈I}\set{\evA_i \mid i \in \setI} of events is said to be independent if ∀J⊂I finite:P(⋂j∈JAj)=∏j∈JP ⁣(Aj) \forall \setJ \subset \setI \text{ finite}: \probP*{\bigcap_{j \in \setJ} \evA_j} = \prod_{j \in \setJ} \probP{\evA_j}

The concept of independence extend to family of events and in particular σ\sigma-algebras.

Definition 1.14 (Indepence of family of events): Let I\setI be an arbitrary index set. For every i∈Ii \in \setI, let Ei⊂F\setE_i \subset \sigmaF be a family of events. The collection {Ei | i∈I}\set{\setE_i \mid i \in \setI} is said to be independent if ∀J⊂I finite,  ∀j∈J,  ∀Akj∈Ej:P(⋂j∈JAkj)=∏j∈JP ⁣(Akj) \forall \setJ \subset \setI \text{ finite}, \sep \forall j \in \setJ, \sep \forall \evA_{k_j} \in \setE_j : \probP*{\bigcap_{j \in \setJ} \evA_{k_j}} = \prod_{j \in \setJ} \probP{\evA_{k_j}}
Note:
  • If each family is made of a single event, Ei={Ai}\setE_i = \set{\evA_i}, then the independence of the families Ei\setE_i is equivalent to the independence of the events Ai\evA_i.
  • {Ei | i∈I} independent  ⟺  ∀J⊂I finite:{Ei | i∈J} independent\set{\setE_i \mid i \in \setI} \text{ independent} \iff \forall \setJ \subset \setI \text{ finite} : \set{\setE_i \mid i \in \setJ} \text{ independent}
  • Let n≥1n \geq 1, the families E1,…,En⊂F\setE_1, \ldots, \setE_n \subset \sigmaF are independent if and only if ∀Akj∈Ej∪{Ω}:P(⋂j∈JAkj)=∏j∈JP ⁣(Akj) \forall A_{k_j} \in \setE_j \cup \set{\Omega} : \probP*{\bigcap_{j \in \setJ} \evA_{k_j}} = \prod_{j \in \setJ} \probP{\evA_{k_j}}

To establish independence of σ\sigma-algebras, it suffices to prove independence of generating π\pi-systems:

Proposition 1.15 (Independence of σ\sigma-algebras): Let I\setI be an arbitrary index set. For the collection of π\pi-systems {Ci⊂F | i∈I}\set{\setC_i \subset \sigmaF \mid i \in \setI} the following equivalence holds: {Ci⊂F | i∈I} independent  ⟺  {σ(Ci)⊂F | i∈I} independent \set{\setC_i \subset \sigmaF \mid i \in \setI} \text{ independent} \iff \set{\sigma(\setC_i) \subset \sigmaF \mid i \in \setI} \text{ independent}
Proposition 1.16 (Independence of complements): Let I\setI be an arbitrary index set and let {Ai | i∈I}\set{\evA_i \mid i \in \setI} be a collection of events, then {Ai | i∈I} independent  ⟺  {Aic | i∈I} independent \set{\evA_i \mid i \in \setI} \text{ independent} \iff \set{\evA_i^c \mid i \in \setI} \text{ independent}