Download Important Formulas for Statistics

Important Formulas for Statistics
Basic stuff
𝑉𝑎𝑟(𝑋) = 𝐶𝑜𝑣(𝑋, 𝑋) = 𝐸[(𝑋 − 𝜇)(𝑋 − 𝜇)] = 𝐸[(𝑋 − 𝜇)2 ] = 𝐸[𝑋 2 ] − (𝐸[𝑋])2
Variance
Properties
Expected
value
Properties
Is a linear operator, E[XY]=E[X]E[Y] if independant.
Limit in
probability
A sequence {Xn} of random variables converges in probability towards X if for all ε > 0
Properties of
prob.
convergence
Gaussian
Integral
Convergence in probability implies convergence in distribution
, x can be a linear function of the integrand variable
Joint,
Marginal and
Conditional
probability
Expectation
and var
conditional
Conditional
Probaibility
=
𝑓(𝑎,𝑏)
𝑓(𝑏)
Sum of a RVits mean
∑(𝑥𝑖 − 𝑥̅ ) = 0
Sample
Variance
Prop of log
Liebniz rule
on the
differential of
an integral
Binomial
Coefficient
(helpful in convolution of poisson)
Expectation
(useful for the proof of Rao Blackwell thm)
in
conditional
Transformations
One Variable
(discreet)
One Variable
(continuous)
Two Variables
Jacobian
Order Statistics
𝑛[𝐹(𝑥)]𝑛−1 𝑓(𝑥)
Maximum
In general
Same distribu.
Min
Between two
(diff
distribution)
=
Between min
and max
Idea behind
multiplication
of supports
𝑛(𝑛 − 1)[𝐹(𝑦) − 𝐹(𝑥)]𝑛−2 𝑓(𝑥)𝑓(𝑦)
If 1(xi<θ) then multiplication can be summarized in 1(x(n)<θ). Conversely if
1(xi<1/θ) 1(θ <1/x(n)).
Convolutions
Always check that the supports are compatible
to establish the limits of the
integration/summation
Remember that in a convolution the support can be seen as a straight line crossing the plane in (z,0)
and (0,z). (e.g. when adding two uniforms the limits of integration vary when z>1)
Sufficiency
Sufficiency (I) Conditional
distribution approach (By
definition)
𝑃𝜃 (𝑋 = 𝑥|𝑇 = 𝑡) = 𝑃(𝑋 = 𝑥|𝑇 = 𝑡). That is this pmf/pdf is free of θ.
𝑃(𝑋 = 𝑥) ∗ 𝑃(𝑇 = 𝑡|𝑋 = 𝑥) 𝑃(𝑋 = 𝑥)
𝑃𝜃 (𝑋 = 𝑥|𝑇 = 𝑡) =
=
∗ 1(𝑇(𝑥) = 𝑡)
𝑃(𝑇 = 𝑡)
𝑃(𝑇 = 𝑡)
′
𝑓𝑋|𝑇=𝑡 (𝑥) 𝑑𝑜𝑒𝑠𝑛 𝑡𝑖𝑛𝑣𝑜𝑙𝑣𝑒 𝜃
Sufficiency (II) Neyman
𝐹𝑎𝑐𝑡𝑜𝑟𝑖𝑧𝑎𝑡𝑖𝑜𝑛: 𝐿(𝜃, 𝑥) = ℎ(𝑥)𝑔(𝑇(𝑥), 𝜃) T(x) is suff.
Factorization theorem
Hint for proof: Dicreete case only: X is contained in T(x). Then
make g(T(x),θ)=Pθ(T(X)=T(x)).
: Use the distribution of T:P𝜃 (T(X) = t). = ∑
→:𝑇(→)=𝑡
𝑦
𝐿(𝜃) to divide in
𝑦
𝐿(𝜃, 𝑥) , to ultimately find that Pθ(X=x|T(X)=t) doesn’t depend on θ.
Minimal sufficiency:
Lehmann Scheffé (also
ℎ(𝑥, 𝑥𝑡 ; 𝜃) =
𝐿(𝜃,𝑥)
𝐿(𝜃,𝑥𝑡 )
↔ 𝑇(𝑥) = 𝑇(𝑥𝑡 ) 𝑠. 𝑡. ℎ 𝑖𝑠 𝑓𝑟𝑒𝑒 𝑓𝑟𝑜𝑚 𝜃 →
𝑇 𝑖𝑠 𝑚𝑖𝑛𝑠𝑢𝑓𝑓 𝑓𝑜𝑟 𝜃.
works with non identically
If we T(x)=T(y) only happens for x=y then we can look at the order
distr)
statistics.
Prof: The sufficiency is with a partition and the →, the minimal is
with ← and using the fact that T is a function of another sufficient.
Information
Helpful in: CRLB, Asymptotic MLE
Arcillarity
Good trick: If the distribution of Y=θX does not involve θ, then T=Xi/Xj, doesn’t either.
Families
i)Location,
ii)Scale
iii)Locationscale.
Ancillary in each family is easy to achieve by subtraction and/or
division by another element of the sample (or an order stat)
Completeness
A stat both Complete and
A function 1-1 of a minsuff is a minsuff
sufficient  minsuff
Basus theorem:
w is complete and sufficient for θ, U is ancillary,  U and W are
independent
Hint proof: WTS: Conditional distribution of U on W is the distribution
of U. Then use completeness using these distributions as functions.
UMVUE and Estimators
UMVUE: Smallest variance in the family of unbiased estimator.
How to find estimators?
Method of moments
Equate the theoretical moments to the sample ones
Maximum likelihood
1. Find the L 2. Ln(L) 3.Maximize (and check is maximum) using FOC
and SOC
Invariance of MLE
You can apply a function to the MLE st you get the MLE of the function
How to compare them?
Mean Squared error
Bias
̂) − 𝜽
B(𝜽)=𝑬(𝜽
BLUE
Sumatory (same weight) of the unbiased estimators
How to improve them?
Rao Blackwell (improve)
T is an unbiased est. of τ(θ), U is (jointly) sufficient for θ. 𝑊 =
𝑔(𝑢) = 𝐸𝑢 [𝑇|𝑈 = 𝑢]. W is unbiased for τ, and has lower variance
than T.
Proof: Use Confitional expectation to check that W is unbiased,
2
and use the Var(T)=E[(T- τ(θ)) ] with W on the middle.
CRLB (compare)
Hint proof: Use chain rule of τ’(θ) so τ’(θ)=Cov(TY)<V(T)V(Y) then
equate with the V(Y)=nE(…)
Lehman Scheffé (find)
T is an unbiased est. of τ(θ), U is (jointly) complete and sufficient
for θ. 𝑊 = 𝑔(𝑢) = 𝐸𝑢 [𝑇|𝑈 = 𝑢]. W is UMVUE.
Hint proof: Suppose W* another UMVUE , use completeness to
prove W*=W
‘How to apply?
1) Find a complete and suff stat for θ
2) Find a function of this compsuff stat that is unbiased for
the thing you are trying to estimate. Sometimes the
function could be a simple multiplication or sum of 1)
THE EXPONENTIAL FAMILY (and reasons to love it)
𝑓(𝑥, 𝜃) = 𝑎(𝜃) ∗ 𝑔(𝑥) ∗ 𝑒 𝑏(𝜃)∗𝑅(𝑥)
1. We can Easily find the Expected value 𝐸𝜃 (𝑅(𝑥)) =
−𝑎′(𝜃)
,
𝑎(𝜃)𝑏′(𝜃)
2. T(x)=ΣR(xi) is minsuff and complete.
3. It follows the MLR property
Tests
Errors
Simple vs simple:
Neymann Pearson Lemma
1)Define the test
STEPS
2)Find the Likelihood ratio
3) Choose the statistics of interest and define Rejection region: An MP
a level test always depend on the sufficient statistic.
4)Operate to a form that is implementable either by knowing the
distribution of the statistic (take it to chi or normal) or by using
𝜃
𝑃𝜃0 (𝑅) = 𝛼 (e.g. ∫𝑘 0 𝑓(𝑥)𝑑𝑥
Remember that if the statistic of interest is decreasing because of the
other constants then k must behave properly. (eg. In a normal when
µ0< µ1, then the rejection is (…)>z, but if µ0> µ1 then the rejection is
(…)<-z.
One side composite
UMP via Neymann Person
Fix a value θ1>θ0. Then we can find the simple vs simple, and say that if
it doesn’t matter which value of theta we choose it also applies here.
MLR property
One side composite with
or
MLR property (Karlin
Rubin)
If MLR Non decreasing:
IF MLR non increasing:
With T(x)
being a suff stat.
GLRT
1)Define the Rejection region
2)Calculate
3) The sup(…) is the MLE, so in the case in which the null hypothesis is
simple then is simply replacing, while for the alternative we shall use
the MLE.
4)Operate to a form that is implementable by operating the K with the
KNOWN variables, that is express the test in terms of the MLE and the
parameter of the null. Check if the exponents are making a two sided
inequality.
𝜃
5)Apply 𝑃𝜃0 (𝑅) = 𝛼 (e.g. ∫𝑘 0 𝑓(𝑥)𝑑𝑥 )
Asymptotic
FOR MLE
Delta Method
You can then operate the left inside affecting the right as regular normal. E.g. multipliying by a
constant will multiply the µ but will be squared with the σ
You can also apply the delta method in this case, in which case you arrive to MLE being Efficient (by
CRLB).
Confidence intervals
Assymptotic: