Download An Introduction to Probability Theory and Its Applications

I
An Introduction
to Probability Theory
and Its Applications
WILLIAM FELLER (1906-1970)
Eugene Higgins Professor of Mathematics
Princeton University
VOLUME I
THIRD EDITION
Revised Printing
John Wiley & Sons, Inc.
New York
• London
• Sydney
Contents
CHAPTER
PAGE
INTRODUCTION:
1.
2.
3.
4.
5.
I
THE SAMPLE SPACE
1.
2.
3.
4.
5.
6.
7.
8.
II
THE NATURE OF PROBABILITY THEORY .
The Background
Procedure
"Statistical" Probability
Summary
Historical Note
1
1
3
4
5
6
7
The Empirical Background
Examples
The Sample Space. Events
Relations among Events
Discrete Sample Spaces
Probabilities in Discrete Sample Spaces: Preparations
The Basic Definitions and Rules
Problems for Solution
ELEMENTS OF COMBINATORIAL ANALYSIS
1. Preliminaries
2. Ordered Samples
3. Examples
4. Subpopulations and Partitions
*5. Application to Occupancy Problems
*5a. Bose-Einstein and Fermi-Dirac Statistics
*5b. Application to Runs
6. The Hypergeometric Distribution
7. Examples for Waiting Times
8. Binomial Coefficients
9. Stirling's Formula
Problems for Solution:
10. Exercises and Examples
7
9
13
14
17
19
22
24
26
26
28
31
34
38
40
42
43
47
50
52
54
54
* Starred sections are not required for the understanding of the sequel and should
be omitted at first reading.
xiii
XIV
CONTENTS
CHAPTER
PAGE
11. Problems and Complements of a Theoretical
Character
12. Problems and Identities Involving Binomial Coefficients
*IH
FLUCTUATIONS IN COIN TOSSING AND RANDOM WALKS .
1.
2.
3.
4.
*5.
6.
7.
8.
9.
10.
*IV
COMBINATION OF EVENTS
1.
2.
3.
4.
5.
6.
V
VI
General Orientation. The Reflection Principle . . .
Random Walks: Basic Notions and Notations . . .
The Main Lemma
Last Visits and Long Leads
Changes of Sign
An Experimental Illustration
Maxima and First Passages
Duality. Position of Maxima
An Equidistribution Theorem
Problems for Solution
63
67
68
73
76
78
84
86
88
91
94
95
98
Union of Events
Application to the Classical Occupancy Problem
The Realization of m among N events
Application to Matching and Guessing
Miscellany
Problems for Solution
CONDITIONAL PROBABILITY.
58
. .
98
101
106
107
109
Ill
STOCHASTIC INDEPENDENCE .
114
1. Conditional Probability
2. Probabilities Defined by Conditional Probabilities.
Urn Models
3. Stochastic Independence
4. Product Spaces. Independent Trials
*5. Applications to Genetics
*6. Sex-Linked Characters
*7. Selection
8. Problems for Solution
114
THE BINOMIAL AND THE POISSON DISTRIBUTIONS . . . .
1.
2.
3.
4.
Bernoulli Trials
The Binomial Distribution
The Central Term and the Tails
The Law of Large Numbers
118
125
128
132
136
139
140
146
146
147
150
152
CONTENTS
XV
CHAPTER
PAGE
5.
6.
7.
8.
9.
10.
VII
The Poisson Approximation
The Poisson Distribution
Observations Fitting the Poisson Distribution . . . .
Waiting Times. The Negative Binomial Distribution
The Multinomial Distribution
Problems for Solution . ^
THE NORMAL APPROXIMATION TO THE BINOMIAL
DISTRIBUTION
1.
2.
3.
4.
5.
*6.
7.
*VIII
Infinite Sequences of Trials
Systems of Gambling
The Borel-Cantelli Lemmas
The Strong Law of Large Numbers
The Law of the Iterated Logarithm
Interpretation in Number Theory Language
Problems for Solution
RANDOM VARIABLES;
1.
2.
3.
4.
5.
6.
*7.
*8.
9.
X
174
The Normal Distribution
Orientation: Symmetric Distributions
The DeMoivre-Laplace Limit Theorem
Examples
Relation to the Poisson Approximation
Large Deviations
Problems for Solution
174
179
182
187
190
192
193
UNLIMITED SEQUENCES OF BERNOULLI TRIALS
1.
2.
3.
4.
5.
6.
7.
IX
153
156
159
164
167
169
. . . .
196
196
198
200
202
204
208
210
EXPECTATION
212
Random Variables
Expectations
Examples and Applications
The Variance
Covariance; Variance of a Sum
Chebyshev's Inequality
Kolmogorov's Inequality
The Correlation Coefficient
Problems for Solution
212
220
223
227
229
233
234
236
237
LAWS OF LARGE NUMBERS
1. Identically Distributed Variables
*2. Proof of the Law of Large Numbers
3. The Theory of "Fair" Games
243
243
246
248
XVI
CONTENTS
CHAPTER
*4.
5.
*6.
*7.
8.
XI
*XII
264
264
266
270
275
279
280
283
BRANCHING PROCESSES . . .
Sums of a Random Number of Variables
The Compound Poisson Distribution
Processes with Independent Increments
Examples for Branching Processes
Extinction Probabilities in Branching Processes . . .
The Total Progeny in Branching Processes
Problems for Solution
286
286
288
292
293
295
298
301
RECURRENT EVENTS.
1.
2.
3.
4.
5.
6.
*7.
*8.
9.
10.
*11.
12.
251
253
256
258
261
Generalities
Convolutions
Equalizations and Waiting Times in Bernoulli Trials
Partial Fraction Expansions
Bivariate Generating Functions
The Continuity Theorem
Problems for Solution
COMPOUND DISTRIBUTIONS.
1.
2.
2a.
3.
4.
5.
6.
XIV
The Petersburg Game
Variable Distributions
Applications to Combinatorial Analysis
The Strong Law of Large Numbers
Problems for Solution
INTEGRAL VALUED VARIABLES. GENERATING FUNCTIONS .
1.
2.
3.
4.
5.
*6.
7.
XIII
PAGE
RENEWAL THEORY
Informal Preparations and Examples
Definitions
The Basic Relations
Examples
Delayed Recurrent Events. A General Limit Theorem
The Number of Occurrences of 8
Application to the Theory of Success Runs
More General Patterns
Lack of Memory of Geometric Waiting Times . . .
Renewal Theory
Proof of the Basic Limit Theorem
Problems for Solution
RANDOM WALK AND RUIN PROBLEMS
1. General Orientation
2. The Classical Ruin Problem
I
303
303
307
311
313
316
320
322
326
328
329
335
338
342
342
344
CONTENTS
XV11
CHAPTER
PAGE
3. Expected Duration of the Game
*4. Generating Functions for the Duration of the Game
and for the First-Passage Times
*5. Explicit Expressions
6. Connection with Diffusion Processes
*7. Random Walks in the Plane and Space
8. The Generalized One-Dimensional Random Walk
(Sequential Sampling)
9. Problems for Solution
XV
MARKOV CHAINS
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
*11.
*12.
13.
14.
*XVI
. .
428
428
432
436
438
443
.
444
444
446
448
451
454
458
General Theory
Examples
Random Walk with Reflecting Barriers
Transient States; Absorption Probabilities
Application to Recurrence Times
General Orientation. Markov Processes
The Poisson Process
The Pure Birth Process
Divergent Birth Processes
The Birth and Death Process
Exponential Holding Times
363
367
372
375
382
384
387
390
392
399
404
406
407
414
419
424
. . . .
THE SIMPLEST TIME-DEPENDENT STOCHASTIC PROCESSES
1.
2.
3.
*4.
5.
6.
349
352
354
359
372
ALGEBRAIC TREATMENT OF FINITE MARKOV CHAINS
1.
2.
3.
4.
5.
XVII
Definition
Illustrative Examples
Higher Transition Probabilities
Closures and Closed Sets
Classification of States
Irreducible Chains. Decompositions
Invariant Distributions
Transient Chains
Periodic Chains
Application to Card Shuffling
Invariant Measures. Ratio Limit Theorems
Reversed Chains. Boundaries
The General Markov Process
Problems for Solution
348
CONTENTS
XVlll
CHAPTER
7.
8.
9.
10.
PAGE
Waiting Line and Servicing Problems
The Backward (Retrospective) Equations
General Processes
Problems for Solution
460
468
470
478
ANSWERS TO PROBLEMS
483
INDEX
499