Download Moments of Satisfaction: Statistical Properties of a Large Random K-CNF formula

Moments of satisfaction: statistical properties of a
large random K-CNF formula
PhysComp96
Extended abstract
Draft, 12 May 1996
Lidror Troyanskyand Naftali Tishby
Institute of Computer Science and Center for Neural Computation
The Hebrew University
Jerusalem 91904, Israel
Little is known about the distribution and structure of satisfying assignments of large K-CNF formulae. One interesting feature, observed by numerical simulations, is that for K 2, there is a
sharp transition from a phase in which almost all formulae are satisable to a phase in which almost
all formulae are un-satisable, as the ratio = M
N goes through some critical value. So far there is
no complete theoretical understanding of this phenomenon for K > 2.
In this paper we study the sat-assignment distribution and overlap, of a large random K-CNF
formula. In particular, we study analytically the second, third and fourth moments of the number
of satisfying assignments and overlap between satisfying assignments. We show that for K > 4 the
average overlap between satisfying assignments undergoes a discontinues (rst order) transition from
a low to high overlap of sat-assignments. This transition occurs at a value of which is just below
the numerically observed critical value. Similar, higher order, transition occur for higher moments of
the distribution for larger K . The transition in the overlap is observed numerically even for N 20.
1 Introduction
averaging, a single typical large random formula exhibits
the same transition, at the same critical value of c . For
this reason, the properties of a single typical formula can
be deduced from the properties of the ensemble of all formulae with the given K and .
A random K-CNF formula is a random structure of binary variables which are subjected to a set of constrains
due to the appearance of the same variables in many
clauses. A K-CNF formula can therefore be regarded as
a system of spins with \quenched" randomness induced
by the random clauses of the formula[11]. The number of
constrains increases with the value of until conicts between assignments of variables in dierent clauses appear
(\frustration") and the formula can no longer be satised.
For K = 2 there exist a complete theory of this phenomena, based on the analysis of the known polynomial time
algorithm for nding a satisfying assignment[4]. Since
no such algorithm is known for K > 2 there exist only
compelling numerical evidences for the transition[12]. Recently there has also been a \replica symmetric" (nonrigorous) statistical mechanical analysis of the problem,
which yields a good approximation to the value of the
entropy of satisfying assignments [13].
However, the special structure of K-CNF formulae,
which include both local weak constrains (\OR" inside
the clauses) and global strong constrains (\AND" of the
clauses) make the exact analytical study of the problem
rather dicult. We would like to clarify the picture near
the transition by directly studying the moments of the ensemble distribution of satisfying assignments for K > 2.
The rst moment of the number of satisfying assignments, i.e. the total number of satisfying assignments of
A K-Conjunctive-Normal-Form (K-CNF) of the N
boolean variables x1 ; x2; : : : ; xN is a boolean formula
made of a conjunction of M disjunctive clauses, each contains precisely K of the N variables or their negation. A
satisfying assignment to the formula is a specic assignment of the N variables in f?1; 1gN , which yields a true
value of the formula. By a random K-CNF we consider a
formula in which, given the number of clauses M, each literal in the formula is uniformly and independently drawn
from the set of the N variables and their negations.
Determining the satisability of a K-CNF formula is a
generic computational problem which became a cornerstone in the theory of computational complexity. In addition, K-CNF formulae serve as general paradigms for
boolean constrains satisfaction and as such are of tremendous practical interest. K-CNF's appear naturally in a
variety of applications, e.g., program and machine testing, VLSI design, logic programming, boolean inference,
machine learning, etc. The statistical properties of random K-CNF formulae, which are likely to appear with
an ecient encoding of the problem, attracts therefore
considerable attention (e.g., [1]{[7]).
When the number of variables in each clause, K 2,
the space of all K-CNF formulae with M 1 clauses
over N 1 variables exhibits a sudden transition from a
phase in which almost all formulae are satisable (SAT)
to a phase in which almost all formulae are un-satisable
(UNSAT). This transition occurs as the value of MN
goes through a critical value c. In fact, due to self Supported in
part by the Clore foundation.
1
2 Evaluation of the second
moment
all the K-CNF formulae with N variables and M clauses,
divided by the total number of formulae, can easily be
evaluated ([5] [9]) as,
The number of pairs of satisfying assignments to a given
formula can be easily shown to be,
1 = (1 ? 2?k )M 2N :
Nas (Nas ? 1) Nas2 ;
2
2
so the expectation value of the number of pairs is asymptotically half of the second moment. In this section this
expectation value is directly evaluated.
Let Ti denote the set of all formulae satised by a given
assignment, Xi (i = 1 : : :2N ). Due to symmetry, the
cardinality of this set, jTij is independent of i, as can be
seen by considering a \rotation" (\gauge") transformation
of both the assignment and its set of formulae. Since only
one of the 2k possible congurations of the K variables in
each of the M clause unsatisfy the clause, we obtain,
(1)
jTi j = (1 ? 2?k )M jT j =: M jT j ;
This expression provides a simple upper bound to the
threshold of satisability, in the limit of large N and
M. When the rst moment is smaller than 1, the vast
majority of formulae are un-satisable. In the language
of statistical physics the logarithm of the rst moment
is referred as the annealed approximation to the zerotemperature entropy of the satisfying assignments. This
approximation ignores the uctuations or higher order
moments of the ensemble. Indeed, both numerical simulations and analytical results suggest that the transition
occurs way before this annealed bound, at least for small
values of K [10].
To gain better understanding of the SAT-UNSAT transition we investigate the next three moments of the distribution of satisfying assignments and sketch the method
for calculating other moments. The higher moments are
then used to bound the number of formulae with very
large number of satisfying assignments. In the limit of
large N and M (the \thermodynamic"limit) the moments
depends on simple order parameters. The second moment
is a function of the average overlap between pairs of satisfying assignments, averaged over all the formulae. Selfaveraging again, as often in large random systems, makes
this ensemble average overlap to be equal to the typical
overlap between satisfying assignments in a single large,
random formula. The order parameter is similar to the
Edwards-Anderson parameter for random spin systems[8].
A remarkable feature of the transition, reveled for the
rst time by our analysis, is that for K > 4 the average
overlap undergoes a discontinuous behavior at a value of close to the numerical estimate of the SAT-UNSAT transition. This overlap transition follows a break-down of the
single typical value of the overlap into two distinct values (this is known as \replica symmetry breaking" in the
context of the replica calculation). When goes through
the transition the distribution of the overlap moves from
small values of the overlap, which correspond to uncorrelated sat-assignments, to high values of the overlap, corresponding to very similar sat-assignments. The discontinuous shift in the value of the overlap indicates a drastic change in the structure of the satisfying assignments,
which is a precursor of the SAT-UNSAT phase transition.
The overlap transition is marked by a discontinuity in the
2
derivative of the second moment with respect to , @
@ .
The third and fourth moments exhibit similar behavior
for higher values of K.
0
where jT j is the total number of K-CNF formulae with N
variables and M clauses. The probability that the pair of
assignments, Xi and Xj , satisfy a random formula, whose
set of satisfying assignments is denoted by L, is given by
(2)
P[Xi 2 L \ Xj 2 L] = jTij\T jTj j :
It turns out that the above probability depends only
on the Hamming distance, or equivalently the overlap between the two assignments, since this is the only invariant
with respect to joint \rotations" of the assignments, for
all N. In order to evaluate the expected number of pairs
we need only to count the number of pairs of satisfying
assignments for all formulae. This can be performed by
considering one specic assignment and count the total
number of pairs of satisfying assignments in all its satised formulae, and then multiply by 2N , the total number
of assignments.
Without loss of generality we can study the case in
which one of the assignments is X1 = (1; 1; : : : :1), due to
the same \rotation" symmetry. The Hamming distance
d1i between X1 and another assignment, Xi , is simply the
relative number of ?1's in Xi .
By denition, the clauses in all the formulae in the set
T1 need to be in one of the 2k ? 1 congurations which
X1 satises, and thus must contain at least one variable
which is not negated. Consider now an arbitrary clause in
T1 which contain at least one variable whose value in the
assignment Xi is ?1. The probability that such a clause
is still satised by Xi is given by,
(3)
Pc = (1 ? 2k 1? 1 ) =: 1 :
2
The function F can be considered as the \free energy
function", since it is a sum of two competing extensive
terms, the entropy and
an \energy" term,
1?(1?d)k
N log2 (1 ? 2k?1 ). In the limit N ! 1 the above
sum is completely dominated its maximalterm, i.e. by the
value of d that maximizes F. This maximum (or \saddle
point") is determined by the solution of the equation
Consider an arbitrary formula in T1 . Let m be the
number of clauses that contain at least one variable whose
value in the assignment Xi is ?1. The number of formulae
with m such clauses is distributed according to a binomial
distribution, with,
P0(d1i ) = 1 ? k1i = 1 ? (1 ? d1i)k :
(4)
As we expected, this distribution is a function of the
boolean variables only through the Hamming distance between the two assignments.
The size of the set T1 \ Ti can now be evaluated by
dividing T1 into \shells" with a xed value of m, with
respect to Xi , denoted by (T1(i;m) ). The relative number
of formulae in each such shell that belong also to Ti is
given by
jT1(i;m) \ Ti j = (1 ? 1 )m = m :
1
jT j
2k ? 1
@F(; d; k) = 0:
@d
Denoting this saddle point value of d by d~ we obtain,
~ k?1
~
=0;
(10)
log d ~ + k k(1 ? d)~ k
1 ? d 2 + (1 ? d) ? 2
or equivalently
(5)
k
~k
~
= 2 + (1 ?~d)k?1? 2 log 1 ?~ d :
d
k(1 ? d)
Each assignment whose Hamming distance from X1 is d
contributes
(11)
For K 4 there is a unique solution to the saddle point
(6) equation. This solution determines the most prevalent
distance between satisfying assignments as a function of
m=1
.
to the total number of satisfying assignments in T1 .
Since our binary variables have values of 1, a more
The number of assignments with Hamming distance d natural
order parameter then the Hamming distance is
from X1 is
the
overlap
between the assignments, i.e., the normalized
n
scalar
product
of the two assignments
dn
: 1 Xi Xj = 1 ? 2dij :
and the total number of satisfying assignments in T1 is
(12)
=
q
ij
N
given by
M
X
M P (d)m (1 ? P (d))(M ?m) m jT j
0
1
1
m 0
n X
n
X
M M 0.8
P0(d)m
dn=1 d n m=0 m
(1 ? P0(d))(M ?m) 1m jT1j
=
dn=1
n
dn
(d) M :
1 ? 2Pk0?
1
0.6
0.5
q
n
X
0.7
(7)
0.4
"K=5, N=20"
"K=4, N=20"
"K=3, N=24"
"K=3, N=16"
"K=2, N=28"
"K=2, N=20"
"K=2, N=12"
0.3
0.2
We denote
0.1
? d)k
F(; d; k) H(d) + log2 (1 ? 1 ?2(1
(8)
0
k?1 ) ;
0
5
10
15
20
25
M/N
where H(d) is the binary entropy,
Figure 1: Theoretical vs. numerical values of q as a function
of for K = 2{5[11].
H(d) ?d log2(d) ? (1 ? d) log2 (1 ? d):
For large N and M N, equation (7) can be written,
The analytic saddle point value of q~ = 1 ? 2d~ is the
using the Stirling approximation, as
ensemble average over all pairs of satisfying assignments,
for the ensemble of all formulae. Due to self-averaging this
n
X
1
NF
(;d;k )
value
is also the average overlap between two satisfying
p
2
:
(9)
Nd(1
?
d)
assignments
of a typical large random formula.
dn=1
3
Fig. 1 shows a comparison between q~ and the experimental value of the average overlap between pairs of satisfying assignments,
*
45
40
35
+
1 X X
(13)
N i j i;j F :
and Apart from deviations in the large values of , which
are clearly nite size eects since they reduce with increasing N, there is an excellent agreement between our
theoretical q and the numerical results. In fact our evaluation of the overlap is performed with no approximation,
besides the large N limit, and is therefore an exact analytic result.
For K > 4 the function (~q) which we derived from
equation (11) is no longer single-valued for all . This is
typical for a rst order transition, where two stable solutions and one unstable solution occur together between
two spinodal points. In terms of the overlap, there is a
range of where there are two local maxima in this distribution of the overlap between satisfying assignments for
a typical formula. The bi-modality of this distribution
reects the fact that the space of satisfying assignments
splits into two clusters (\replica symmetry breaking").
Figure 2 depicts the experimental ensemble averages,
over 100 random formulae, of the overlap distributions
with K = 5, N = 25 and between 16 and 20. As can
be seen, at the value = 18 the distribution becomes bimodal, where the most likely value of the overlap jumps
discontinuously from a low value (near 0) to a much higher
value (close to 1), as the theory predicts. Figure 3 gives
the overlap histogram for a single (typical) formula with
K = 5, N = 25 and = 19, to demonstrate the selfaveraging.
30
25
N(q)
q =:
20
15
10
5
0
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
q
Figure 3: Overlap histogram for a single formula in the bimodal regime.
close to the numerical value of the SAT-UNSAT transition, 20:9. For K = 6 the overlap transition occurs at
= 42:8 { again, just below the numerical value of the
SAT-UNSAT transition, 43:2. The discontinuous jump
in the overlap are due to the fact that the local maximum with the higher q~ becomes the global maximum of
F at this value of . This change corresponds to a sharp
change in the structure of the assignments space from typical small overlap between assignments to a much small
set of assignments with high overlap, just before they disappear completely. This structural shift can have signicant computational ramications, particularly for the
design of random algorithms.
1
10
0.8
8
0.6
q
9
7
log(N(q))
0.4
6
0.2
5
4
0
16
17
18
19
20
21
22
23
24
25
26
17
18
19
20
21
M/N
22
23
24
25
26
3
2
0.2
1
0
16
0.15
F
18
20
M/N
−0.6
−0.4
0
−0.2
0.2
0.4
0.6
0.8
0.1
1
0.05
q
0
16
Figure 2: Ensemble average overlap histogram.
~ are plotted for K = 5
The functions q() and F((d))
in gure 4. For K = 5 the eective F shows a discontinuous change in its derivative at the value = 20:6, - very
Figure 4: Theoretical q and F as functions of .
4
4 Approximate SAT probability
3 Higher moments
Another interesting property of the random formula satisability problem is the scaling of the probability of satisability, near the critical . This scaling was rst demonstrated by Scott Kirkpatrick[11, 12]. In order to evaluate the scaled probability we could, in principle, use the
inclusion-exclusion formula,
X
X
PSAT = jT1 j jTi j ? jT1j2 jTi \ Tj j
i
ij
1
(15)
+ : : : jT j2N jT1 \ T2 \ : : :T2N j) :
1
P
0
~
F3 = [log(1 ? 2k ) + log(1 ? 2k ? 1 )
Evaluating this sum is generally impossible for large N.
P
P
1
2
In
fact, as we have seen, even the rst few terms are quite
+ log(1 ? 2k ? 1 ? 2k ? 2 )]
complicated.
However, in case that,
!
~
1 jT \ T \ : : :T j = jTij j ;
+H(~ ) + ~ H (3 ?~ 1)
j
2
jT jn 1 2
jT j
!
~
~
distance between all the assign(14) (as it is if the Hamming
+(1 ? ~ ))H ? (3 ?~ 1)=2 ;
1?
ments is exactly 21 ) all the elements in the j-th term in
the sum are all equal to (0M )j . In this limiting case the
where ~ = 1 ? d~ = (~q + 1)=2 is the saddle point value sum can be rewritten as,
of overlap between pairs of satisfying assignments that
2N N X
2 ( M )j = 1 ? ?1 ? M 2N ; (16)
brings F3 to its maximum. At the saddle point, the
value
PSAT =
0
0
k
j =1 j
of F~3 is determined by P0 = 1 ? ~ , P1 = ~k ? 3~2?1
k
and thus the probability,
and P2 = 1 ? 3~k + 2 3~2?1 .
?
N
?
N
The fourth moment can also be evaluated, using simiPUN ?SAT = 1 ? (0M ) 2 = 1 ? (0 )N 2 : (17)
lar techniques (with even more complicated calculations).
In order to observe the convergence of the probability
Plots of the normalized logarithm of the rst four mo1
of
satisability near the SAT-UNSAT transition, c, a
ments, i log(Mi ) as functions of , for K = 3, are given
re-scaled
parameter, ~ , was introduced[11, 12],
in gure 5.
~ ? c
0.7
The third moment of the number of satisfying assignments
can be evaluated from the expected number of triplets
of satisfying assignments, using a similar, though more
complicated, calculation.
Analysis of the various types of congurations of three
assignments, with counting arguments, yields a sum of
rather complicated terms. In the thermodynamic limit
the sum is dominated by the maximal 3-assignments \free
energy",
1st Moment
2nd Moment
3rd Moment
4th Moment
0.6
and
= c(~ + 1):
Equation (18) can now be rewritten as
0.5
0.4
1/i log(Mi)
c
PUN ?SAT = 1 ? (0c )(~ +1)N
0.3
2N
:
(18)
In the above approximation
ln 12 ; =) c = 1
c = ln(
0
2
0)
and
0.2
0.1
0
-0.1
0
1
2
3
M/N
4
5
(
~ +1)N
PUN ?SAT = 1 ? 21
6
Figure 5: Normalized logarithm of the rst four moments as
functions of for K = 3.
?N
~
= 1 ? 2 2N
5
2N
!2N
~
?! e?2N
:
(19)
This functional limit form serves as a good approximation for K > 2, providing that[11, 12]
The approximate critical , c =
[4] Chvatal, Vasek and B. Reed, \Mick gets some (the odds
are on his side)", Proc. 33rd IEEE symposium on Foundation of Computer Science (1992), 620{627.
[5] Chvatal and Endre Szemeredi, \Many hard examples
for resolution", Journal of the ACM 35 (1988), 759{768.
[6] Dubois, Olivier, \Counting the number of solutions for
instances of satisability", Theoretical Computing Science 81 (1991), 49{64.
[7] Dubois, Olivier and Jacques Carlier, \Probabilistic approach to satisability problem", Theoretical Computing
Science 81 (1991), 65{75.
[8] Edwards, S. F. and Phil W. Anderson, \Theory of spin
glasses", J. Phys. F 5 (1975), 965{974.
[9] Franco, John and Marvin Paull, \Probabilistic analysis
of the Davis Putman procedure for solving the satisability problem", Discrete Applied Mathematics, 5 (1983),
77{87.
[10] Kamath, Amil, Rajeev Motwani, Krishna Palem and
Paul Spirakis, \Tail bounds for occupancy and the satisability thresholds conjecture", Proc. 35rd IEEE symposium on Foundation of Computer Science (1994), 592{
603.
[11] Kirkpatrick, Scott, Geza Gyorgyi, Naftali Tishby and
Lidror Troyansky, \The Statistical Mechanics of KSatisfaction", Advances in Neural Information Processing
Systems 6, (1993) 439{446.
[12] Kirkpatrick, Scott and Bart Selman, \Critical behavior in the satisability of random Boolean expressions",
Science 264 (1994), 1297{1301.
[13] Monasson, Remi and Riccardo Zecchina, \The entropy of the K-satisfaction problem", Phys. Rev. Lett.
76 (1996), 3881{3884.
ln 1
2
ln( 0 )
is replaced
with the correct critical , which until now can only
be obtained from numerical simulations.
The number of variables is scaled, N 7?! N 1 , where
the exponent = (K) need to be found numerically.
Figure 6. shows the convergence with K of the scaled
approximate formula for K =2,3,4,5 and 6 to our analytic
limiting function. As can be seen there is a very good
agreement with the re-scaled experimental curve for
K > 2, which improves with increasing k's.
1
0.9
k=6, N = 40
k=5, N = 40
k=4, N = 65
k=3, N = 100
k=2, N = 500
annealed limit
fraction of formulae unsatisfied
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
-2
-1
0
1
2
y
3
4
5
6
Figure 6: Scaled probability of UN-SAT near the transition.
Acknowledgments
We would like to thank Nati Linial for pointing out to
us the existence of the SAT-UNSAT transition, and to
Geza Gyorgyi, Scott Kirkpatrick, and Sebastian Seung
for valuable discussions and collaboration in the course of
this study.
References
[1] Broder, Andrei, Alan Frieze, and Eli Upfal, \On the
satisability of random 3-CNF , Proc. of Symposium on
Discrete Algorithms (1993), 322{330.
[2] Chao, Ming-Te, and John Franco, \Probabilistic analysis of two heuristics for 3-satisability problem", SIAM
Journal on Computing, 15 (1986), 1106{1118.
[3] Chao, Ming-Te, and John Franco, \Probabilistic analysis of a generalization of the unit-clause literal selection
heuristics for the K satisable problem", Information science 51 (1990), 289{314.
6