* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Moments of Satisfaction: Statistical Properties of a Large Random K-CNF formula
Survey
Document related concepts
Renormalization group wikipedia , lookup
Birthday problem wikipedia , lookup
Vector generalized linear model wikipedia , lookup
Knapsack problem wikipedia , lookup
Granular computing wikipedia , lookup
Mean field particle methods wikipedia , lookup
Generalized linear model wikipedia , lookup
Numerical continuation wikipedia , lookup
Pattern recognition wikipedia , lookup
Drift plus penalty wikipedia , lookup
Least squares wikipedia , lookup
Eigenstate thermalization hypothesis wikipedia , lookup
Transcript
Moments of satisfaction: statistical properties of a large random K-CNF formula PhysComp96 Extended abstract Draft, 12 May 1996 Lidror Troyanskyand Naftali Tishby Institute of Computer Science and Center for Neural Computation The Hebrew University Jerusalem 91904, Israel Little is known about the distribution and structure of satisfying assignments of large K-CNF formulae. One interesting feature, observed by numerical simulations, is that for K 2, there is a sharp transition from a phase in which almost all formulae are satisable to a phase in which almost all formulae are un-satisable, as the ratio = M N goes through some critical value. So far there is no complete theoretical understanding of this phenomenon for K > 2. In this paper we study the sat-assignment distribution and overlap, of a large random K-CNF formula. In particular, we study analytically the second, third and fourth moments of the number of satisfying assignments and overlap between satisfying assignments. We show that for K > 4 the average overlap between satisfying assignments undergoes a discontinues (rst order) transition from a low to high overlap of sat-assignments. This transition occurs at a value of which is just below the numerically observed critical value. Similar, higher order, transition occur for higher moments of the distribution for larger K . The transition in the overlap is observed numerically even for N 20. 1 Introduction averaging, a single typical large random formula exhibits the same transition, at the same critical value of c . For this reason, the properties of a single typical formula can be deduced from the properties of the ensemble of all formulae with the given K and . A random K-CNF formula is a random structure of binary variables which are subjected to a set of constrains due to the appearance of the same variables in many clauses. A K-CNF formula can therefore be regarded as a system of spins with \quenched" randomness induced by the random clauses of the formula[11]. The number of constrains increases with the value of until conicts between assignments of variables in dierent clauses appear (\frustration") and the formula can no longer be satised. For K = 2 there exist a complete theory of this phenomena, based on the analysis of the known polynomial time algorithm for nding a satisfying assignment[4]. Since no such algorithm is known for K > 2 there exist only compelling numerical evidences for the transition[12]. Recently there has also been a \replica symmetric" (nonrigorous) statistical mechanical analysis of the problem, which yields a good approximation to the value of the entropy of satisfying assignments [13]. However, the special structure of K-CNF formulae, which include both local weak constrains (\OR" inside the clauses) and global strong constrains (\AND" of the clauses) make the exact analytical study of the problem rather dicult. We would like to clarify the picture near the transition by directly studying the moments of the ensemble distribution of satisfying assignments for K > 2. The rst moment of the number of satisfying assignments, i.e. the total number of satisfying assignments of A K-Conjunctive-Normal-Form (K-CNF) of the N boolean variables x1 ; x2; : : : ; xN is a boolean formula made of a conjunction of M disjunctive clauses, each contains precisely K of the N variables or their negation. A satisfying assignment to the formula is a specic assignment of the N variables in f?1; 1gN , which yields a true value of the formula. By a random K-CNF we consider a formula in which, given the number of clauses M, each literal in the formula is uniformly and independently drawn from the set of the N variables and their negations. Determining the satisability of a K-CNF formula is a generic computational problem which became a cornerstone in the theory of computational complexity. In addition, K-CNF formulae serve as general paradigms for boolean constrains satisfaction and as such are of tremendous practical interest. K-CNF's appear naturally in a variety of applications, e.g., program and machine testing, VLSI design, logic programming, boolean inference, machine learning, etc. The statistical properties of random K-CNF formulae, which are likely to appear with an ecient encoding of the problem, attracts therefore considerable attention (e.g., [1]{[7]). When the number of variables in each clause, K 2, the space of all K-CNF formulae with M 1 clauses over N 1 variables exhibits a sudden transition from a phase in which almost all formulae are satisable (SAT) to a phase in which almost all formulae are un-satisable (UNSAT). This transition occurs as the value of MN goes through a critical value c. In fact, due to self Supported in part by the Clore foundation. 1 2 Evaluation of the second moment all the K-CNF formulae with N variables and M clauses, divided by the total number of formulae, can easily be evaluated ([5] [9]) as, The number of pairs of satisfying assignments to a given formula can be easily shown to be, 1 = (1 ? 2?k )M 2N : Nas (Nas ? 1) Nas2 ; 2 2 so the expectation value of the number of pairs is asymptotically half of the second moment. In this section this expectation value is directly evaluated. Let Ti denote the set of all formulae satised by a given assignment, Xi (i = 1 : : :2N ). Due to symmetry, the cardinality of this set, jTij is independent of i, as can be seen by considering a \rotation" (\gauge") transformation of both the assignment and its set of formulae. Since only one of the 2k possible congurations of the K variables in each of the M clause unsatisfy the clause, we obtain, (1) jTi j = (1 ? 2?k )M jT j =: M jT j ; This expression provides a simple upper bound to the threshold of satisability, in the limit of large N and M. When the rst moment is smaller than 1, the vast majority of formulae are un-satisable. In the language of statistical physics the logarithm of the rst moment is referred as the annealed approximation to the zerotemperature entropy of the satisfying assignments. This approximation ignores the uctuations or higher order moments of the ensemble. Indeed, both numerical simulations and analytical results suggest that the transition occurs way before this annealed bound, at least for small values of K [10]. To gain better understanding of the SAT-UNSAT transition we investigate the next three moments of the distribution of satisfying assignments and sketch the method for calculating other moments. The higher moments are then used to bound the number of formulae with very large number of satisfying assignments. In the limit of large N and M (the \thermodynamic"limit) the moments depends on simple order parameters. The second moment is a function of the average overlap between pairs of satisfying assignments, averaged over all the formulae. Selfaveraging again, as often in large random systems, makes this ensemble average overlap to be equal to the typical overlap between satisfying assignments in a single large, random formula. The order parameter is similar to the Edwards-Anderson parameter for random spin systems[8]. A remarkable feature of the transition, reveled for the rst time by our analysis, is that for K > 4 the average overlap undergoes a discontinuous behavior at a value of close to the numerical estimate of the SAT-UNSAT transition. This overlap transition follows a break-down of the single typical value of the overlap into two distinct values (this is known as \replica symmetry breaking" in the context of the replica calculation). When goes through the transition the distribution of the overlap moves from small values of the overlap, which correspond to uncorrelated sat-assignments, to high values of the overlap, corresponding to very similar sat-assignments. The discontinuous shift in the value of the overlap indicates a drastic change in the structure of the satisfying assignments, which is a precursor of the SAT-UNSAT phase transition. The overlap transition is marked by a discontinuity in the 2 derivative of the second moment with respect to , @ @ . The third and fourth moments exhibit similar behavior for higher values of K. 0 where jT j is the total number of K-CNF formulae with N variables and M clauses. The probability that the pair of assignments, Xi and Xj , satisfy a random formula, whose set of satisfying assignments is denoted by L, is given by (2) P[Xi 2 L \ Xj 2 L] = jTij\T jTj j : It turns out that the above probability depends only on the Hamming distance, or equivalently the overlap between the two assignments, since this is the only invariant with respect to joint \rotations" of the assignments, for all N. In order to evaluate the expected number of pairs we need only to count the number of pairs of satisfying assignments for all formulae. This can be performed by considering one specic assignment and count the total number of pairs of satisfying assignments in all its satised formulae, and then multiply by 2N , the total number of assignments. Without loss of generality we can study the case in which one of the assignments is X1 = (1; 1; : : : :1), due to the same \rotation" symmetry. The Hamming distance d1i between X1 and another assignment, Xi , is simply the relative number of ?1's in Xi . By denition, the clauses in all the formulae in the set T1 need to be in one of the 2k ? 1 congurations which X1 satises, and thus must contain at least one variable which is not negated. Consider now an arbitrary clause in T1 which contain at least one variable whose value in the assignment Xi is ?1. The probability that such a clause is still satised by Xi is given by, (3) Pc = (1 ? 2k 1? 1 ) =: 1 : 2 The function F can be considered as the \free energy function", since it is a sum of two competing extensive terms, the entropy and an \energy" term, 1?(1?d)k N log2 (1 ? 2k?1 ). In the limit N ! 1 the above sum is completely dominated its maximalterm, i.e. by the value of d that maximizes F. This maximum (or \saddle point") is determined by the solution of the equation Consider an arbitrary formula in T1 . Let m be the number of clauses that contain at least one variable whose value in the assignment Xi is ?1. The number of formulae with m such clauses is distributed according to a binomial distribution, with, P0(d1i ) = 1 ? k1i = 1 ? (1 ? d1i)k : (4) As we expected, this distribution is a function of the boolean variables only through the Hamming distance between the two assignments. The size of the set T1 \ Ti can now be evaluated by dividing T1 into \shells" with a xed value of m, with respect to Xi , denoted by (T1(i;m) ). The relative number of formulae in each such shell that belong also to Ti is given by jT1(i;m) \ Ti j = (1 ? 1 )m = m : 1 jT j 2k ? 1 @F(; d; k) = 0: @d Denoting this saddle point value of d by d~ we obtain, ~ k?1 ~ =0; (10) log d ~ + k k(1 ? d)~ k 1 ? d 2 + (1 ? d) ? 2 or equivalently (5) k ~k ~ = 2 + (1 ?~d)k?1? 2 log 1 ?~ d : d k(1 ? d) Each assignment whose Hamming distance from X1 is d contributes (11) For K 4 there is a unique solution to the saddle point (6) equation. This solution determines the most prevalent distance between satisfying assignments as a function of m=1 . to the total number of satisfying assignments in T1 . Since our binary variables have values of 1, a more The number of assignments with Hamming distance d natural order parameter then the Hamming distance is from X1 is the overlap between the assignments, i.e., the normalized n scalar product of the two assignments dn : 1 Xi Xj = 1 ? 2dij : and the total number of satisfying assignments in T1 is (12) = q ij N given by M X M P (d)m (1 ? P (d))(M ?m) m jT j 0 1 1 m 0 n X n X M M 0.8 P0(d)m dn=1 d n m=0 m (1 ? P0(d))(M ?m) 1m jT1j = dn=1 n dn (d) M : 1 ? 2Pk0? 1 0.6 0.5 q n X 0.7 (7) 0.4 "K=5, N=20" "K=4, N=20" "K=3, N=24" "K=3, N=16" "K=2, N=28" "K=2, N=20" "K=2, N=12" 0.3 0.2 We denote 0.1 ? d)k F(; d; k) H(d) + log2 (1 ? 1 ?2(1 (8) 0 k?1 ) ; 0 5 10 15 20 25 M/N where H(d) is the binary entropy, Figure 1: Theoretical vs. numerical values of q as a function of for K = 2{5[11]. H(d) ?d log2(d) ? (1 ? d) log2 (1 ? d): For large N and M N, equation (7) can be written, The analytic saddle point value of q~ = 1 ? 2d~ is the using the Stirling approximation, as ensemble average over all pairs of satisfying assignments, for the ensemble of all formulae. Due to self-averaging this n X 1 NF (;d;k ) value is also the average overlap between two satisfying p 2 : (9) Nd(1 ? d) assignments of a typical large random formula. dn=1 3 Fig. 1 shows a comparison between q~ and the experimental value of the average overlap between pairs of satisfying assignments, * 45 40 35 + 1 X X (13) N i j i;j F : and Apart from deviations in the large values of , which are clearly nite size eects since they reduce with increasing N, there is an excellent agreement between our theoretical q and the numerical results. In fact our evaluation of the overlap is performed with no approximation, besides the large N limit, and is therefore an exact analytic result. For K > 4 the function (~q) which we derived from equation (11) is no longer single-valued for all . This is typical for a rst order transition, where two stable solutions and one unstable solution occur together between two spinodal points. In terms of the overlap, there is a range of where there are two local maxima in this distribution of the overlap between satisfying assignments for a typical formula. The bi-modality of this distribution reects the fact that the space of satisfying assignments splits into two clusters (\replica symmetry breaking"). Figure 2 depicts the experimental ensemble averages, over 100 random formulae, of the overlap distributions with K = 5, N = 25 and between 16 and 20. As can be seen, at the value = 18 the distribution becomes bimodal, where the most likely value of the overlap jumps discontinuously from a low value (near 0) to a much higher value (close to 1), as the theory predicts. Figure 3 gives the overlap histogram for a single (typical) formula with K = 5, N = 25 and = 19, to demonstrate the selfaveraging. 30 25 N(q) q =: 20 15 10 5 0 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 q Figure 3: Overlap histogram for a single formula in the bimodal regime. close to the numerical value of the SAT-UNSAT transition, 20:9. For K = 6 the overlap transition occurs at = 42:8 { again, just below the numerical value of the SAT-UNSAT transition, 43:2. The discontinuous jump in the overlap are due to the fact that the local maximum with the higher q~ becomes the global maximum of F at this value of . This change corresponds to a sharp change in the structure of the assignments space from typical small overlap between assignments to a much small set of assignments with high overlap, just before they disappear completely. This structural shift can have signicant computational ramications, particularly for the design of random algorithms. 1 10 0.8 8 0.6 q 9 7 log(N(q)) 0.4 6 0.2 5 4 0 16 17 18 19 20 21 22 23 24 25 26 17 18 19 20 21 M/N 22 23 24 25 26 3 2 0.2 1 0 16 0.15 F 18 20 M/N −0.6 −0.4 0 −0.2 0.2 0.4 0.6 0.8 0.1 1 0.05 q 0 16 Figure 2: Ensemble average overlap histogram. ~ are plotted for K = 5 The functions q() and F((d)) in gure 4. For K = 5 the eective F shows a discontinuous change in its derivative at the value = 20:6, - very Figure 4: Theoretical q and F as functions of . 4 4 Approximate SAT probability 3 Higher moments Another interesting property of the random formula satisability problem is the scaling of the probability of satisability, near the critical . This scaling was rst demonstrated by Scott Kirkpatrick[11, 12]. In order to evaluate the scaled probability we could, in principle, use the inclusion-exclusion formula, X X PSAT = jT1 j jTi j ? jT1j2 jTi \ Tj j i ij 1 (15) + : : : jT j2N jT1 \ T2 \ : : :T2N j) : 1 P 0 ~ F3 = [log(1 ? 2k ) + log(1 ? 2k ? 1 ) Evaluating this sum is generally impossible for large N. P P 1 2 In fact, as we have seen, even the rst few terms are quite + log(1 ? 2k ? 1 ? 2k ? 2 )] complicated. However, in case that, ! ~ 1 jT \ T \ : : :T j = jTij j ; +H(~ ) + ~ H (3 ?~ 1) j 2 jT jn 1 2 jT j ! ~ ~ distance between all the assign(14) (as it is if the Hamming +(1 ? ~ ))H ? (3 ?~ 1)=2 ; 1? ments is exactly 21 ) all the elements in the j-th term in the sum are all equal to (0M )j . In this limiting case the where ~ = 1 ? d~ = (~q + 1)=2 is the saddle point value sum can be rewritten as, of overlap between pairs of satisfying assignments that 2N N X 2 ( M )j = 1 ? ?1 ? M 2N ; (16) brings F3 to its maximum. At the saddle point, the value PSAT = 0 0 k j =1 j of F~3 is determined by P0 = 1 ? ~ , P1 = ~k ? 3~2?1 k and thus the probability, and P2 = 1 ? 3~k + 2 3~2?1 . ? N ? N The fourth moment can also be evaluated, using simiPUN ?SAT = 1 ? (0M ) 2 = 1 ? (0 )N 2 : (17) lar techniques (with even more complicated calculations). In order to observe the convergence of the probability Plots of the normalized logarithm of the rst four mo1 of satisability near the SAT-UNSAT transition, c, a ments, i log(Mi ) as functions of , for K = 3, are given re-scaled parameter, ~ , was introduced[11, 12], in gure 5. ~ ? c 0.7 The third moment of the number of satisfying assignments can be evaluated from the expected number of triplets of satisfying assignments, using a similar, though more complicated, calculation. Analysis of the various types of congurations of three assignments, with counting arguments, yields a sum of rather complicated terms. In the thermodynamic limit the sum is dominated by the maximal 3-assignments \free energy", 1st Moment 2nd Moment 3rd Moment 4th Moment 0.6 and = c(~ + 1): Equation (18) can now be rewritten as 0.5 0.4 1/i log(Mi) c PUN ?SAT = 1 ? (0c )(~ +1)N 0.3 2N : (18) In the above approximation ln 12 ; =) c = 1 c = ln( 0 2 0) and 0.2 0.1 0 -0.1 0 1 2 3 M/N 4 5 ( ~ +1)N PUN ?SAT = 1 ? 21 6 Figure 5: Normalized logarithm of the rst four moments as functions of for K = 3. ?N ~ = 1 ? 2 2N 5 2N !2N ~ ?! e?2N : (19) This functional limit form serves as a good approximation for K > 2, providing that[11, 12] The approximate critical , c = [4] Chvatal, Vasek and B. Reed, \Mick gets some (the odds are on his side)", Proc. 33rd IEEE symposium on Foundation of Computer Science (1992), 620{627. [5] Chvatal and Endre Szemeredi, \Many hard examples for resolution", Journal of the ACM 35 (1988), 759{768. [6] Dubois, Olivier, \Counting the number of solutions for instances of satisability", Theoretical Computing Science 81 (1991), 49{64. [7] Dubois, Olivier and Jacques Carlier, \Probabilistic approach to satisability problem", Theoretical Computing Science 81 (1991), 65{75. [8] Edwards, S. F. and Phil W. Anderson, \Theory of spin glasses", J. Phys. F 5 (1975), 965{974. [9] Franco, John and Marvin Paull, \Probabilistic analysis of the Davis Putman procedure for solving the satisability problem", Discrete Applied Mathematics, 5 (1983), 77{87. [10] Kamath, Amil, Rajeev Motwani, Krishna Palem and Paul Spirakis, \Tail bounds for occupancy and the satisability thresholds conjecture", Proc. 35rd IEEE symposium on Foundation of Computer Science (1994), 592{ 603. [11] Kirkpatrick, Scott, Geza Gyorgyi, Naftali Tishby and Lidror Troyansky, \The Statistical Mechanics of KSatisfaction", Advances in Neural Information Processing Systems 6, (1993) 439{446. [12] Kirkpatrick, Scott and Bart Selman, \Critical behavior in the satisability of random Boolean expressions", Science 264 (1994), 1297{1301. [13] Monasson, Remi and Riccardo Zecchina, \The entropy of the K-satisfaction problem", Phys. Rev. Lett. 76 (1996), 3881{3884. ln 1 2 ln( 0 ) is replaced with the correct critical , which until now can only be obtained from numerical simulations. The number of variables is scaled, N 7?! N 1 , where the exponent = (K) need to be found numerically. Figure 6. shows the convergence with K of the scaled approximate formula for K =2,3,4,5 and 6 to our analytic limiting function. As can be seen there is a very good agreement with the re-scaled experimental curve for K > 2, which improves with increasing k's. 1 0.9 k=6, N = 40 k=5, N = 40 k=4, N = 65 k=3, N = 100 k=2, N = 500 annealed limit fraction of formulae unsatisfied 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 -2 -1 0 1 2 y 3 4 5 6 Figure 6: Scaled probability of UN-SAT near the transition. Acknowledgments We would like to thank Nati Linial for pointing out to us the existence of the SAT-UNSAT transition, and to Geza Gyorgyi, Scott Kirkpatrick, and Sebastian Seung for valuable discussions and collaboration in the course of this study. References [1] Broder, Andrei, Alan Frieze, and Eli Upfal, \On the satisability of random 3-CNF , Proc. of Symposium on Discrete Algorithms (1993), 322{330. [2] Chao, Ming-Te, and John Franco, \Probabilistic analysis of two heuristics for 3-satisability problem", SIAM Journal on Computing, 15 (1986), 1106{1118. [3] Chao, Ming-Te, and John Franco, \Probabilistic analysis of a generalization of the unit-clause literal selection heuristics for the K satisable problem", Information science 51 (1990), 289{314. 6