Importance sampling the union of rare events with an application to power systems analysis

We consider importance sampling to estimate the probability μ of a union of J rare events H _j defined by a random variable x. The sampler we study has been used in spatial statistics, genomics and combinatorics going back at least to Karp and Luby (1983). It works by sampling one event at random, then sampling x conditionally on that event happening and it constructs an unbiased estimate of μ by multiplying an inverse moment of the number of occuring events by the union bound. We prove some variance bounds for this sampler. For a sample size of n, it has a variance no larger than (Formula Presented) where (Formula Presented) is the union bound. It also has a coefficient of variation no larger than (Formula Presented) regardless of the overlap pattern among the J events. Our motivating problem comes from power system reliability, where the phase differences between connected nodes have a joint Gaussian distribution and the J rare events arise from unacceptably large phase differences. In the grid reliability problems even some events defined by 5772 constraints in 326 dimensions, with probability below 10 ⁻²² , are estimated with a coefficient of variation of about 0.0024 with only n = 10,000 sample values.