Confidence Inference Function in Big Data

Statistical inference along with the strategy of divide-and-combine for Big Data analysis has been little studied.  As an effective inferential tool, confidence distribution (CD) has attracted a surge of renewed attention. The essence in constructing confidence distribution pertains to the availability of suitable pivotal quantities, which are usually obtained from the (asymptotical) distribution of point maximum likelihood estimator. We propose to use inference function, from which the parameter estimate is obtained, as the basis of constructing the pivotal. Compared to the existing CD, the proposed confidence inference function (CIF) inherits several advantages of estimating functions. In addition, the proposed CIF is closely related to the generalized method of moments (GMM) and Crowder’s optimality.  Thus, CIF, which includes maximum likelihood estimation as a special case, provides us a unified framework for many kinds of statistical methods, which is illustrated via numerical examples in the context of divide-and-combine approaches to Big Data analysis. 

http://www-personal.umich.edu/~pxsong/