# Information criteria for non-normalized models

@article{Matsuda2019InformationCF, title={Information criteria for non-normalized models}, author={Takeru Matsuda and Masatoshi Uehara and Aapo Hyv{\"a}rinen}, journal={ArXiv}, year={2019}, volume={abs/1905.05976} }

Many statistical models are given in the form of non-normalized densities with an intractable normalization constant. Since maximum likelihood estimation is computationally intensive for these models, several estimation methods have been developed which do not require explicit computation of the normalization constant, such as noise contrastive estimation (NCE) and score matching. However, model selection methods for general non-normalized models have not been proposed so far. In this study, we… Expand

#### 2 Citations

On Selection Criteria for the Tuning Parameter in Robust Divergence

- Computer Science, Mathematics
- Entropy
- 2021

A selection criterion based on an asymptotic approximation of the Hyvarinen score applied to an unnormalized model defined by robust divergence is proposed, which demonstrates the usefulness of the proposed method via numerical studies using normal distributions and regularized linear regression. Expand

General Bayesian Loss Function Selection and the use of Improper Models

- Mathematics
- 2021

Statisticians often face the choice between using probability models or a paradigm defined by minimising a loss function. Both approaches are useful and, if the loss can be re-cast into a proper… Expand

#### References

SHOWING 1-10 OF 75 REFERENCES

Estimation of Non-Normalized Statistical Models by Score Matching

- Computer Science, Mathematics
- J. Mach. Learn. Res.
- 2005

While the estimation of the gradient of log-density function is, in principle, a very difficult non-parametric problem, it is proved a surprising result that gives a simple formula that simplifies to a sample average of a sum of some derivatives of the log- density given by the model. Expand

A Unified Statistically Efficient Estimation Framework for Unnormalized Models

- Computer Science
- AISTATS
- 2020

A unified, statistically efficient estimation framework for unnormalized models and several efficient estimators, whose asymptotic variance is the same as the MLE, is proposed. Expand

Estimation of Non-Normalized Mixture Models

- Computer Science
- AISTATS
- 2019

The proposed method provides a probabilistically principled clustering method that is able to utilize a deep representation and applications to clustering of natural images and neuroimag- ing data give promising results. Expand

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

- Mathematics, Computer Science
- AISTATS
- 2010

A new estimation principle is presented to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise, using the model log-density function in the regression nonlinearity, which leads to a consistent (convergent) estimator of the parameters. Expand

Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics

- Computer Science, Mathematics
- J. Mach. Learn. Res.
- 2012

The basic idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise and it is shown that the new method strikes a competitive trade-off in comparison to other estimation methods for unnormalized models. Expand

Some extensions of score matching

- Mathematics, Computer Science
- Comput. Stat. Data Anal.
- 2007

It is shown how to estimate non-normalized models defined in the non-negative real domain, i.e. R"+^n", and it is shown that the score matching estimator can be obtained in closed form for some exponential families. Expand

Analysis of Noise Contrastive Estimation from the Perspective of Asymptotic Variance

- Computer Science, Mathematics
- ArXiv
- 2018

This paper proposes a method for reducing asymptotic variance by estimating the parameters of the auxiliary distribution of NCE, and determines the form of the objective functions, where the asymPTotic variance takes the smallest values in the original estimator class and the proposed estimator classes. Expand

Generalized Score Matching for Non-Negative Data

- Medicine, Mathematics
- J. Mach. Learn. Res.
- 2019

This paper gives a generalized form of score matching for non-negative data that improves estimation efficiency and addresses an overlooked inexistence problem by generalizing the regularized score matching method of Lin et al. (2016) and improving its theoretical guarantees fornon-negative Gaussian graphical models. Expand

Bayesian Model Comparison with the Hyvärinen Score: Computation and Consistency

- Mathematics
- Journal of the American Statistical Association
- 2019

ABSTRACT The Bayes factor is a widely used criterion in model comparison and its logarithm is a difference of out-of-sample predictive scores under the logarithmic scoring rule. However, when some of… Expand

Imputation estimators for unnormalized models with missing data

- Mathematics, Computer Science
- AISTATS
- 2020

Simulation results with truncated Gaussian graphical models and the application to real data of wind direction reveal that the proposed methods effectively enable statistical inference with unnormalized models from missing data. Expand