The Open UniversitySkip to content
 

A Confident Information First Principle for Parametric Reduction and Model Selection of Boltzmann Machines

Zhao, Xiaozhao; Hou, Yuexian; Song, Dawei and Li, Wenjie (2017). A Confident Information First Principle for Parametric Reduction and Model Selection of Boltzmann Machines. IEEE Transactions on Neural Networks and Learning Systems. (Early Access).

Full text available as:
[img]
Preview
PDF (Accepted Manuscript) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (573kB) | Preview
DOI (Digital Object Identifier) Link: https://doi.org/10.1109/TNNLS.2017.2664100
Google Scholar: Look up in Google Scholar

Abstract

Typical dimensionality reduction (DR) methods are data-oriented, focusing on directly reducing the number of random variables (or features) while retaining the maximal variations in the high-dimensional data. Targeting unsupervised situations, this paper aims to address the problem from a novel perspective and considers model-oriented dimensionality reduction in parameter spaces of binary multivariate distributions. Specifically, we propose a general parameter reduction criterion, called Confident-Information-First (CIF) principle, to maximally preserve confident parameters and rule out less confident ones. Formally, the confidence of each parameter can be assessed by its contribution to the expected Fisher information distance within a geometric manifold over the neighbourhood of the underlying real distribution. Then we demonstrate two implementations of CIF in different scenarios. First, when there are no observed samples, we revisit the Boltzmann Machines (BM) from a model selection perspective and theoretically show that both the fully visible BM (VBM) and the BM with hidden units can be derived from the general binary multivariate distribution using the CIF principle. This finding would help us uncover and formalize the essential parts of the target density that BM aims to capture and the non-essential parts that BM should discard. Second, when there exist observed samples, we apply CIF to the model selection for BM, which is in turn made adaptive to the observed samples. The sample-specific CIF is a heuristic method to decide the priority order of parameters, which can improve the search efficiency without degrading the quality of model selection results as shown in a series of density estimation experiments.

Item Type: Journal Item
Copyright Holders: 2017 IEEE
ISSN: 2162-237X
Keywords: information geometry; Boltzmann Machine; parametric reduction; Fisher information
Academic Unit/School: Faculty of Science, Technology, Engineering and Mathematics (STEM)
Faculty of Science, Technology, Engineering and Mathematics (STEM) > Computing and Communications
Related URLs:
Item ID: 48465
Depositing User: Dawei Song
Date Deposited: 13 Feb 2017 16:12
Last Modified: 06 Oct 2017 08:41
URI: http://oro.open.ac.uk/id/eprint/48465
Share this page:

Altmetrics

Download history for this item

These details should be considered as only a guide to the number of downloads performed manually. Algorithmic methods have been applied in an attempt to remove automated downloads from the displayed statistics but no guarantee can be made as to the accuracy of the figures.

Actions (login may be required)

Policies | Disclaimer

© The Open University   contact the OU