Downloads & Free Reading Options - Results

The problems of model and variable selections for classification trees are jointly considered. A penalized criterion is proposed which explicitly takes into account the number of variables, and a risk bound inequality is provided for the tree classifier minimizing this criterion. This penalized criterion is compared to the one used during the pruning step of the CART algorithm. It is shown that the two criteria are similar under some specific margin assumptions. In practice, the tuning parameter of the CART penalty has to be calibrated by hold-out. Simulation studies are performed which confirm that the hold-out procedure mimics the form of the proposed penalized criterion.

We estimate the selection constant in the following geometric selection theorem by Pach: For every positive integer $d$ there is a constant $c_d > 0$ such that whenever $X_1,..., X_{d+1}$ are $n$-element subsets of $\mathbb{R}^d$, then we can find a point $\mathbf{p} \in \mathbb{R}^d$ and subsets $Y_i \subseteq X_i$ for every $i \in [d+1]$, each of size at least $c_d n$, such that $\mathbf{p}$ belongs to all {\em rainbow} $d$-simplices determined by $Y_1,..., Y_{d+1}$, that is, simplices with one vertex in each $Y_i$. We show a super-exponentially decreasing upper bound $c_d\leq e^{-(1/2-o(1))(d \ln d)}$. The ideas used in the proof of the upper bound also help us prove Pach's theorem with $c_d \geq 2^{-2^{d^2 + O(d)}}$, which is a lower bound doubly exponentially decreasing in $d$ (up to some polynomial in the exponent). For comparison, Pach's original approach yields a triply exponentially decreasing lower bound. On the other hand, Fox, Pach, and Suk recently obtained a hypergraph density result implying a proof of Pach's theorem with $c_d \geq2^{-O(d^2\log d)}$. In our construction for the upper bound, we use the fact that the minimum solid angle of every $d$-simplex is super-exponentially small. This fact was previously unknown and might be of independent interest. For the lower bound, we improve the "separation" part of the argument by showing that in one of the key steps only $d+1$ separations are necessary, compared to $2^d$ separations in the original proof. We also provide a measure version of Pach's theorem.

This paper deals with the problem of finding the globally optimal subset of h elements from a larger set of n elements in d space dimensions so as to minimize a quadratic criterion, with an special emphasis on applications to computing the Least Trimmed Squares Estimator (LTSE) for robust regression. The computation of the LTSE is a challenging subset selection problem involving a nonlinear program with continuous and binary variables, linked in a highly nonlinear fashion. The selection of a globally optimal subset using the branch and bound (BB) algorithm is limited to problems in very low dimension, tipically d

A pervasive task in the differential privacy literature is to select the $k$ items of "highest quality" out of a set of $d$ items, where the quality of each item depends on a sensitive dataset that must be protected. Variants of this task arise naturally in fundamental problems like feature selection and hypothesis testing, and also as subroutines for many sophisticated differentially private algorithms. The standard approaches to these tasks---repeated use of the exponential mechanism or the sparse vector technique---approximately solve this problem given a dataset of $n = O(\sqrt{k}\log d)$ samples. We provide a tight lower bound for some very simple variants of the private selection problem. Our lower bound shows that a sample of size $n = \Omega(\sqrt{k} \log d)$ is required even to achieve a very minimal accuracy guarantee. Our results are based on an extension of the fingerprinting method to sparse selection problems. Previously, the fingerprinting method has been used to provide tight lower bounds for answering an entire set of $d$ queries, but often only some much smaller set of $k$ queries are relevant. Our extension allows us to prove lower bounds that depend on both the number of relevant queries and the total number of queries.

In the Best-$k$-Arm problem, we are given $n$ stochastic bandit arms, each associated with an unknown reward distribution. We are required to identify the $k$ arms with the largest means by taking as few samples as possible. In this paper, we make progress towards a complete characterization of the instance-wise sample complexity bounds for the Best-$k$-Arm problem. On the lower bound side, we obtain a novel complexity term to measure the sample complexity that every Best-$k$-Arm instance requires. This is derived by an interesting and nontrivial reduction from the Best-$1$-Arm problem. We also provide an elimination-based algorithm that matches the instance-wise lower bound within doubly-logarithmic factors. The sample complexity of our algorithm strictly dominates the state-of-the-art for Best-$k$-Arm (module constant factors).

We introduce the model confidence bounds (MCBs) for variable selection in the context of nested parametric models. Similarly to the endpoints in the familiar confidence interval for parameter estimation, the MCBs identify two nested models (upper and lower confidence bound models) containing the true model at a given level of confidence. Instead of trusting a single selected model by a given selection method, the MCBs width and composition enable the practitioner to assess the overall model uncertainty. The MCBs methodology is implemented by a fast bootstrap algorithm which is shown to yield the correct asymptotic coverage under rather general conditions. A new graphical tool -- the model uncertainty curve (MUC) -- is introduced to visualize the variability of model selection and compare different model selection procedures. Our Monte Carlo simulations and real data examples confirm the validity and illustrate the advantages of the proposed method.

We introduce the model confidence bounds (MCBs) for variable selection in the context of nested parametric models. Similarly to the endpoints in the familiar confidence interval for parameter estimation, the MCBs identify two nested models (upper and lower confidence bound models) containing the true model at a given level of confidence. Instead of trusting a single selected model by a given selection method, the MCBs width and composition enable the practitioner to assess the overall model uncertainty. The MCBs methodology is implemented by a fast bootstrap algorithm which is shown to yield the correct asymptotic coverage under rather general conditions. A new graphical tool -- the model uncertainty curve (MUC) -- is introduced to visualize the variability of model selection and compare different model selection procedures. Our Monte Carlo simulations and real data examples confirm the validity and illustrate the advantages of the proposed method.

We consider the problem of estimating the underlying graph associated with a Markov random field, with the added twist that the decoding algorithm can iteratively choose which subsets of nodes to sample based on the previous samples, resulting in an active learning setting. Considering both Ising and Gaussian models, we provide algorithm-independent lower bounds for high-probability recovery within the class of degree-bounded graphs. Our main results are minimax lower bounds for the active setting that match the best known lower bounds for the passive setting, which in turn are known to be tight in several cases of interest. Our analysis is based on Fano's inequality, along with novel mutual information bounds for the active learning setting, and the application of restricted graph ensembles. While we consider ensembles that are similar or identical to those used in the passive setting, we require different analysis techniques, with a key challenge being bounding a mutual information quantity associated with observed subsets of nodes, as opposed to full observations.