Downloads & Free Reading Options - Results

Conformal prediction is a method of producing prediction sets that can be applied on top of a wide range of prediction algorithms. The method has a guaranteed coverage probability under the standard IID assumption regardless of whether the assumptions (often considerably more restrictive) of the underlying algorithm are satisfied. However, for the method to be really useful it is desirable that in the case where the assumptions of the underlying algorithm are satisfied, the conformal predictor loses little in efficiency as compared with the underlying algorithm (whereas being a conformal predictor, it has the stronger guarantee of validity). In this paper we explore the degree to which this additional requirement of efficiency is satisfied in the case of Bayesian ridge regression; we find that asymptotically conformal prediction sets differ little from ridge regression prediction intervals when the standard Bayesian assumptions are satisfied.

Accurate approximations to density functionals have recently been obtained via machine learning (ML). By applying ML to a simple function of one variable without any random sampling, we extract the qualitative dependence of errors on hyperparameters. We find universal features of the behavior in extreme limits, including both very small and very large length scales, and the noise-free limit. We show how such features arise in ML models of density functionals.

We investigate the application of sufficient dimension reduction (SDR) to a noiseless data set derived from a deterministic function of several variables. In this context, SDR provides a framework for ridge recovery. A ridge function is a function of a few linear combinations of the variables---i.e., a composition of a low-dimensional linear transformation and a nonlinear function. The goal of ridge recovery is: using only point evaluations of the function, estimate the subspace that is the span of the ridge vectors that define the linear combinations. SDR provides the foundation for algorithms that search for this ridge subspace. We study two inverse regression methods for SDR---sliced inverse regression (SIR) and sliced average variance estimation (SAVE)---that approximate the ridge subspace using point evaluations of the function at samples drawn from a given probability density function. We provide convergence results for these algorithms for solving the ridge recovery problem, and we demonstrate the methods on simple numerical test problems.

This work examined minimax linear estimation in multiple linear regression. The application of minimax estimation to regression led to the development of ridge regression estimators with stochastic ridge parameters. These estimators were seen to be invariant under linear transformation; a property which has not been established for other ridge estimators. These minimax-motivated estimators were examined in several simulation studies. In particular, flaws in other simulation studies of ridge estimators were depicted. Consequently, an improved simulation procedure was used. It was observed from these studies that, contrary to published statements, a ridge estimator can be considerably superior to the ordinary least squares estimator, especially when high pairwise correlations exist among the regression variables. Robustness considerations were used to suggest a requirement that a 'good' generalized ridge regression estimator should satisfy. (Author)

This work examined minimax linear estimation in multiple linear regression. The application of minimax estimation to regression led to the development of ridge regression estimators with stochastic ridge parameters. These estimators were seen to be invariant under linear transformation; a property which has not been established for other ridge estimators. These minimax-motivated estimators were examined in several simulation studies. In particular, flaws in other simulation studies of ridge estimators were depicted. Consequently, an improved simulation procedure was used. It was observed from these studies that, contrary to published statements, a ridge estimator can be considerably superior to the ordinary least squares estimator, especially when high pairwise correlations exist among the regression variables. Robustness considerations were used to suggest a requirement that a 'good' generalized ridge regression estimator should satisfy. (Author)

In this paper, we investigate a divide and conquer approach to Kernel Ridge Regression (KRR). Given n samples, the division step involves separating the points based on some underlying disjoint partition of the input space (possibly via clustering), and then computing a KRR estimate for each partition. The conquering step is simple: for each partition, we only consider its own local estimate for prediction. We establish conditions under which we can give generalization bounds for this estimator, as well as achieve optimal minimax rates. We also show that the approximation error component of the generalization error is lesser than when a single KRR estimate is fit on the data: thus providing both statistical and computational advantages over a single KRR estimate over the entire data (or an averaging over random partitions as in other recent work, [30]). Lastly, we provide experimental validation for our proposed estimator and our assumptions.

In this paper, we investigate a divide and conquer approach to Kernel Ridge Regression (KRR). Given n samples, the division step involves separating the points based on some underlying disjoint partition of the input space (possibly via clustering), and then computing a KRR estimate for each partition. The conquering step is simple: for each partition, we only consider its own local estimate for prediction. We establish conditions under which we can give generalization bounds for this estimator, as well as achieve optimal minimax rates. We also show that the approximation error component of the generalization error is lesser than when a single KRR estimate is fit on the data: thus providing both statistical and computational advantages over a single KRR estimate over the entire data (or an averaging over random partitions as in other recent work, [30]). Lastly, we provide experimental validation for our proposed estimator and our assumptions.

Bibliography: l. 36

A class of estimators, variously known as ridge estimators, is considered for the linear regression model Y=X theta + epsilon, where theta is an unknown parameter vector to be estimated. Some properties of the ridge estimators are given. It is shown that certain ridge estimators have uniformly smaller mean squared error than the least squares estimator. (Author)

视频主要在EViews 12中演示弹性净估计的新功能，包括时间序列交叉验证、可变权重和估计后诊断。 完整的视频教程，请登录 科学软件网 www.sciencesoftware.com.cn 观看。

This paper derives an identity connecting the square loss of ridge regression in on-line mode with the loss of the retrospectively best regressor. Some corollaries about the properties of the cumulative loss of on-line ridge regression are also obtained.

This work examined minimax linear estimation in multiple linear regression. The application of minimax estimation to regression led to the development of ridge regression estimators with stochastic ridge parameters. These estimators were seen to be invariant under linear transformation; a property which has not been established for other ridge estimators. These minimax-motivated estimators were examined in several simulation studies. In particular, flaws in other simulation studies of ridge estimators were depicted. Consequently, an improved simulation procedure was used. It was observed from these studies that, contrary to published statements, a ridge estimator can be considerably superior to the ordinary least squares estimator, especially when high pairwise correlations exist among the regression variables. Robustness considerations were used to suggest a requirement that a 'good' generalized ridge regression estimator should satisfy. (Author)

This work examined minimax linear estimation in multiple linear regression. The application of minimax estimation to regression led to the development of ridge regression estimators with stochastic ridge parameters. These estimators were seen to be invariant under linear transformation; a property which has not been established for other ridge estimators. These minimax-motivated estimators were examined in several simulation studies. In particular, flaws in other simulation studies of ridge estimators were depicted. Consequently, an improved simulation procedure was used. It was observed from these studies that, contrary to published statements, a ridge estimator can be considerably superior to the ordinary least squares estimator, especially when high pairwise correlations exist among the regression variables. Robustness considerations were used to suggest a requirement that a 'good' generalized ridge regression estimator should satisfy. (Author)

This paper discusses the effect of hubness in zero-shot learning, when ridge regression is used to find a mapping between the example space to the label space. Contrary to the existing approach, which attempts to find a mapping from the example space to the label space, we show that mapping labels into the example space is desirable to suppress the emergence of hubs in the subsequent nearest neighbor search step. Assuming a simple data model, we prove that the proposed approach indeed reduces hubness. This was verified empirically on the tasks of bilingual lexicon extraction and image labeling: hubness was reduced with both of these tasks and the accuracy was improved accordingly.

We study asymptotic minimax problems for estimating a $d$-dimensional regression parameter over spheres of growing dimension ($d\to \infty$). Assuming that the data follows a linear model with Gaussian predictors and errors, we show that ridge regression is asymptotically minimax and derive new closed form expressions for its asymptotic risk under squared-error loss. The asymptotic risk of ridge regression is closely related to the Stieltjes transform of the Mar\v{c}enko-Pastur distribution and the spectral distribution of the predictors from the linear model. Adaptive ridge estimators are also proposed (which adapt to the unknown radius of the sphere) and connections with equivariant estimation are highlighted. Our results are mostly relevant for asymptotic settings where the number of observations, $n$, is proportional to the number of predictors, that is, $d/n\to\rho\in(0,\infty)$.

We consider the most common variants of linear regression, including Ridge, Lasso and Support-vector regression, in a setting where the learner is allowed to observe only a fixed number of attributes of each example at training time. We present simple and efficient algorithms for these problems: for Lasso and Ridge regression they need the same total number of attributes (up to constants) as do full-information algorithms, for reaching a certain accuracy. For Support-vector regression, we require exponentially less attributes compared to the state of the art. By that, we resolve an open problem recently posed by Cesa-Bianchi et al. (2010). Experiments show the theoretical bounds to be justified by superior performance compared to the state of the art.

Unlike the ordinary least-squares (OLS) estimator for the linear model, a ridge regression linear model provides coefficient estimates via shrinkage, usually with improved mean-square and prediction error. This is true especially when the observed design matrix is ill-conditioned or singular, either as a result of highly-correlated covariates or the number of covariates exceeding the sample size. This paper introduces novel and fast marginal maximum likelihood (MML) algorithms for estimating the shrinkage parameter(s) for the Bayesian ridge and power ridge regression models, and an automatic plug-in MML estimator for the Bayesian generalized ridge regression model. With the aid of the singular value decomposition of the observed covariate design matrix, these MML estimation methods are quite fast even for data sets where either the sample size (n) or the number of covariates (p) is very large, and even when p>n. On several real data sets varying widely in terms of n and p, the computation times of the MML estimation methods for the three ridge models, respectively, are compared with the times of other methods for estimating the shrinkage parameter in ridge, LASSO and Elastic Net (EN) models, with the other methods based on minimizing prediction error according to cross-validation or information criteria. Also, the ridge, LASSO, and EN models, and their associated estimation methods, are compared in terms of prediction accuracy. Furthermore, a simulation study compares the ridge models under MML estimation, against the LASSO and EN models, in terms of their ability to differentiate between truly-significant covariates (i.e., with non-zero slope coefficients) and truly-insignificant covariates (with zero coefficients).

We propose new model selection criteria based on generalized ridge estimators dominating the maximum likelihood estimator under the squared risk and the Kullback-Leibler risk in multivariate linear regression. Our model selection criteria have the following favorite properties: consistency, unbiasedness, uniformly minimum variance. Consistency is proven under an asymptotic structure $\frac{p}{n}\to c$ where $n$ is the sample size and $p$ is the parameter dimension of the response variables. In particular, our proposed class of estimators dominates the maximum likelihood estimator under the squared risk even when the model does not include the true model. Experimental results show that the risks of our model selection criteria are smaller than the ones based on the maximum likelihood estimator and that our proposed criteria specify the true model under some conditions.

The introduction of the Global Positioning System (GPS) into the National Airspace System (NAS) necessitates the development of Receiver Autonomous Integrity Monitoring (RAIM) techniques. In order to guarantee a certain level of integrity, a thorough understanding of modern estimation techniques applied to navigational problems is required. The extended Kalman filter (EKF) is derived and analyzed under poor geometry conditions. It was found that the performance of the EKF is difficult to predict, since the EKF is designed for a Gaussian environment. A novel approach is implemented which incorporates ridge regression to explain the behavior of an EKF in the presence of dynamics under poor geometry conditions. The basic principles of ridge regression theory are presented, followed by the derivation of a linearized recursive ridge estimator. Computer simulations are performed to confirm the underlying theory and to provide a comparative analysis of the EKF and the recursive ridge estimator.

视频主要在EViews 12中演示弹性净估计的新功能，包括时间序列交叉验证、可变权重和估计后诊断。 完整的视频教程，请登录 科学软件网 www.sciencesoftware.com.cn 观看。

视频主要在EViews 12中演示弹性净估计的新功能，包括时间序列交叉验证、可变权重和估计后诊断。 完整的视频教程，请登录 科学软件网 www.sciencesoftware.com.cn 观看。

This article looked at non-experimental data via an ordinary least squares (OLS) model and compared its results to ridge regression models in terms of cross-validation predictor weighting precision when using fixed and random predictor cases and small and large p/n ratio models. A majority of the time with two random predictor cases, ridge regression accuracy was superior to OLS in estimating beta weights. Thus, ridge regression was very useful under this condition. However, when the fixed predictor case was reviewed, OLS was much more precise at estimating predictor weights than the ridge techniques regardless of the p/n ratio. In determining the cross validation accuracy of the ridge estimated weights in respect to the OLS estimated weights, ridge models were superior for improving the accuracy of model prediction. An appendix is included. (Contains 2 tables.)

We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that the risk of this ordinary least squares method is within a constant factor (namely 4) of the risk of ridge regression.

We propose a specialized string kernel for small bio-molecules, peptides and pseudo-sequences of binding interfaces. The kernel incorporates physico-chemical properties of amino acids and elegantly generalize eight kernels, such as the Oligo, the Weighted Degree, the Blended Spectrum, and the Radial Basis Function. We provide a low complexity dynamic programming algorithm for the exact computation of the kernel and a linear time algorithm for it's approximation. Combined with kernel ridge regression and SupCK, a novel binding pocket kernel, the proposed kernel yields biologically relevant and good prediction accuracy on the PepX database. For the first time, a machine learning predictor is capable of accurately predicting the binding affinity of any peptide to any protein. The method was also applied to both single-target and pan-specific Major Histocompatibility Complex class II benchmark datasets and three Quantitative Structure Affinity Model benchmark datasets. On all benchmarks, our method significantly (p-value < 0.057) outperforms the current state-of-the-art methods at predicting peptide-protein binding affinities. The proposed approach is flexible and can be applied to predict any quantitative biological activity. The method should be of value to a large segment of the research community with the potential to accelerate peptide-based drug and vaccine development.

Let y=A\beta+\epsilon, where y is an N\times1 vector of observations, \beta is a p\times1 vector of unknown regression coefficients, A is an N\times p design matrix and \epsilon is a spherically symmetric error term with unknown scale parameter \sigma. We consider estimation of \beta under general quadratic loss functions, and, in particular, extend the work of Strawderman [J. Amer. Statist. Assoc. 73 (1978) 623-627] and Casella [Ann. Statist. 8 (1980) 1036-1056, J. Amer. Statist. Assoc. 80 (1985) 753-758] by finding adaptive minimax estimators (which are, under the normality assumption, also generalized Bayes) of \beta, which have greater numerical stability (i.e., smaller condition number) than the usual least squares estimator. In particular, we give a subclass of such estimators which, surprisingly, has a very simple form. We also show that under certain conditions the generalized Bayes minimax estimators in the normal case are also generalized Bayes and minimax in the general case of spherically symmetric errors.

General predictive models do not provide a measure of confidence in predictions without Bayesian assumptions. A way to circumvent potential restrictions is to use conformal methods for constructing non-parametric confidence regions, that offer guarantees regarding validity. In this paper we provide a detailed description of a computationally efficient conformal procedure for Kernel Ridge Regression (KRR), and conduct a comparative numerical study to see how well conformal regions perform against the Bayesian confidence sets. The results suggest that conformalized KRR can yield predictive confidence regions with specified coverage rate, which is essential in constructing anomaly detection systems based on predictive models.

Ridge Dataset

Ridge Dataset

This work examined minimax linear estimation in multiple linear regression. The application of minimax estimation to regression led to the development of ridge regression estimators with stochastic ridge parameters. These estimators were seen to be invariant under linear transformation; a property which has not been established for other ridge estimators. These minimax-motivated estimators were examined in several simulation studies. In particular, flaws in other simulation studies of ridge estimators were depicted. Consequently, an improved simulation procedure was used. It was observed from these studies that, contrary to published statements, a ridge estimator can be considerably superior to the ordinary least squares estimator, especially when high pairwise correlations exist among the regression variables. Robustness considerations were used to suggest a requirement that a 'good' generalized ridge regression estimator should satisfy. (Author)

This work examined minimax linear estimation in multiple linear regression. The application of minimax estimation to regression led to the development of ridge regression estimators with stochastic ridge parameters. These estimators were seen to be invariant under linear transformation; a property which has not been established for other ridge estimators. These minimax-motivated estimators were examined in several simulation studies. In particular, flaws in other simulation studies of ridge estimators were depicted. Consequently, an improved simulation procedure was used. It was observed from these studies that, contrary to published statements, a ridge estimator can be considerably superior to the ordinary least squares estimator, especially when high pairwise correlations exist among the regression variables. Robustness considerations were used to suggest a requirement that a 'good' generalized ridge regression estimator should satisfy. (Author)

Anomalies persist in the foundations of ridge regression as set forth in Hoerl and Kennard (1970) and subsequently. Conventional ridge estimators and their properties do not follow on constraining lengths of solution vectors using LaGrange's method, as claimed. Estimators so constrained have singular distributions; the proposed solutions are not necessarily minimizing; and heretofore undiscovered bounds are exhibited for the ridge parameter. None of the considerable literature on estimation, prediction, cross--validation, choice of ridge parameter, and related issues, collectively known as ridge regression, is consistent with constrained optimization, nor with corresponding inequality constraints. The problem is traced to a misapplication of LaGrange's principle, failure to recognize the singularity of distributions, and misplaced links between constraints and the ridge parameter. Other principles, based on condition numbers, are seen to validate both conventional ridge and surrogate ridge regression to be defined. Numerical studies illustrate that ridge analysis often exhibits some of the same pathologies it is intended to redress.

This article is from Bioinformatics , volume 30 . Abstract Motivation: Ancestral character state reconstruction describes a set of techniques for estimating phenotypic or genetic features of species or related individuals that are the predecessors of those present today. Such reconstructions can reach into the distant past and can provide insights into the history of a population or a set of species when fossil data are not available, or they can be used to test evolutionary hypotheses, e.g. on the co-evolution of traits. Typical methods for ancestral character state reconstruction of continuous characters consider the phylogeny of the underlying data and estimate the ancestral process along the branches of the tree. They usually assume a Brownian motion model of character evolution or extensions thereof, requiring specific assumptions on the rate of phenotypic evolution.Results: We suggest using ridge regression to infer rates for each branch of the tree and the ancestral values at each inner node. We performed extensive simulations to evaluate the performance of this method and have shown that the accuracy of its reconstructed ancestral values is competitive to reconstructions using other state-of-the-art software. Using a hierarchical clustering of gene mutation profiles from an ovarian cancer dataset, we demonstrate the use of the method as a feature selection tool.Availability and implementation: The algorithm described here is implemented in C++ as a stand-alone program, and the source code is freely available at http://algbio.cs.uni-duesseldorf.de/software/RidgeRace.tar.gz.Contact:mchardy@hhu .deSupplementary information:Supplementary data are available at Bioinformatics online.

We propose LOCO, an algorithm for large-scale ridge regression which distributes the features across workers on a cluster. Important dependencies between variables are preserved using structured random projections which are cheap to compute and must only be communicated once. We show that LOCO obtains a solution which is close to the exact ridge regression solution in the fixed design setting. We verify this experimentally in a simulation study as well as an application to climate prediction. Furthermore, we show that LOCO achieves significant speedups compared with a state-of-the-art distributed algorithm on a large-scale regression problem.

Multivariate regression is a widespread computational technique that may give meaningless results if the explanatory variables are too numerous or highly collinear. Tikhonov regularization, or ridge regression, is a popular approach to address this issue. We reveal here a formal analogy between ridge regression and statistical mechanics, where the objective function is comparable to a free energy and the ridge parameter plays the role of temperature. This analogy suggests two new criteria to select a suitable ridge parameter: the specific-heat (Cv) and the maximum penalty (MP) fits. We apply these methods to the calibration of the force constant in elastic network models (ENM). This key parameter determines the amplitude of the predicted atomic fluctuations, and is commonly obtained by fitting crystallographic B-factors. However, rigid-body motions are often partially neglected in such fits, even though their importance has been repeatedly stressed. Considering the full set of rigid-body and internal degrees of freedom bears significant risks of overfitting, due to strong correlations between explanatory variables, and requires thus careful regularization. Using simulated data, we show that ridge regression with the Cv or MP criterion markedly reduces the error of the estimated force constant, its across-protein variation, and the number of proteins with unphysical values of the fit parameters, in comparison with popular regularization schemes such as generalized cross-validation. When applied to protein crystals, the new methods provide a more robust calibration of ENM force constants, even though rigid-body motions account on average for more than 80% of the amplitude of B-factors. While MP emerges as the optimal choice for fitting crystallographic B-factors, the Cv fit is more robust to the nature of the data, and is thus an interesting candidate for other applications.

We propose statistical inferential procedures for panel data models with interactive fixed effects in a kernel ridge regression framework.Compared with traditional sieve methods, our method is automatic in the sense that it does not require the choice of basis functions and truncation parameters.Model complexity is controlled by a continuous regularization parameter which can be automatically selected by generalized cross validation. Based on empirical processes theory and functional analysis tools, we derive joint asymptotic distributions for the estimators in the heterogeneous setting. These joint asymptotic results are then used to construct confidence intervals for the regression means and prediction intervals for the future observations, both being the first provably valid intervals in literature. Marginal asymptotic normality of the functional estimators in homogeneous setting is also obtained. Simulation and real data analysis demonstrate the advantages of our method.

We propose statistical inferential procedures for panel data models with interactive fixed effects in a kernel ridge regression framework.Compared with traditional sieve methods, our method is automatic in the sense that it does not require the choice of basis functions and truncation parameters.Model complexity is controlled by a continuous regularization parameter which can be automatically selected by generalized cross validation. Based on empirical processes theory and functional analysis tools, we derive joint asymptotic distributions for the estimators in the heterogeneous setting. These joint asymptotic results are then used to construct confidence intervals for the regression means and prediction intervals for the future observations, both being the first provably valid intervals in literature. Marginal asymptotic normality of the functional estimators in homogeneous setting is also obtained. Simulation and real data analysis demonstrate the advantages of our method.

We propose statistical inferential procedures for panel data models with interactive fixed effects in a kernel ridge regression framework.Compared with traditional sieve methods, our method is automatic in the sense that it does not require the choice of basis functions and truncation parameters.Model complexity is controlled by a continuous regularization parameter which can be automatically selected by generalized cross validation. Based on empirical processes theory and functional analysis tools, we derive joint asymptotic distributions for the estimators in the heterogeneous setting. These joint asymptotic results are then used to construct confidence intervals for the regression means and prediction intervals for the future observations, both being the first provably valid intervals in literature. Marginal asymptotic normality of the functional estimators in homogeneous setting is also obtained. Simulation and real data analysis demonstrate the advantages of our method.

This work examined minimax linear estimation in multiple linear regression. The application of minimax estimation to regression led to the development of ridge regression estimators with stochastic ridge parameters. These estimators were seen to be invariant under linear transformation; a property which has not been established for other ridge estimators. These minimax-motivated estimators were examined in several simulation studies. In particular, flaws in other simulation studies of ridge estimators were depicted. Consequently, an improved simulation procedure was used. It was observed from these studies that, contrary to published statements, a ridge estimator can be considerably superior to the ordinary least squares estimator, especially when high pairwise correlations exist among the regression variables. Robustness considerations were used to suggest a requirement that a 'good' generalized ridge regression estimator should satisfy. (Author)

The technique of ridge regression has become a popular tool for data analysts faced with a high degree of multicollinearity in their data. By using a ridge estimator, it was hoped that one could both stabilize the estimates (lower the condition number of the design matrix) and improve upon the squared error loss of the least squares estimator. Recently classes of ridge regression estimators have been developed which dominate the usual estimator in risk, and hence are minimax. This paper derives conditions that are necessary and sufficient for minimaxity of a large class of ridge regression estimators. The conditions derived here are very similar to those derived for minimaxity of some Stein-type estimators.

This work examined minimax linear estimation in multiple linear regression. The application of minimax estimation to regression led to the development of ridge regression estimators with stochastic ridge parameters. These estimators were seen to be invariant under linear transformation; a property which has not been established for other ridge estimators. These minimax-motivated estimators were examined in several simulation studies. In particular, flaws in other simulation studies of ridge estimators were depicted. Consequently, an improved simulation procedure was used. It was observed from these studies that, contrary to published statements, a ridge estimator can be considerably superior to the ordinary least squares estimator, especially when high pairwise correlations exist among the regression variables. Robustness considerations were used to suggest a requirement that a 'good' generalized ridge regression estimator should satisfy. (Author)

Current navigation requirements depend on a geometric dilution of precision (GDOP) criterion. As long as the GDOP stays below a specific value, navigation requirements are met. The GDOP will exceed the specified value when the measurement geometry becomes too collinear. A new signal processing technique, called Ridge Regression Processing, can reduce the effects of nearly collinear measurement geometry; thereby reducing the inflation of the measurement errors. It is shown that the Ridge signal processor gives a consistently better mean squared error (MSE) in position than the Ordinary Least Mean Squares (OLS) estimator. The applicability of this technique is currently being investigated to improve the following areas: receiver autonomous integrity monitoring (RAIM), coverage requirements, availability requirements, and precision approaches.

When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem 2 refines the Puffer preconditioner to make the Lasso select the same model as removing the elements of the OLS solution with the largest p-values. Using a generalized Puffer preconditioner, Theorem 3 relates ridge regression to the preconditioned Lasso; this result is for the high dimensional setting, p > n. Where the standard Lasso is akin to forward selection [Efron et al., 2004], Theorems 1, 2, and 3 suggest that the preconditioned Lasso is more akin to backward elimination. These results hold for sparse penalties beyond l1; for a broad class of sparse and non-convex techniques (e.g. SCAD and MC+), the results hold for all local minima.

We propose a new two stage algorithm LING for large scale regression problems. LING has the same risk as the well known Ridge Regression under the fixed design setting and can be computed much faster. Our experiments have shown that LING performs well in terms of both prediction accuracy and computational efficiency compared with other large scale regression algorithms like Gradient Descent, Stochastic Gradient Descent and Principal Component Regression on both simulated and real datasets.

We develop a novel preconditioning method for ridge regression, based on recent linear sketching methods. By equipping Stochastic Variance Reduced Gradient (SVRG) with this preconditioning process, we obtain a significant speed-up relative to fast stochastic methods such as SVRG, SDCA and SAG.

Regularization is an essential element of virtually all kernel methods for nonparametric regression problems. A critical factor in the effectiveness of a given kernel method is the type of regularization that is employed. This article compares and contrasts members from a general class of regularization techniques, which notably includes ridge regression and principal component regression. We derive an explicit finite-sample risk bound for regularization-based estimators that simultaneously accounts for (i) the structure of the ambient function space, (ii) the regularity of the true regression function, and (iii) the adaptability (or qualification) of the regularization. A simple consequence of this upper bound is that the risk of the regularization-based estimators matches the minimax rate in a variety of settings. The general bound also illustrates how some regularization techniques are more adaptable than others to favorable regularity properties that the true regression function may possess. This, in particular, demonstrates a striking difference between kernel ridge regression and kernel principal component regression. Our theoretical results are supported by numerical experiments.

This paper characterizes the conditional distribution properties of the finite sample ridge regression estimator and uses that result to evaluate total regression and generalization errors that incorporate the inaccuracies committed at the time of parameter estimation. The paper provides explicit formulas for those errors. Unlike other classical references in this setup, our results take place in a fully singular setup that does not assume the existence of a solution for the non-regularized regression problem. In exchange, we invoke a conditional homoscedasticity hypothesis on the regularized regression residuals that is crucial in our developments.

The technique of linear regression has been applied as a tool for predicting the cost of an item based on its most important characteristics. Often these characteristics (variables) tend to be highly intercorrelated (the data are said to exhibit multicollinearity) causing least squares estimates of the regression coefficients to be unstable and possibly leading to erroneous predictions. Ridge regression, a possible remedy for the problems caused by multicollinearity proposed by Hoerl and Kennard, is a biased estimation technique which reduces the variance of estimators and provides more precision (as measured by mean square error of the coefficients) than ordinary least squares (OLS) estimators. A comparison was made between these techniques to determine when ridge regression provides better cost equation coefficient estimates than OLS as a function of the degree of multicollinearity in the data, the number of predictor variables in the model, the degree of model fit (R2), and the amount of bias (k) of the estimate. A regression analysis of both sets showed that the degree of multicollinearity and amount of bias interact in explaining the major part of the improvement (degradation) in the mean square coefficient error.

Regression-based adjusted plus-minus statistics were developed in basketball and have recently come to hockey. The purpose of these statistics is to provide an estimate of each player's contribution to his team, independent of the strength of his teammates, the strength of his opponents, and other variables that are out of his control. One of the main downsides of the ordinary least squares regression models is that the estimates have large error bounds. Since certain pairs of teammates play together frequently, collinearity is present in the data and is one reason for the large errors. In hockey, the relative lack of scoring compared to basketball is another reason. To deal with these issues, we use ridge regression, a method that is commonly used in lieu of ordinary least squares regression when collinearity is present in the data. We also create models that use not only goals, but also shots, Fenwick rating (shots plus missed shots), and Corsi rating (shots, missed shots, and blocked shots). One benefit of using these statistics is that there are roughly ten times as many shots as goals, so there is much more data when using these statistics and the resulting estimates have smaller error bounds. The results of our ridge regression models are estimates of the offensive and defensive contributions of forwards and defensemen during even strength, power play, and short handed situations, in terms of goals per 60 minutes. The estimates are independent of strength of teammates, strength of opponents, and the zone in which a player's shift begins.

We provide a unified analysis of the predictive risk of ridge regression and regularized discriminant analysis in a dense random effects model. We work in a high-dimensional asymptotic regime where $p, n \to \infty$ and $p/n \to \gamma \in (0, \, \infty)$, and allow for arbitrary covariance among the features. For both methods, we provide an explicit and efficiently computable expression for the limiting predictive risk, which depends only on the spectrum of the feature-covariance matrix, the signal strength, and the aspect ratio $\gamma$. Especially in the case of regularized discriminant analysis, we find that predictive accuracy has a nuanced dependence on the eigenvalue distribution of the covariance matrix, suggesting that analyses based on the operator norm of the covariance matrix may not be sharp. Our results also uncover several qualitative insights about both methods: for example, with ridge regression, there is an exact inverse relation between the limiting predictive risk and the limiting estimation risk given a fixed signal strength. Our analysis builds on recent advances in random matrix theory.

Consider the linear regression model Y = X Theta + epsilon. Recently, a class of estimators, variously known as ridge estimators, has been proposed as an alternative to the least squares estimators in the case of collinearity, that is, when the design matrix X'X is nearly singular. The ridge estimator is given by Theta-cap = (1/(X'X + KI)) X'Y, where K is a constant to be determined. An optimal choice of the value of K is not known. This paper examines the risk (mean squared error) of the ridge estimator under the constraint Theta'Theta r or = c and determines optimal values of K for which the risk is smaller than the risk of the least squares estimators where c is a constant. (Author)