Hine Learning Least Squares Manual Before You

Minimization tends to set many coefficients to zero Why is this good Why dont we always do it, then Any problems 33 B.

If you wish to download it, please recommend it to your friends in any social system.

Hine Learning Least Squares Manual Before You

Read the TexPoint manual before you delete this box.: AA A AA A A A A A AAAAAA A AAAA A A A AA A A.

Learn a continuous function y ( x ) to predict the function value for a new input x.

Steps towards a solution Choose a form of the function y ( x, w ) with parameters w.

Optimize E ( w ) for w to find a good solution.

Derive the properties of this solution and think about its limitations.

B. Leibe.

This function specifies how a deviation from the target value should be weighted.

Popular choice: sum-of-squares error Definition Minimize the error: compute the derivative and set it to zero 6 B.

Since the error is a quadratic function of w, its derivative will be linear in w.

Same basic setup as in least-squares classification Only the values are now continuous.

B. Leibe Slide credit: Bernt Schiele.

Bishop, 2006 Massive overfitting.

Perfect fit to the training data with M 9, but poor representation of the original function.

Why is that After all, M 9 contains M 3 as a special case 12 B.

Goal is a good prediction of future data Our target function should fit well to the training data, but also generalize.

Measure generalization performance on independent test set.

Square root ensures E RMS is measured on the same scale (and in the same units) as the target variable t.

B. Leibe Overfitting Image source: C.M.

Bishop, 2006.

Coefficients: 16 B.

Leibe Image source: C.M.

Bishop, 2006 Slide credit: Bernt Schiele.

Workaround: Regularization Penalize large coefficient values Here weve simply added a quadratic regularizer, which is simple to optimize The resulting form of the problem is called Ridge Regression.

Note: w 0 is often omitted from the regularizer.) 17 B.

Leibe.

Leibe 18 Effect of regularization: Keeps the inverse well-conditioned.

How can we prevent overfitting systematically How can we avoid the need for validation on separate test data What does it mean to do regularization 21 B.

In the simplest case, we use linear basis functions: d ( x ) x d.

Lets take a look at some other possible basis functions.

Leibe Slide adapted from C.M.

Bishop, 2006 Data term Regularization term is called the regularization coefficient.

Leibe LassoRidge Regression Slide adapted from C.M.

Bishop, 2006 Image source: C.M.

Machine Learning Least Squares