Assignment 3 - Pen & Paper Exercises

Due Date: Nov 24 (13:15 -- in class)

This exercises are taken from:
Christopher M. Bishop: Pattern Recognition and Machine Learning; Springer, 2007 - Website

4.8 (10 points)

Using (4.57) and (4.58), derive the result (4.65) for the posterior class probability in the two-class generative model with Gaussian densities,



and verify the results (4.66) and (4.67) for the parameters w and w_0.

4.16 (10 points)

Consider a binary classification problem in which each observation x_n is known to belong to one of two classes, corresponding to t = 0 and t = 1,
and suppose that the procedure for collecting training data is imperfect, so that training points are sometimes mislabelled.
For every data point x_n, instead of having a value t for the class label, we have instead a value π_n representing the probability that t_n = 1.
Given a probabilistic model p(t =1|φ), write down the log likelihood function appropriate to such a data set.

5.1 (15 points)

Consider a two-layer network function of the form (5.7)

in which the hidden-unit nonlinear activation functions g(·) are given by logistic sigmoid functions of the form (5.191)

Showthat there exists an equivalent network, which computes exactly the same function,
but with hidden unit activation functions given by tanh(a) where the tanh function is defined by (5.59).

Hint: first find the relation between σ(a) and tanh(a), and then show that the parameters of the two networks differ by linear transformations.

5.7 (10 points)

Show the derivative of the error function (5.24)

with respect to the activation a_k for output units having a softmax activation function satisfies (5.18).

7.2 (10 points)

Show that, if the 1 on the right-hand side of the constraint (7.5)

is replaced by some arbitrary constant γ>0, the solution for the maximum margin hyperplane is unchanged.

7.3 (15 points)

Show that, irrespective of the dimensionality of the data space, a data set consisting of just two data points,
one from each class, is sufficient to determine the location of the maximum-margin hyperplane.

Total: 70 points

Procedure and Submission

Please submit a PDF-document with your answers to Moodle or print your solutions and hand them in in class. Use the following naming scheme for your submission: "matrikelnumber_A3.pdf".

Late submission

Late Submissions are NOT possible. Any assignment submitted late will receive zero points.

Academic Honesty