MACHINE LEARNING | LOGISTIC REGRESSION COST FUNCTION

MACHINE LEARNING - DAY 8

LOGISTIC REGRESSION COST FUNCTION AND ANALYSIS

For the basics, you can check the earlier articles.

Terms used in this article can be understood from:

DAY 2: Linear Regression

DAY 6: Classification and Logistic Regression

DAY 7: Decision Boundary for Logistic Regression

Continuing our learning in logistic regression today we’ll learn about the cost function in logistic regression and make it efficient to be able to find proper parameter values for the model.

COST FUNCTION:

Cost function as in linear regression is used to compute the parameter, Θ_i, values automatically which will give the best fit for a given model.

The graph of Θ and number of iterations should always be a convex curve to achieve a global minima.

Training set: {(x¹,y¹),(x²,y²),…..,(x^m,y^m)}  (Number of examples: m)

x = [x₀;x₁;x₂;….;x_n]        x₀ = 1, y ∈ {0,1}

The dimension of the x matrix is n+1 x 1.

h_Θ(x) = 1/(1+(e^(-Θ^TX))

How to choose parameter value for Θ_i :

In linear regression the cost function was

         n

J(Θ)=1/m ∑(1/2)( h_Θ(xⁱ) − yⁱ)²

        i=1

          n 

J(Θ) = 1/m ∑ cost( h_Θ(xⁱ) − yⁱ)

         i=1

cost( h_Θ(xⁱ) − yⁱ) = 1/2 ( h_Θ(xⁱ) − yⁱ)²

The only difference between the cost function of linear regression and logistic regression is that of the hypothesis. The hypothesis in logistic regression is:

h_Θ(x) = 1/(1+(e^(-Θ^TX))

This will give a non- convex graph or output curve which will lead to multiple local optima.

To prevent the non- convex output and get a convex output with a single optima i.e., global optima, we make some alterations in the cost function for logistic regression.

cost( h_Θ(x) − y) = { - log(h_Θ(x)), if y = 1} and { - log(1 - h_Θ(x)), if y = 0}

For simplification the cost function can be written in a single line as,

cost( h_Θ(x) − y) = - [y * log(h_Θ(x)) + (1 - y) * log(1 - h_Θ(x))]

for y = 0,

cost( h_Θ(x) − y) = - [0 * log(h_Θ(x)) + (1 - 0) * log(1 - h_Θ(x))]

cost( h_Θ(x) − y) = - [0 + (1) * log(1 - h_Θ(x))] ≈  - log(1 - h_Θ(x))

for y = 1,

cost( h_Θ(x) − y) = - [1 * log(h_Θ(x)) + (1 - 1) * log(1 - h_Θ(x))]

cost( h_Θ(x) − y) = - [log(h_Θ(x)) + 0] ≈  - log(h_Θ(x))

Hence, it gives the same equations as we have seen earlier.

GRADIENT DESCENT:

Gradient descent helps in iterating the equation until and unless a global minima is achieved.

Compute:

Notice, the gradient descent is also similar to the one used in the linear regression. The only difference here also is of the hypothesis used in linear regression and logistic regression which is:

h_Θ(x) = 1/(1+(e^(-Θ^TX))

That’s all for day 8. Today we learned about the cost function in logistic regression and how to make it efficient to attain a global minima for our parameters values to obtain the perfect fit for a model. We also learned the gradient descent for logistic regression.

In day 9, we will be learning about the Multi-Class Classification problem which includes more possible outcomes than 0 and 1.

If you think this article helped you in learning something new or can help someone then do share this article among others.

Till then Happy Learning!!!