MACHINE LEARNING | LOGISTIC REGRESSION COST FUNCTION

MACHINE LEARNING - DAY 8

LOGISTIC REGRESSION COST FUNCTION AND ANALYSIS

For the basics, you can check the earlier articles.

Terms used in this article can be understood from:

DAY 2: Linear Regression

DAY 6: Classification and Logistic Regression

DAY 7: Decision Boundary for Logistic Regression

Continuing our learning in logistic regression today we’ll learn about the cost function in logistic regression and make it efficient to be able to find proper parameter values for the model.

COST FUNCTION:

Cost function as in linear regression is used to compute the parameter, Θ_i, values automatically which will give the best fit for a given model.

The graph of Θ and number of iterations should always be a convex curve to achieve a global minima.

Training set: {(x¹,y¹),(x²,y²),…..,(x^m,y^m)}  (Number of examples: m)

x = [x₀;x₁;x₂;….;x_n]        x₀ = 1, y ∈ {0,1}

The dimension of the x matrix is n+1 x 1.

h_Θ(x) = 1/(1+(e^(-Θ^TX))

How to choose parameter value for Θ_i :

In linear regression the cost function was

         n

J(Θ)=1/m ∑(1/2)( h_Θ(xⁱ) − yⁱ)²

        i=1

          n 

J(Θ) = 1/m ∑ cost( h_Θ(xⁱ) − yⁱ)

         i=1

cost( h_Θ(xⁱ) − yⁱ) = 1/2 ( h_Θ(xⁱ) − yⁱ)²

The only difference between the cost function of linear regression and logistic regression is that of the hypothesis. The hypothesis in logistic regression is:

h_Θ(x) = 1/(1+(e^(-Θ^TX))

This will give a non- convex graph or output curve which will lead to multiple local optima.

To prevent the non- convex output and get a convex output with a single optima i.e., global optima, we make some alterations in the cost function for logistic regression.

cost( h_Θ(x) − y) = { - log(h_Θ(x)), if y = 1} and { - log(1 - h_Θ(x)), if y = 0}

For simplification the cost function can be written in a single line as,

cost( h_Θ(x) − y) = - [y * log(h_Θ(x)) + (1 - y) * log(1 - h_Θ(x))]

for y = 0,

cost( h_Θ(x) − y) = - [0 * log(h_Θ(x)) + (1 - 0) * log(1 - h_Θ(x))]

cost( h_Θ(x) − y) = - [0 + (1) * log(1 - h_Θ(x))] ≈  - log(1 - h_Θ(x))

for y = 1,

cost( h_Θ(x) − y) = - [1 * log(h_Θ(x)) + (1 - 1) * log(1 - h_Θ(x))]

cost( h_Θ(x) − y) = - [log(h_Θ(x)) + 0] ≈  - log(h_Θ(x))

Hence, it gives the same equations as we have seen earlier.

GRADIENT DESCENT:

Gradient descent helps in iterating the equation until and unless a global minima is achieved.

Compute:

Notice, the gradient descent is also similar to the one used in the linear regression. The only difference here also is of the hypothesis used in linear regression and logistic regression which is:

h_Θ(x) = 1/(1+(e^(-Θ^TX))

That’s all for day 8. Today we learned about the cost function in logistic regression and how to make it efficient to attain a global minima for our parameters values to obtain the perfect fit for a model. We also learned the gradient descent for logistic regression.

In day 9, we will be learning about the Multi-Class Classification problem which includes more possible outcomes than 0 and 1.

If you think this article helped you in learning something new or can help someone then do share this article among others.

Till then Happy Learning!!!

  

MACHINE LEARNING | DECISION BOUNDARY

MACHINE LEARNING - DAY 7

DECISION BOUNDARY FOR LOGISTIC REGRESSION

For the basics, you can check the earlier articles.

Terms used in this article can be understood from:

DAY 6: Logistic Regression

DECISION BOUNDARY

Decision boundary means the shape of the curve dividing the data into 2 segments, one which has y = 1 and the other category with y = 0.

h_Θ(x) = g(Θ^TX) = P(y = 1 | x; Θ )

g(z) = 1/(1+(e^(-z)) = 1/(1+(e^(-Θ^TX))

Suppose, we want to predict y=1 then

                             

h_Θ(x) ≥ 0.5

and, for prediction of y=0,

h_Θ(x) < 0.5

Now, let’s see when are these values possible.

1. y=1 when,

g(z) ≥ 0.5

When z ≥ 0

h_Θ(x) = g(Θ^TX) ≥ 0.5

y = 1, when Θ^TX ≥ 0.

2. y = 0 when,

g(z) < 0.5

When z < 0

h_Θ(x) = g(Θ^TX) < 0.5

y = 0, when Θ^TX < 0.

Now, let’s discuss about the decision boundaries with the help of some examples.

l Example 1:

h_Θ(x) = g(Θ₀ + Θ₁x₁ + Θ₂x₂)

Let Θ₀ = -3, Θ₁ = 1, Θ₂ = 1

Θ = [-3;1;1]

Dimension of Θ matrix is 3X1.

Predict y = 1, if

-3 + x₁ + x₂ ≥ 0 ≈ g(z) > 0.5 ≈ z > 0.

x₁ + x₂ ≥ 3.

And for y = 0,

x₁ + x₂ < 3

NON - LINEAR DECISION BOUNDARIES

Sometimes, the data points are arranged in such a manner that the curve separating them takes a complex shape then a straight line.

l Example

Hypothesis: h_Θ(x) = g(Θ₀ + Θ₁x₁ + Θ₂x₂ + Θ₃x₁² + Θ₄x₂²)

Let Θ₀ = -1, Θ₁ = 0, Θ₂ = 0, Θ₃ =1, Θ₄ = 1 (for now we’ll see how to find the parameters automatically under upcoming lessons.)

Θ = [-1;0;0;1;1]

The dimension of the matrix is 5X1.

To predict:

y = 1 if,

-1 + x₁² + x₂²  ≥ 0 ≈ x₁² + x₂²  ≥ 1(equation of a circle with center at origin).

NOTE: Decision boundaries depends upon the parameters i.e., Θ values.

Decision boundaries can vary depending upon the hypothesis. It can get complex or it can also get simplified with the increase of the parameters and the variables.

Points to remember:

lg(z) ≥ 0.5 ≈ z ≥ 0

lz = 0, e⁰ = 1, g(z) = 1/2

lz → ∞, e^-∞ → 0 → g(z) = 1

lz → -∞, e^∞ → ∞ → g(z) = 0

That’s all for day 7. Today we learned about the decision boundaries in classification problems, especially in logistic regression.

In day 8, we will be learning about the cost function of logistic regression which will help us in figuring out the parameter i.e., Θ values automatically for the best fit and we will also learn about the concept of multi-class classification in logistic regression.

If you think this article helped you in learning something new or can help someone then do share this article among the peers.

Till then Happy Learning!!!

MACHINE LEARNING | CLASSIFICATION AND LOGISTIC REGRESSION

MACHINE LEARNING - DAY 6

CLASSIFICATION AND LOGISTIC REGRESSION MODEL

Before moving to today's topic, if you haven't checked the earlier topics which are the basics then visit the following links:

DAY 1: Machine Learning and Types of Machine Learning Algorithms

DAY 2: Linear Regression

DAY 3: Multiple Linear Regression

DAY 4: Gradient Descent and Learning Rate

DAY 5: Normal Equations

CLASSIFICATION:

E-mail : spam/ not spam

Online transactions: Fraudulent(yes/no)

These two are examples of where classification algorithms are used.

y belongs to {0,1}

0: negative class(absence of something or some value eg. not spam mail).

1: positive class(presence of something or some value eg. spam email).

0 and 1 are classification levels or indicators, it doesn’t matters which class they belong to. No hard rules.

LOGISTIC REGRESSION MODEL:

A classification algorithm which helps in providing the probability of the occurrence of an event or the probability of non-occurrence of an event.

It gives the output in 1 or 0.

HYPOTHESIS:

In linear regression the hypothesis was:

h_Θ(x) = Θ^TX

In logistic regression the hypothesis is:

h_Θ(x) = g(Θ^TX)

where,

g(z) = 1/(1+(e^(-z))

z = Θ^TX

1/(1+(e^(-z)) is called the sigmoid function or the logistic function.

Therefore, the hypothesis for the logistic regression is:

h_Θ(x)  = 1/(1+(e^(-z))

INTERPRETATION OF HYPOTHESIS OUTPUT

h_Θ(x) = estimated probability that y = 1 on input x.

Eg. If x = [x₀  = [1

x₁]   tumorsize]

h_Θ(x) = 0.7, y = 1

It means that there is a chance of 70% that the patient has a malignant tumor.

It can be written in probability as

h_Θ(x) = P(y = 1 | x; Θ )

P(y = 0 | x; Θ) = 1 - P(y = 1 | x; Θ )

It is read like ‘probability that y = 1, on a given x, parameterized by Θ.

LINEAR REGRESSION VS LOGISTIC REGRESSION

For now, let’s discuss classification algorithm for 2 possible outcomes i.e., 0 and 1. We’ll see classification for multiple outcomes i.e., 0,1,2,3 etc. in the upcoming blogs.

Example for classification problem:

Parameters can be threshold classifier output h_Θ(x) at 0.5.

if h_Θ(x) >= 0.5, predict y=1,

if h_Θ(x) < 0.5, predict y=0.

In the above example the linear regression model fits nicely but what if there is a point a bit far towards positive x-axis as shown in the figure below,

the fit gets changed and is not good now. In the previous example, it was just a matter of luck or chance that regression model was able to fit the hypothesis.

NOTE:

1. In linear regression:

y = 0 or 1

but

h_Θ(x) can be greater than 1( h_Θ(x) > 1) or less than 0( h_Θ(x) < 0 ).

2. In logistic regression:

y = 0 or 1

and

h_Θ(x) is also between 0 and 1, i.e., 0 <= h_Θ(x) <= 1.

So, to conclude in classification problems, linear regression is not a good choice, only logistic regression should be preferred.

SUMMARY:

lWhat is Classification Algorithm?

lLogistic Regression

lHypothesis of Logistic Regression

lInterpretation of output in Logistic Regression

lLinear Regression vs Logistic Regression

That's all for DAY 6. In Day 7 we will be learning about the working of the logistic 
regression and multi-class linear regression.

If you like this blog or if you think it can help someone gain some knowledge in
machine learning then do share it with them. If you have any doubt mention them in
the comment section.

Till then Happy Learning!!!