MACHINE LEARNING | MULTICLASS CLASSIFICATION

MACHINE LEARNING - DAY 9

MULTI-CLASS CLASSIFICATION: ONE-VS-ALL

For the basics, you can check the earlier articles.

Terms used in this article can be understood from:

DAY 6: CLASSIFICATION AND LOGISTIC REGRESSION

Continuing our learning in machine learning today we’ll learn about the multi-class classification in logistic regression also known as one vs all.

Till now we have discussed about the 2 classification possibilities or 2 outcomes i.e., 1 or 0. Now, let’s see what happens when there are more number of possibilities.

for eg.,

lWeather: sunny, rainy, pleasant, windy

The outcome or the categorical value can be: 0, 1, 2, 3

lHealth: ill, dizzy, well

  

   The outcome or the categorical value can be: 0, 1, 2

The numbering doesn’t matter. It can be 1,2,3,4 or 0,1,2,3. These are just values which categorizes the given data or output into different categories.

y ∈ {0,1,2…,n}

h_Θ⁽⁰⁾(x) =  P(y = 0 | x; Θ )

h_Θ⁽¹⁾(x) =  P(y = 1 | x; Θ )

.

.

.

h_Θ⁽ⁿ⁾(x) =  P(y = n | x; Θ )

prediction : max(h_Θ⁽ⁱ⁾(x))

           i

STEPS OF COMPUTATION:

1. Plot the data

2. Take the classes one by one and rest of the 2 classes will behave as a single class or category. The probability of the single class is calculated in this way.

   For eg,

CONCLUSION:

Train a logistic regression h_Θ(x) for each class to predict the probability that y = i.

To make a prediction on a new x, pick the class that maximizes h_Θ(x) and that will be the output.

That’s all for day 9. Today we learned about the multi-class classification and how to compute it.

In day 10, we will be learning about the issue known as Overfitting which originates due to over-training of the model. The solution for this issue is Regularization which we’ll also cover in the next article.

If you think this article helped you in learning something new or can help someone then do share this article among the peers.

Till then Happy Learning!!!

MACHINE LEARNING | LOGISTIC REGRESSION COST FUNCTION

MACHINE LEARNING - DAY 8

LOGISTIC REGRESSION COST FUNCTION AND ANALYSIS

For the basics, you can check the earlier articles.

Terms used in this article can be understood from:

DAY 2: Linear Regression

DAY 6: Classification and Logistic Regression

DAY 7: Decision Boundary for Logistic Regression

Continuing our learning in logistic regression today we’ll learn about the cost function in logistic regression and make it efficient to be able to find proper parameter values for the model.

COST FUNCTION:

Cost function as in linear regression is used to compute the parameter, Θ_i, values automatically which will give the best fit for a given model.

The graph of Θ and number of iterations should always be a convex curve to achieve a global minima.

Training set: {(x¹,y¹),(x²,y²),…..,(x^m,y^m)}  (Number of examples: m)

x = [x₀;x₁;x₂;….;x_n]        x₀ = 1, y ∈ {0,1}

The dimension of the x matrix is n+1 x 1.

h_Θ(x) = 1/(1+(e^(-Θ^TX))

How to choose parameter value for Θ_i :

In linear regression the cost function was

         n

J(Θ)=1/m ∑(1/2)( h_Θ(xⁱ) − yⁱ)²

        i=1

          n 

J(Θ) = 1/m ∑ cost( h_Θ(xⁱ) − yⁱ)

         i=1

cost( h_Θ(xⁱ) − yⁱ) = 1/2 ( h_Θ(xⁱ) − yⁱ)²

The only difference between the cost function of linear regression and logistic regression is that of the hypothesis. The hypothesis in logistic regression is:

h_Θ(x) = 1/(1+(e^(-Θ^TX))

This will give a non- convex graph or output curve which will lead to multiple local optima.

To prevent the non- convex output and get a convex output with a single optima i.e., global optima, we make some alterations in the cost function for logistic regression.

cost( h_Θ(x) − y) = { - log(h_Θ(x)), if y = 1} and { - log(1 - h_Θ(x)), if y = 0}

For simplification the cost function can be written in a single line as,

cost( h_Θ(x) − y) = - [y * log(h_Θ(x)) + (1 - y) * log(1 - h_Θ(x))]

for y = 0,

cost( h_Θ(x) − y) = - [0 * log(h_Θ(x)) + (1 - 0) * log(1 - h_Θ(x))]

cost( h_Θ(x) − y) = - [0 + (1) * log(1 - h_Θ(x))] ≈  - log(1 - h_Θ(x))

for y = 1,

cost( h_Θ(x) − y) = - [1 * log(h_Θ(x)) + (1 - 1) * log(1 - h_Θ(x))]

cost( h_Θ(x) − y) = - [log(h_Θ(x)) + 0] ≈  - log(h_Θ(x))

Hence, it gives the same equations as we have seen earlier.

GRADIENT DESCENT:

Gradient descent helps in iterating the equation until and unless a global minima is achieved.

Compute:

Notice, the gradient descent is also similar to the one used in the linear regression. The only difference here also is of the hypothesis used in linear regression and logistic regression which is:

h_Θ(x) = 1/(1+(e^(-Θ^TX))

That’s all for day 8. Today we learned about the cost function in logistic regression and how to make it efficient to attain a global minima for our parameters values to obtain the perfect fit for a model. We also learned the gradient descent for logistic regression.

In day 9, we will be learning about the Multi-Class Classification problem which includes more possible outcomes than 0 and 1.

If you think this article helped you in learning something new or can help someone then do share this article among others.

Till then Happy Learning!!!

  

MACHINE LEARNING | DECISION BOUNDARY

MACHINE LEARNING - DAY 7

DECISION BOUNDARY FOR LOGISTIC REGRESSION

For the basics, you can check the earlier articles.

Terms used in this article can be understood from:

DAY 6: Logistic Regression

DECISION BOUNDARY

Decision boundary means the shape of the curve dividing the data into 2 segments, one which has y = 1 and the other category with y = 0.

h_Θ(x) = g(Θ^TX) = P(y = 1 | x; Θ )

g(z) = 1/(1+(e^(-z)) = 1/(1+(e^(-Θ^TX))

Suppose, we want to predict y=1 then

                             

h_Θ(x) ≥ 0.5

and, for prediction of y=0,

h_Θ(x) < 0.5

Now, let’s see when are these values possible.

1. y=1 when,

g(z) ≥ 0.5

When z ≥ 0

h_Θ(x) = g(Θ^TX) ≥ 0.5

y = 1, when Θ^TX ≥ 0.

2. y = 0 when,

g(z) < 0.5

When z < 0

h_Θ(x) = g(Θ^TX) < 0.5

y = 0, when Θ^TX < 0.

Now, let’s discuss about the decision boundaries with the help of some examples.

l Example 1:

h_Θ(x) = g(Θ₀ + Θ₁x₁ + Θ₂x₂)

Let Θ₀ = -3, Θ₁ = 1, Θ₂ = 1

Θ = [-3;1;1]

Dimension of Θ matrix is 3X1.

Predict y = 1, if

-3 + x₁ + x₂ ≥ 0 ≈ g(z) > 0.5 ≈ z > 0.

x₁ + x₂ ≥ 3.

And for y = 0,

x₁ + x₂ < 3

NON - LINEAR DECISION BOUNDARIES

Sometimes, the data points are arranged in such a manner that the curve separating them takes a complex shape then a straight line.

l Example

Hypothesis: h_Θ(x) = g(Θ₀ + Θ₁x₁ + Θ₂x₂ + Θ₃x₁² + Θ₄x₂²)

Let Θ₀ = -1, Θ₁ = 0, Θ₂ = 0, Θ₃ =1, Θ₄ = 1 (for now we’ll see how to find the parameters automatically under upcoming lessons.)

Θ = [-1;0;0;1;1]

The dimension of the matrix is 5X1.

To predict:

y = 1 if,

-1 + x₁² + x₂²  ≥ 0 ≈ x₁² + x₂²  ≥ 1(equation of a circle with center at origin).

NOTE: Decision boundaries depends upon the parameters i.e., Θ values.

Decision boundaries can vary depending upon the hypothesis. It can get complex or it can also get simplified with the increase of the parameters and the variables.

Points to remember:

lg(z) ≥ 0.5 ≈ z ≥ 0

lz = 0, e⁰ = 1, g(z) = 1/2

lz → ∞, e^-∞ → 0 → g(z) = 1

lz → -∞, e^∞ → ∞ → g(z) = 0

That’s all for day 7. Today we learned about the decision boundaries in classification problems, especially in logistic regression.

In day 8, we will be learning about the cost function of logistic regression which will help us in figuring out the parameter i.e., Θ values automatically for the best fit and we will also learn about the concept of multi-class classification in logistic regression.

If you think this article helped you in learning something new or can help someone then do share this article among the peers.

Till then Happy Learning!!!