R code for Gradient Descent

Assume that we have one dimensional data and y=1.2*(x-2)^2+3.2. This implies that y has a closed form solution, known apriori. Then it is straight forward to obtain the first derivative and perform a gradient descent. Here is the R code for the above.

Now suppose, we still have one dimensional data but the functional form of y is unknown. How would we do Gradient Descent in that case? Below is the R code to do just that.



Now here are some variants of these to experiment with:
a. Replace the squared loss with a differentiable loss function of your choice and observe the impact on your favorite data set.
b. How does the parameters of the algorithm (alpha, # of iterations, choice of starting point) affect the convergence time of the algorithm?
c. How will you modify this code to implement the stochastic gradient descent algorithm?
d. Suppose we added a bias term to your hypothesis and asked you to repeat the experiments. What do you observe -- is the bias term beneficial?
e. Suppose we changed the hypothesis to be nonlinear for e.g. h(x) = w^2 x+ wx + b. Is the solution you find any better?
f. How will you modify the code above to implement Newton's algorithm (Clue: You need to use Taylor expansion for representing the function with higher order terms).

If you can report the above results on your favorite data set, I'd like to hear from you!

Also, feel free to ask questions in the comments if you are wondering about something.

Contour Plots of Loss Functions in R

In machine learning, loss functions are used to estimate how well learning algorithms perform. It is often written as loss = L(y, y_hat) where y is the true label and y_hat is predicted. Commonly used loss functions include Squared, Absolute or Laplace, Huber, Hinge, Logistic and others.

Using the Iris data set from UCIrvine,  I demonstrate how contour plots of loss functions can be obtained using R.

Background Information: The Iris data after being downloaded, was pre-processed in the following manner: Only two classes (100 examples) were selected to ensure the problem remained that of binary classification. Furthermore, two attributes were selected to enable visualizations via contour plots. The glmnet package was used to build a lasso model as shown below:



The filled contour plot generated from this is shown below:

Add in your favorite loss function -- Huber or Hinge and see some nice contour plots with the Iris data!

Story Telling at Country Parkway Elementary School

Reading a story entitled "A Little World of Mud" from Ruskin Bond's Children's Omnibus -- showcasing the work of a great Indian author of British descent -- on the occasion of International Night at my son's school. Here's the first part of the recording for my friends who missed it.Video Courtesy: Agneya Dutta Pooleery

Tindora in poppy seed sauce

When you live in a foreign land, you try and learn to adapt to the environment. Darwinism, I think. So if you live in a place where pointed gourd is not a native, you make do with close relatives of it. I like to think that the ivy gourd is a relative of the pointed gourd. Entirely "my opinion" and may/may not be true. I did not go to the length of checking facts. Well, at least they are both gourds -- so in some sense that may be true.

People from my native place drool over pointed gourd (also known as parval or potol) in poppy seed sauce. Why not try the adaption, ivy gourd (also known as tindora) with the same sauce? Simply logic -- lead to a really yummy preparation.

Simply slit the tindora in half along the longer side, sprinkle turmeric powder and salt and keep aside for about five minutes. Grid poppy seeds and make a thick paste using as little water as possible. Heat oil in a pan, add green chillies (if you like some heat) and throw in the tindora. Stir fry until slightly (but not overly) browned. Add in the poppy seed mix and let it boil until the tindora cooks and is well done. The poppy seed paste should dry up making a thick coating on the tindora.

And there you go folks -- tindora in poppy seed sauce to be had with roti, naan or rice! Enjoy!

SVM Loss Functions, Gradients and Hessians

Two popular loss functions used in the primal SVM formulation are: (a) Squared Loss and (b) Hinge Loss. The following note(s) show how the first and second derivatives of these losses are computed based on the assumptions of the model - a.k.a. linear model (with or without bias). Clearly, these can be extended to the non-linear kernel, however, those are left as an exercise for the reader.

Squared Loss and its first and second derivatives.

Hinge Loss and its first and second derivatives

Baby arugula and spinach salad

Baby arugula and spinach salad tossed with olive oil, pear balsamic vinegar,
dried cranberries, chopped walnuts, and dried apricots. 

From the French patisserie

Baguette toasted with olive oil and pistachio macarons at tea time.
Heavenly, I promise.