Spark-Shell Startup Errors

I have downloaded Spark binaries for Windows and tried running spark-shell batch file and ran into some issues. Following are the issues and the respective solutions:

Problem 1

ERROR Shell: Failed to locate the winutils binary in the hadoop binary path Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(
at org.apache.hadoop.util.Shell.getWinUtilsPath(
at org.apache.hadoop.util.Shell.(
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(
at org.apache.hadoop.hive.conf.HiveConf.(


Download the winutils.exe from and copy it to some directory say c:/hadoop/bin

Define the environmental variable HADOOP_HOME as C:/hadoop

Problem 2

ERROR SparkContext: Error initializing SparkContext.
java.lang.AssertionError: assertion failed: Expected hostname
at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.util.Utils$.checkHost(Utils.scala:931)
at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:31)


Set environmental variable SPARK_LOCAL_HOSTNAME as localhost

An excellent guide on Java Garbage Collection

Couple of days back I came across the Plumbr’s handbook for Java Garbage Collection. I went through it and found it very useful and easy to understand. Best think is that the book very nicely explains how to read and understand GC logs.

You can download this handbook here.

Enjoy Learning,


What is Monkey Patching?

I came across this buzz word while I was reading something on unit testing. The concept is not new but the term is new for me. Monkey Patching is a term mostly used in context of Python and unit testing. This means modifying the code in a dynamic programming language at run time without affecting the source code.

Please refer the following wiki page for details about Monkey Patch.

Monkey Patch

Do share and comment about similar buzz words you know of in Software Development.

Enjoy Learning,


Add/Update a Class File in a Jar

Updating a class file directly in a jar comes very handy when you are working on a remote server and you need to update some classes in your deployed jar file. With update feature of the jar command, we can directly update the selective classes in the jar rather than building and moving the jar file itself to the remote server.

Let us assume we have abc.jar and we need to update the file Foo.class in the jar. Foo.class is in the package

In order to update the jar we need to create a directory structure reflecting the package structure of the class which we need to replace and place the class there. In this case we need to place the class, Foo.class, in the directory com/tm/example. Also, let us keep the directory com is on the same level as that of abc.jar.

Now we can execute the following command to update the class file into the jar file.

        jar uf abc.jar com/tm/example/Foo.class

The above command works for both Linux and Windows.

Please note if we do not provide the directory structure and try to add the class like

        jar uf abc.jar Foo.class

this will add the Foo.class in the jar on the root level.

In case we need to add/update multiple files in jar, we can do it by separating the file name with spaces like:

        jar uf abc.jar com/tm/example/Foo.class com/rm/example/another/Bar.class

For more details visit this java tutorial.

Enjoy Learning,

AspectJ: Applying Advice to Methods Based on an Annotation

I am new to AspectJ (not new to AOP though) and have been recently doing some R&D on using AspectJ for intercepting methods annotated with an annotation.

What I Tried …
Use AspectJ to intercept all the public methods (belonging to classes of specified packages) based on an annotation applied.

What I Needed …
Aspect J Development Tool plugin for eclipse. [equipped my eclipse 3.7 with AJDT plugin]
Searched for AspectJ getting started tutorials available on the internet. [Visit this link  for a good AspectJ getting started tutorial]
Searched how to write a pointcut for intercepting a method based on annotation.

This is how I did it …

Created a simple annotation named MyAnnotation.

package com.technicalmusings.examples.aspectj.annotation;

public @interface MyAnnotation {}

Created three classes Hello1, Hello2 and Hello3 having public and private methods annotated with MyAnnotation. The intent was to apply a before advice to all the public methods (annotated with MyAnnotation) present in the package com.technicalmusings.examples and its sub-packages. Therefore these three classes have been kept in three different packages. Please refer to the class organization in the screenshot below:


Hello1 class has a method without annotation as well to ensure that the pointcuts are doing exactly what they are supposed to.

package com.technicalmusings.examples.hello1;
import com.technicalmusings.examples.aspectj.annotation.MyAnnotation;

public class Hello1 {

    public static void main(String[] args) {

    public static void methodOfHello1WithoutAnnotation() {
        System.out.println("Greetings from a method without annotation from Hello1 class!");

    public static void greetPubliclyFromHello1() {
        System.out.println("Greetings from public method of Hello1 class!");

package com.technicalmusings.examples.aspectj.hello2;
import com.technicalmusings.examples.aspectj.annotation.MyAnnotation;

public class Hello2 {

    public static void main(String[] args) {

    public static void greetPubliclyFromHello2() {
        System.out.println("Greetings from public method of Hello2 class!");

    private static void greetPrivatelyFromHello2() {
        System.out.println("Greetings from private method of Hello2 class!");

package com.technicalmusings.partofexamples;
import com.technicalmusings.examples.aspectj.annotation.MyAnnotation;

public class Hello3 {

    public static void main(String[] args) {

    public static void greetPubliclyFromHello3() {
        System.out.println("Greetings from public method of Hello3 class!");

    private static void greetPrivatelyFromHello3() {
        System.out.println("Greetings from private method of Hello3 class!");

Now the important part, the AspectJ stuff…

Created an aspect, HelloAspect, with one after advice and one before advice (just to differentiate between the two scenario)
a) The after advice is applied to all the methods annotated with MyAnnotation (irrespective of package)
b) The before advice is applied to all the mehods anotated with MyAnnotation belonging to classes of specified package or subpackages


package com.technicalmusings.examples;

public aspect HelloAspect {

    // Pointcut for all public methods
    pointcut publicPointcut(): execution(public * *(..));

    // Pointcut for all the methods with the specified annotation
    pointcut annotationPointcutAll()
        : execution(@com.technicalmusings.examples.aspectj.annotation.MyAnnotation * *(..));

    // Pointcut for all the methods with the specified annotation within specified package
    pointcut annotationPointcutPackage()
        : execution( @com.technicalmusings.examples.aspectj.annotation.MyAnnotation
         * com.technicalmusings.examples..*(..));

    after() : annotationPointcutAll() && publicPointcut() {
        System.out.println("annotationPointcutAll: Intercepted the after call of : " 
        + thisJoinPoint.getSignature());

    before() : annotationPointcutPackage() && publicPointcut() {
        System.out.println("annotationPointcutPackage: Intercepted the before call of : "
        + thisJoinPoint.getSignature());

By executing the classes Hello1, Hello2 and Hello3 individually, we can see how the advices are being applied. The after advice which does not have the package filter should include the method of Hello3 class whereas the other should not.

Below snapshot from ‘Cross References’ view of eclipse shows how the HelloAspect is advising the methods.


Enjoy Learning,

String Utility Classes in Java

Manipulating strings is a part and parcel of a programmer’s day to day life. There are many string related operations which we use quite often. Some of the string operations are provided by Java (classes like String, StringBuilder etc.) and some we devise ourselves. For example, as a good practice the first thing we do while working with string is to check it for null and emptiness as shown below:

if (myString != null || !"".equals(myString)) {
// work with myString

For the above and some other frequently used cases of string manipulation we can use the string utility classes already provided by the third party libraries. The string utility class available in apache commons project, org.apache.commons.lang3.StringUtils, is one of the widely used string utility classes. This class has many utility methods which take care of commonly used string operations and most of these are null safe.

For example we can use the isBlank, isNotBlank methods from the StringUtils class to validate the string for empty and null as shown below:

if(StringUtils.isNotBlank(myString)) {
//work with myString

The java documentation of the StringUtils.isBlank method says:

public static boolean isBlank(CharSequence cs)
Checks if a CharSequence is whitespace, empty ("") or null.

StringUtils.isBlank(null) = true
StringUtils.isBlank("") = true
StringUtils.isBlank(" ") = true
StringUtils.isBlank("bob") = false
StringUtils.isBlank(" bob ") = false

Based on requirements we can also consider using isEmpty and isNotEmpty.

You can check the API documentation of org.apache.commons.lang3.StringUtils and have a look on other utility methods it provides.

If you are already using Spring framework in your application, you can consider using the StringUtils class available in Spring Framework for the utility methods it provides. This class has been mainly used inside Spring Framework but as it is available, we can use this. Some of the methods I liked from this class are:

// Convert an array or a collection to a comma delimited string (say, for creating a csv file) and vice versa
public static java.lang.String arrayToCommaDelimitedString(java.lang.Object[] arr)
public static java.lang.String[] commaDelimitedListToStringArray(java.lang.String str)

public static java.lang.String collectionToCommaDelimitedString(java.util.Collection<?> coll)
public static java.util.Set<java.lang.String> commaDelimitedListToSet(java.lang.String str)

//Trim an array of strings
public static java.lang.String[] trimArrayElements(java.lang.String[] array)

For more information on Spring Framework’s StringUtil class, check it’s API documentation.

Enjoy Learning,

Inside Java: Concurrent modification when using collection and iterator

ConcurrentModificationException may be thrown by methods that have detected concurrent modification of an object when such modification is not permissible. This exception is encountered more often in case of collections when one thread is iterating over the collection and other tries to modify it.

This article explains how java detects concurrent modifications to a List with the help of an example. The comments provided along with the code are self explanatory so let us go through the example first:

package com.technicalmusings.examples.collection;

import java.util.Iterator;
import java.util.LinkedList;
import java.util.List;

* The class explains how concurrent modification is detected in List.

public class ConcurrentModificationTest {

public static void main(String[] args) {

// Instantiating the list sets the modCount field of list to 0
// modCount is a protected integer variable defined in the AbstractList class.

List<Object> myList = new LinkedList<Object>();

myList.add(new Object()); // modCount incremented to 1
myList.add(new Object()); // modCount incremented to 2
myList.add(new Object()); // modCount incremented to 3

// Getting an iterator sets the value of expectedModCount field
// equal to that of the modCount field of list.

Iterator<Object> itr1 = myList.iterator(); //itr1: expectedModCount = 3
Iterator<Object> itr2 = myList.iterator(); //itr2: expectedModCount = 3

// The first Iterator traverses the list once;

// The second iterator traverses and removes the first element;
// In the previous step Itr2 modifies the list and sets the
// expectedModCount field to 4. The modCount field of the list
// has also been now incremented to 4

// The first iterator again tries to traverse the list;

// java.util.ConcurrentModificationException is thrown at this stage
// because itr1 finds its expectedModCount (which is still 3) to be not
// matching with the modCount field of the list

The class shown above explains how java works behind the scenes to detect concurrent modification and throw the ConcurrentModificationException in case of a List. List has an integer field modCount, which is incremented each time a structural modification is done in the list. Structural modification includes methods like add, remove, clear etc. The Iterator on the other hand uses another integer field, expectedModCount, to take care of concurrent modifications in the list. Iterator expects both the variables modCount and expectedModCount to be the same. In case the list is modified by any other means the modCount and expectedModCount differs and the exception is thrown.