The bane of being a polyglot developer

In the recent months, there has been a bitter realization for me as I have given quite a few interviews as I was looking for a job. My resume was decorated with most of what I have been doing as part of my earlier job as well as what I did in my spare time. However where I failed were the areas that require mugging up the most. Consider the following questions:

• What is the difference between get and load methods in hibernate?
• What are the different active record validations?
• What are the different types of active resource nesting?
• What is different between event handling and event bubbling?

Answers to these are no far than a google search or documentation lookup away but in all these questions, I wore a blank look. The first one was a hibernate question and after searching for answers, I came to know that it was actually one of the most popular hibernate interview questions. The other two were rails questions, but since I have done very little professional ruby work, I could not explain the same. The last one was a jQuery killer, appearing after I had rated myself 5/10 in javascript and was again nothing, but technology buzzword.

Very few interviewers were interested in asking something challenging questions in scalability, design pattern or architectural issues. This really leaves me angry at recruiters only looking to fill up position involving technology X people and also at myself at allowing myself to stray. The real trouble that I feel is that I am maybe easily persuaded. I started with java, then moved over to j2ee and for a time felt that I had nailed it when I finally deployed a ejb 2.0 environment. However, then came spring and hibernate and I felt that they were the actual mecca and medina for me. However, later on I was sucked by the very vocal ruby community that said, 'wait a minute, you a java guy, we can make you 10X productive'. Duh!
Then came a string of languages, python, .net, scala, f# and whatever the titbits that I had left - from playing with arduinos at home to mainframes at work (and I still harbor desire to purchase a raspberry pi one day and do more cool hacks on it).
However, these plethora of languages and technologies have taken a trool on my confidence, especially when I come across people who only know or like a single technology and are comfortable in it.
But that's not for me as I thrive on change. For example I am writing this rant after 10 pm and today I've:

• studied redis in the morning
• tried to implement a scalable solution on redis to handel >10 million records
• did AdWords and facebook api integration into a rails4 app in office in the day
• will be writing/improving the draft chapters of the upcoming book that I am writing in robot framework.

I think Steve yegge captures some of my deficiencies clearly as I am an advance learner in a lot of technology that I have on my resume, and I guess that's the way it is going to be in the near future.

Book Review: Embedded Android

This is the review of the book, 'Embedded Android' by Karim Yaghmour, which is an essential reference for any developer doing customization of android platform. As I am an application and a device developer primarily when it comes to android or embedded device management respectively, I read this book as a novice attempting to glean some understanding of the android internals.





The first thing that struck me was the updated nature of the book wrt the android ecosystem and how some of the mystifying concepts of android were explained, like the comparison between gnu linux and its kernal and of dalvik with java and the algorithms employed by the OS for optimizing constrained resource devices. The next involved Karim taking me on a guided tour to the different parts of android that I did not imagined such as building and customization of the android SDK itself and custom roms for headless devices.

Make no mistake, this is not a book for android application development, or its native counterpart but explains you part-by-part about the different aspects involved in development of the android project.

Book contents

The book starts with the introduction of the android project's history and its differences with classical open source projects. It then dives deep into the internals of the android OS covering the architecture and comparing it with linux. It then covers dalvik and system services.
The next couple of chapters focus entirely on the android open source project and involves building of various components of the project. The build system of android is also discussed and compared with conventional makefile based systems. One considerable mention is the presence of build recipes and hacks that give more insight about what is being covered.
Hardware and popular systems are covered next and development on different boards and SOCs are covered in this chapter which is followed up by a thorough discussion of the filesystem, components as well as commonly used tools.
Finally the book discusses the android framework where different utilities, extension into support of newer hardware, components, services and parts of the android OS is covered.

The appendixes cover portions such as legacy user-space and adding support for new hardware as well as customizing the default lists of packages. The default init.rc files are also provided alongwith informative links to various websites that cover the latest in the topics that are covered in the book.

While I cannot honestly comment the usability of the book from an android modder/image customizer's point of view, in general I found the book to be of great use and armed with my preexisting knowledge of android development and linux, I was able to understand the topics very well.

Disclaimer: This book was provided to me by Oreilly as part of their blogger review program.

DevOps Dojo

and for those, who think devops is just a fancy term for maintenance; I got 2 words: Automation & Collaboration.
This concept features more than just integrating the different teams. However, on many job having devops as their Job Description this is not the case. Apopular tweet that is going rounds around twitter captures the essence of devops : 'It’s not always about starting small, it’s about tackling the hardest apps first'.

Atlassian has released a recent advertisement where it cleverly displays its in house products. It is hearning to see the extensive use of open source projects together with, understandably altassian products. The website is cleverly put together of two different dojos that cover the technical and cultural aspects of this integration.

A rant on design patterns

There are few things a java programmer is expected to demonstrate after a few years of working in trenches and design patterns is cardinal to it. It is not sufficient to implement a design pattern where it is begging to be implemented, to lessen the sagging code logic and improve readability, extensibility or both but also to insert it into every nook, corner and alley of code where it might or might not serve a purpose.
I recently did a code submission for a multinational company and was rejected due to this precise reason - my code was not extensible, they said.

However, what was bitterly amusing for me was the fact that I was shipped with a document that said KISS should be followed and emphasis would be over test driven development. The main point in this frustrating experience is how can one keep it simple and stupid... or minimized without using design patterns. Either I do not have an understanding of the DPs and there is any pattern that can be used to further minimize the code, something that I did in the first place. Merely creating methods and class hierarchy to conform to a pattern(when the implementation is merely 10 lines of code) in every case seems to be an overkill, even if the code is meant to be extensible.

I think as programmer, to create a solution, there are arguably ten ways of performing a task, each with its own advantage and caveat and what is important for a programmer who is reviewing it (for recruitment or as an exercise, while merging open source changes or even for a team member) has to get this basic fundamental correct.

MapReduce2.0: Next Generation Mapreduce using YARN

There has been a lot of long overdue changes in apache hadoop, the most popular framework to perform mapreduce over big data.
As the users of hadoop may recall, one of the vulnerability in mapreduce was the presence of a single node that performed work of job tracker which performed the tasks of job scheduling as well as management and handling errors in map and reduce operations done by these jobs. Until now, there had been workarounds that addressed the failure of this job management node but the separation of work in the jobtracker was not addressed. In particular, several aspects regarding scalability, cluster utilization, change/upgradation, supporting workloads other than MapReduce itself.

YARN(Yet-Another-Resource-Negotiator) separates this problem by splitting up resource management and job scheduler into different daemons that are then applicable on applications - either as a single job or a graph of interrelated jobs. Resource management is done by a global ResourceManager(RM) that resembles the old job tracker, but exists independently of the job or its application. To perform the jobs for each application, ApplicationMaster(AM) is used which acts as an intermediary between the RM and the nodes, which are in turn manipulated through NodeManager(NM) and it relays the node health and resource requirements to the RM as needed.

The YARN archiecture

A key advantage of YARN is that it supports algorithms other than MapReduce. There are problems that need to have intermediate or realtime results while processing and traditionally can't be used in mapreduce such as graph processing. Simply put in context of hadoop, if one node's results are required by another after or during processing of one or the both records, then batch processing is the answer.

People generally get confused in the context of the newer version of hadoop(the top level project) with MRv2 as the hadoop supports both the MR1 as well as MRv2. The change is simply in the architecture, that can determine the performance(especially post the map stage - wherein the mapreduce implementations often left nodes underutilized at the reduce phase) of the application in use. One key thing to note here is the fact that the YARN is a framework in itself and what hadoop does is to run mapreduce as a job on the YARN. Thus hadoop preserves its existing API but reaps the benefits arising out of dynamic node allocation done internally by YARN.