Enchanting cloud with juju !

Automating infrastructure is not a new thing, but doing so with the ease of setting up paas is still in its infancy. As an interesting solution to this, Cannonical has recently come out with a PAAS tool Juju that can easily setup, configure, provision, include and change master/slaves, provide continuous integration environment among other things. This tool has not come out from wild, but is currently in use: Juju and MaaS are the default deployers for Openstack. So all large Openstack deployments that are currently happening on Ubuntu use Juju and MaaS. Additionally Ubuntu One and other Ubuntu cloud services are all running on top of Juju.
Juju might sound similar to Puppet or Chef, but rather than just providing a server, this provides various services. Internal to each juju are different charms that call various programs and provide the infrastructure to the juju. The charms App Store contains charms on various services which can be used in the juju to solve  a particular task. There are currently hundreds of different charms available - from databases to php/ruby/python servers to nosql and hadoop instances and a growing community of developers who customize and release their own customized charms.
One feature that is worth mentioning is the ability to create a traditional Paas such as CloudFoundary as a charm to run over juju.
For instance, here's the procedure to boot off a rails sever:
juju deploy rails myapp --config sample-app.yml
(This yml file contains the url to the github repository containing the charm)
juju deploy haproxy
juju add-relation haproxy myapp
Now the database can be created:
juju deploy postgresql
juju add-relation postgresql:db myapp
then migrate the database
juju ssh myapp/0 run rake db:migrate
Finally expose the proxy
juju expose haproxy
and view the status, which should provide you with the public url
juju status haproxy
The coolest feature now is to add/remove servers horizontally through gui or through command as:
juju add-unit myapp 
juju remove-unit myapp
Currently, there is also a charm-championship taking place  that encourages developers to take part in developing customized charms for various themes.
So what are you waiting for?
Jump right in at https://jujucharms.com to see it in action.

Easier hadoop job creation through mrjob

Today, I came across mrjob, which is a python package that enables creation of a mapreduce job on a hadoop cluster using python. This is indeed useful as other python based mapreduce frameworks like octopy are easier to setup but do not scale well. This was created and opensourced by Yelp, the food rating website. You can find it on github here.
Basically mrjob performs the following main operations for you:

• Write multi-step MapReduce jobs in pure Python
• Test on your local machine
• Run on a Hadoop cluster
• Run in the cloud using Amazon Elastic MapReduce (EMR)

Thus, it allows a developer to have separate development, staging and production environments.
Mrjob is available for python 2.5+ and installs without any glitch on my python 2.7 setup.
The python script in use can be used both as standalone or as a regular job on hadoop cluster. It can consist one of mapper, combiner or reduce function. For example, the sample program consisted of the following:

from mrjob.job import MRJob
import re

WORD_RE = re.compile(r"[\w']+")


class MRWordFreqCount(MRJob):

    def mapper(self, _, line):
        for word in WORD_RE.findall(line):
            yield word.lower(), 1

    def combiner(self, word, counts):
        yield word, sum(counts)

    def reducer(self, word, counts):
        yield word, sum(counts)


if __name__ == '__main__':
    MRWordFreqCount.run()

The raw data gets sent to mapper() function that creates the keys in the form of word, 1 for each mapped word. this data is then sent to whichever processer is in use by transmitting through json (although there are other binary protocols as well, which can better be applied in larger scenarios). These protocols are basically the serialization mechanism between the program and its backend here.
The configuration file does the actual work of deciding whether to use python based mapreduce on the same machine itself, or on a local/remote hadoop cluster and even for amazon elastic mapreduce environment. To do this, I simply put the .mrjob.conf file on my home folder with the following:

runners:
  hadoop:
    base_tmp_dir:/tmp/hjobs
    python_archives: &python_archives
    setup_cmds: &setup_cmds
    upload_files: &upload_files
  local:
    base_tmp_dir: /home/sumit/mrjob/tmp
    python_archives: *python_archives

I ran the supplied test over both local install of python as well as on my local hadoop cluster without any problems and am suitably impressed with its compactness, which might come handy to me or you someday!

MapReduce2.0: Next Generation Mapreduce using YARN

There has been a lot of long overdue changes in apache hadoop, the most popular framework to perform mapreduce over big data.
As the users of hadoop may recall, one of the vulnerability in mapreduce was the presence of a single node that performed work of job tracker which performed the tasks of job scheduling as well as management and handling errors in map and reduce operations done by these jobs. Until now, there had been workarounds that addressed the failure of this job management node but the separation of work in the jobtracker was not addressed. In particular, several aspects regarding scalability, cluster utilization, change/upgradation, supporting workloads other than MapReduce itself.

YARN(Yet-Another-Resource-Negotiator) separates this problem by splitting up resource management and job scheduler into different daemons that are then applicable on applications - either as a single job or a graph of interrelated jobs. Resource management is done by a global ResourceManager(RM) that resembles the old job tracker, but exists independently of the job or its application. To perform the jobs for each application, ApplicationMaster(AM) is used which acts as an intermediary between the RM and the nodes, which are in turn manipulated through NodeManager(NM) and it relays the node health and resource requirements to the RM as needed.

The YARN archiecture

A key advantage of YARN is that it supports algorithms other than MapReduce. There are problems that need to have intermediate or realtime results while processing and traditionally can't be used in mapreduce such as graph processing. Simply put in context of hadoop, if one node's results are required by another after or during processing of one or the both records, then batch processing is the answer.

People generally get confused in the context of the newer version of hadoop(the top level project) with MRv2 as the hadoop supports both the MR1 as well as MRv2. The change is simply in the architecture, that can determine the performance(especially post the map stage - wherein the mapreduce implementations often left nodes underutilized at the reduce phase) of the application in use. One key thing to note here is the fact that the YARN is a framework in itself and what hadoop does is to run mapreduce as a job on the YARN. Thus hadoop preserves its existing API but reaps the benefits arising out of dynamic node allocation done internally by YARN.