Enchanting cloud with juju !

Automating infrastructure is not a new thing, but doing so with the ease of setting up paas is still in its infancy. As an interesting solution to this, Cannonical has recently come out with a PAAS tool Juju that can easily setup, configure, provision, include and change master/slaves, provide continuous integration environment among other things. This tool has not come out from wild, but is currently in use: Juju and MaaS are the default deployers for Openstack. So all large Openstack deployments that are currently happening on Ubuntu use Juju and MaaS. Additionally Ubuntu One and other Ubuntu cloud services are all running on top of Juju.
Juju might sound similar to Puppet or Chef, but rather than just providing a server, this provides various services. Internal to each juju are different charms that call various programs and provide the infrastructure to the juju. The charms App Store contains charms on various services which can be used in the juju to solve  a particular task. There are currently hundreds of different charms available - from databases to php/ruby/python servers to nosql and hadoop instances and a growing community of developers who customize and release their own customized charms.
One feature that is worth mentioning is the ability to create a traditional Paas such as CloudFoundary as a charm to run over juju.
For instance, here's the procedure to boot off a rails sever:
juju deploy rails myapp --config sample-app.yml
(This yml file contains the url to the github repository containing the charm)
juju deploy haproxy
juju add-relation haproxy myapp
Now the database can be created:
juju deploy postgresql
juju add-relation postgresql:db myapp
then migrate the database
juju ssh myapp/0 run rake db:migrate
Finally expose the proxy
juju expose haproxy
and view the status, which should provide you with the public url
juju status haproxy
The coolest feature now is to add/remove servers horizontally through gui or through command as:
juju add-unit myapp 
juju remove-unit myapp
Currently, there is also a charm-championship taking place  that encourages developers to take part in developing customized charms for various themes.
So what are you waiting for?
Jump right in at https://jujucharms.com to see it in action.

Performing cron jobs in rails through clockwork

The Clockwork Gem allows us to run cron jobs in a ruby application. Since I was using this today, its documentation is a bit of a blocker. The github wiki gives the following example:

require 'clockwork'
include Clockwork

handler do |job|
  puts "Running #{job}"
end

every(10.seconds, 'frequent.job')
every(3.minutes, 'less.frequent.job')
every(1.hour, 'hourly.job')

every(1.day, 'midnight.job', :at => '00:00')

However, I needed to run custom tasks, which is not mentioned there. Here's how I did the necessary work

require_relative "../config/environment"
require_relative "../config/boot"
require 'clockwork'

include Clockwork

handler do |job|
 puts "running the scheduled job #{job}."
end

every(30.seconds, 'test job execution'){Module::Class.new.method_to_call}

I added the require_relative calls to environment.rb & boot.rb to load the rails framework from this clock.rb file, which is placed in lib folder. This example is calling a custom module placed in the lib folder in the same rails app.
As this application is to be deployed over heroku, we need to have 2 dynos, one for the rails and another for the extra task. We define this through Procfile file present at rails root with the following contents:

web    bundle exec unicorn -p $PORT -c ./config/unicorn.rb
clock  bundle exec clockwork app/lib/clock.rb

Which does the necessary things for me at heroku. To run long running tasks, you may want to use delayed jobs, but it will involve another dyno for management so I left it in this solution.

Relax, have fun and Play.. er Learn

Relax Play and Learn is the new mantra.. at least in learning Rails and related web tools used for development.
Maybe its just me, but over the past few days, I am increasingly finding newer and diverse forms of learning web development, especially using advanced frameworks like rails.
Some of these are directed at learning inter-spread with videos that users can learn at their own pace like the rails for zombies and others that focus on user provided content created by users themselves. Then, there are language specific courses like http://www.rubylearning.com .
When I started learning rails, how much I wish that I had access to similar resources as these are godsend for learners; they serve the dual purpose of ease in learning as well as enjoyment for beginners.
As I write this post, I am also casually looking at various websites like peepcode.com that offer much higher quality screen casts than railscasts.com especially for beginners.
What is surprising is that other technologies do not have this type of support, but web application development centered around rails, scripting languages and HTML5, CSS3 and jQuery combined are increasingly finding this support. One can only hope that this type of open and user friendly learning increases its umbrella and allows more technologies into its fold in the near future.

So If you are a beginner or are considering using rails, my humble advice to you is to bookmark these and start when you have some extra/free time.

Demystifying DevOps

There has been a lot of noise recently about this 'DevOps' movement. In this blogpost, I present my take on the topic and hopefully, help the reader towards understanding it.

Wikipedia puts this as:
 "DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals."

Image: Wikimedia Commons

However, owing to pressures often within large corporations having plentiful resources at their disposal, it is easy to create a hype around a process that in general, is common sense and at the very best is automation.
Speaking of agile delivery, it is easy to deliver a product in small teams that the big ones. As my experiences go, in my present company the product gets a new release every evening(India time), prior to which the changes made in the release are tested by the commiters of the changes itself before the decision about build is taken. After the release is done, basic testing as well as testing of the new features is carried out. While this is pain staking methodology, this works well as the team size is relatively less, leading to fewer overall changes. As an astute reader would have pointed out by now, what is missing is automation as well as demonstration of scalability - about what to do when the product goes live. Some uneasy questions like these are answered by the DevOps which:

• Focuses on convergence of Development and Operations.
• Minimizes the gulf/barrier between development and production.
• Brings about QA into development.
• Is agile, and strives the processes after development to be agile/flexible
• Required as rapid scaling & uptime of applications is necessary.
• Focus on automation, towards reducing repetitive work

So, where does a software developer like me comes in?
Here's what I typically deal while playing with web applications: Java, ruby and some combination of these thrown in together with the odd php or python based applications. In this recession prone economy, there is a requirement for cutting costs; which can be effectively met by ensuring that developers also stick in charge for the deployment process. As this lean process gains its acceptance, it is obvious that development is going to be easier for operations people and similarly for developers, deployment/production environment is going to reduce its level of complexity.

Where does it affects us ?

For the java developers, the heavy duty work gets 'magically' done by the enterprise servers that we deploy on. As java is built with performance in mind, it is easy to ramp up the scale of operations (tomcat is insufficient, deploy on glassfish, Too many queries, use memcached, database connections are piling, use connection pooling).
However, the (physical) servers over which these applications are deployed in production also need to be profiled and studied.

To run load/stress tests, one needs to have a production environment. These are easily available as Amazon Machine Instances or rackspace or any other myriad cloudproviders, however this costs money and given the increasing hardware capabilities, one can perform similar tasks without using the cloud(and paying for such kinds of testing).

Virtualbox needs no introduction and is a free virtulization tool from Oracle (it was a sun offering earlier). However, for our example, even virtualbox fails out of box as it can only provision a single vm. To enhance its capability, one can use Vagrant, which is a tool for building and distributing virtual development environments. In other words, it is what rvm is for ruby; a manager that lets the user have clean state (of the server) as well as abstracted setup(different server configurations from same set of VMs). Vagrant is primarily a driver for virtualbox vms.

One can work with it on a per-project basis(similar to a version control) and configure it through a plaintext file, Vagrantfile which vagrant uses as a DSL. The nicest thing about it is that it does everything that you need virtualbox UI for, which is really beneficial while running numerous headless servers.
For example:

Vagrant::Config.run do |config|

   config.vm.define :app do|app_serv_config|

     app_serv_config.vm.customize ["modifyvm", :id, "--name", "app", "--memory", "512"]

   end

   config.vm.define :app2 do|app_serv_config|

     app_serv_config.vm.customize ["modifyvm", :id, "--name", "app2", "--memory", "512"]

   end

   config.vm.define :db do|db_config|

     app_serv_config.vm.customize ["modifyvm", :id, "--name", "db", "--memory", "1024"]

   end 
end

This is just getting started with different servers as in real life, you'd need to provision the vms - setup them with useful stuff and services, deploy and run your programs on them. Two of the widely popular tools that deserve a mention are: puppet and chef.
While the documentation and discussions around them typically involve rails, there is no reason why java, and for that matter any other technology stack can not be used.
Puppet helps us chiefly with setting up the stuff; it is not big on automation, but saves us the work of setting up different softwares in a vm manually.
For instance, the following puppet config file sets up apache and ensures that it runs as a service during vm startup:

Exec {

  path => "usr/local/sbin:/usr/local/bin:/bin:/opt:/usr/sbin"

}

package {

  "apache2":

      ensure => present

}

service {

    "apache2"

         ensure => true,

         enable => true

}

Once we've set up our environment, only half of the battle is over as we also need to monitor and report the status of the environment to evaluate the areas of concern in the setup.
Nagios is a wonderful monitoring tool that displays the health of server - which is needed in a headless environment. To use it, we need to build a puppet module and  enable its service and use the nagios server interface on the vm itself to gleam the details of the vm, displayed as services  as well as re-shedule it when needed. To monitor the nagios instance itself, one can use a service like pingdom.

There are various other tools related with setting up of the servers, but would stretch the discussion far from the concept of devopts in practice. Thus, there is a need to resist the urge of going on to cloud, but keep the production environment close to the development environment itself. This would not result in the elimination of cloud as production environments are now predominantly becoming cloud based, but will go a long way in ensuring lesser pains when the product goes 'live'.

Going to classes, the online way

Recently, there have been a lot of announcements regarding free online classrooms for instructor-led E-learning. I was a bit skeptical of it at first, but now am glad that I enrolled myself into it.

Some of the advantages that online courses offer over conventional courses:

• All the comforts of an online\distance learning course for free.
• Vibrant discussion forums that help erase doubts.
• High quality material: These courses are taught as-is in the real classes in top universities.
• Hands-on learning: The learner gets to create programs and solve practical problems related with the topic.
• Stay focused on a specific topic - working as a professional developer, it is natural to get stuck with the mundane aspects of programming and forget the big picture as there is the excitement into hacking into different topics without much time constraints, like in college. 

I am currently enrolled in two courses at coursera.org and one at edx.
My first course is a 10 week course about Big Data, which I had an inking that it would be related with hadoop and other high level aspects of it, but what surprised me was that the first three weeks of this course have passed and the instructor is tirelessly helping us understand the basics/theory of the importance of big data and its key challenges. This being an IIT post graduate level course, further lends credibility to its undivided concentration over the background aspects of  big data (and intelligently breaking it down to look, listen, learn, connect, predict, correct).
Another course that has just started is Programming in Scala by none other than its inventor itself, Martin Odensky. What is pleasant about the course is that right from day one, the course uses test driven development and sbt for building & submitting programming excercises.
I also have high hopes for the upcoming SAAS course offered by MIT that seems promising and offers me to create a cloud based solution after a long time.

These myriad courses do take up time, especially the after work hours, but is a better use to put my time rather than just browsing around. So, lot of efforts and headaches are induced into this process, but is worth a try - whether you are a student, a developer or a curious bystander who happens to have some of these type of subjects affecting them in one way or another.