Optical Character Recognition with Nodejs

Today, I was prototyping a OCR tool to use as a web based API. My first intention was to develop a desktop version in python and provide it via flask application, but using node proved to be a lot easier.
Node.js has a library binding with Tesseract which proved to be quite handy.
I simply installed the library first using npm

npm install nodecr

Next, in a simple node application, I processed a user uploaded image:

ncr.process(filePath, function(error, text){ ....

This callback function performed the task of parsing the image and providing the text.

I have uploaded it into a generic application at https://github.com/SumitBisht/node-ocr and hope you will find it helpful.
Note that this is a really dumb form of OCR and the image sanitation needs to be provided first into it, on which I am working upon.

Book Review: Thinking With Data

Thinking With Data - 

How to turn Information into Insights

 During the past week, I read this book by Max Shron which addresses the big data challenge from a different perspective - What questions to ask from data to gain the best possible or the most beneficial answer for its owners. The importance of this book is not just confined to people working in some niche segment, but associated with big data in general- from students/researchers to data analysts and developers. if you are expecting some big data technology or implementation, you'll be disappointed. Instead, the book focuses essentially on the problem of deciding What to find from data. For instance, the scope of this problem is further extended into 4 parts - context, needs, vision, outcome.

As I am currently working full time on big data project where similar problems crop up - for instance, we know how to perform a predictive analysis algorithm but the main challenge is to select a specific algorithm and fields to obtain the mathematical result that can infer results which actually solve business problems. As the author is into data strategy consultancy and a former data scientist, the tone of the book is quite practical and uses real world examples to better explain its concepts where needed.

One of areas where I felt the book was weak was in this assumption that the problem/challenge will be a greenfield one and the legacy/existing systems will not influence in the decision/role making process of proposed solutions. The presence of existing big data strategies in place can act as the guideline for the future process. Another thing which was missing were the presence of anti-patterns of the big data formulation strategy - such as what not to do while tinkering with the data and algorithms to extract intelligence from data.

In spite of being shorter than other books of the same topic, this book does a overall good job in discussing the problem of what to extract for big data analysis and is definitely a must read and reference for anyone dealing with the same and avoiding showing unnecessary noise instead of meaningful data.

Disclaimer: This book has been provided me by OReilly under their Blogger Review program.

Book Review: Ruby Under A Microscope

 

 

 Ruby Under A Microscope

 

Author: Pat Shaughnessy

During the past few weeks I've been reading this book, 'Ruby Under A Microscope' by Pat Shaughnessy which covers internals of virtual machine used by different implementations of Ruby programming language. I would consider myself lucky to have an opportunity to review this book under the Oreilly's blogger review program as this is a must-read for any Ruby developer and specifically to someone doing a Ruby vs Java comparison. While the virtual machine internals and algorithms used are widely discussed and known in Java, the same cannot be said for Ruby and Ruby developers generally shrug it off when confronted with interpreter/platform specific issues and seek refuge with changing/scaling up resources.
Detailing of internals help in understanding the behavior of the language as the optimizations made internally by the compiler have a direct bearing on the behavior of the program and can exhibit unexpected behavior. What is of peculiar interest here is that this book delves into the internals down to the compiler in order to understand the resultant behavior/performance.
The book starts with the tokenizing and parser mechanism preset in ruby and continues to the compilation of interpreted script into YARV instructions. This is then explained in next chapter where the program call stack and variables used internally is detailed.

Control branching and method dispatch are discussed before the discussion moves into Class, Object and Method mechanism. The hash mechanism used to save objects by ruby is detailed next before blocks are discussed and Lambda and Procs are discussed together with Stack vs Heap memory. This is followed with advice which is worth reading for those not familiar with metaprogramming ruby and while this is not as verbose as 'Metaprogramming Ruby', it does a good job explaining various reflection based constructs.
After covering the language internals, two of the leading platforms, JRuby and Rubinius are detailed and compared against MRI(YARV). Lastly Garbage collection is discussed and comparison between ruby and java garbage collectors is discussed. Pat really does well in explaining the differences and also provides ideas to explore things based on the GC profile report here. I'd also advise to check out his blog for more such topics.

I'd heartily recommend this book to anyone who is interested - from students to expert pros alike as anyone coding/using Ruby ought to know its internals and have more confidence in developing and using this platform for non-trivial tasks.
Disclaimer: This book has been provided me by OReilly under their Blogger Review program.

User management in Node through Lockit.js

Lockit.js is a user management tool in express.js that performs the signup/login of users and create corresponding user entries in the database using standard best practices. To explain in a nutshell, it is quite similar to Devise, a user authentication gem used in Rails based projects. As Lockit is available as an express module, it can easily be used in any project by npm install:

npm install lockit

Although this project is currently in its infancy, it offers an uncluttered approach towards user management at cost of being opinionated, which is okay as long as you need to save and manage standard user actions.

As the project github and website page explains, you can easily setup the application in any node based application that uses either couchdb or mongodb and in a mongodb based application, you only require a config.js file that takes care of configuration and specify it inside the main application file as:

  lockit(app, config);

Only the lockit and config are needed to be required in the main module of the express application.

Inside this config file, you can specify parameters like:

• Database(URL) [Mandatory]
• Application Details
• Confirmation email sender's mail configuration
• Login attempt, account lock and verification link settings
• Routes to various actions
• Verification and confirmation mail templates

Depending upon your requirements, you would require changing its internals which can easily be done in the dependent libraries from the Lockit. While listing from npm, it looks like the following diagram which makes the description of Lockit more clearer.

As it is opinionated, its views require twitter bootstrap css to display and apart from mongodb and couchdb, other databases are not supported at the time of this writing.
Update: The views are customizable - the config file provides various templates; depending upon your requirements, you can specify them manually:

It is worth mentioning that you can use the signup or password reset token and append it to signup or reset password routes, depending upon object state to keep exploring the library without acutal authentication mail sending facility.
While exploring this library, I've created a sample node application hosted at github that tries to cover this library while using minimum code.

Rant against SAP

For the past 2 months, I've been involved with one sap hana project of which I am doing predictive analysis(PAL). Since this is a proprietary technology, there is an inheritable handicap when it comes to trying various thing out of the ordinary. Basically the lowdown of this tech is that SAP HANA is a memory resident database. The definition ends there - no one has compared it with an existing solution like sqlite and especially this when the people at SAP are trying to market this inside a bundle of software aimed at startups - not cool attitude here, sap and I wonder why someone getting this step-motherly treatment would base his/her startup over this app stack?

For doing predictive analysis, the recommended way is to use the pal library- which consumes memory on the server itself for speeding things up - but currently is liable to crash the server by getting stuck. Call my approach stupid but I try to run PAL analysis over entire table - it basically freezes in execution and any other PAL job cannot run. Today, it took me entire day to get a calculation view straight - and don't ask me how because for the majority of the day I was getting aggregates of the results when the sql script simply contained:
var_out = select * from ..

There is a huge talk about integrating PAL with hadoop, but it is only for non-structured stuff and no MR job is present to simplify things for server jobs. I hope to see some active work in future for this.

What is most baffling for me is the SAP Hana studio that has tons of blocking calls like right clicking on the packages to create a new calculation view:

What I fail to understand is why the menu is not displayed immediately. Also, the overly usage of blocking IO is a very poor way - there are ways to run non blocking jobs in eclipse rcp and when developing eclipse plugin based projects, we were dissuaded from following the easier approach of blocking the user and running the job first. I understand that the team developing sap is having low latency to their sap cloud, but there are developers from India like me for whom network latency does play an important role.
I do hope things improve in the future versions where these problems do not harass us developers.