“Time for IBM to Take Its Hadoop Case To The Streets” is the title of Jeff Kelly’s post on his ServicesAngle blog. Before I say anything, I want to do a usual disclaimer that opinions I express here are mine and mine alone. I don’t speak for IBM. I can however, talk about how I see things as both a member of the IBM Hadoop (BigInsights) team and a community participant. And I have to agree with Jeff … it is time that IBM took its Hadoop big data message out of the C-level executive offices where it has indeed played very well and brought it in to the trenches. In my observation, Hadoop community had a fairly guarded reception of the IBM Hadoop story so far. That is understandable. Here comes big multinational corporation getting in to the cool kids play pan. Well, more of these players are coming in to the world of Hadoop and , in my biased opinion, IBM is the nicest of them.
IBM has a long-standing commitment to open source. I know, everyone says that, but I think IBM actually demonstrated this consistently. The way IBM got in to Hadoop is a perfect example. Right of the bat, IBM said that it will not fork Hadoop. One would expect open source guys would stand up and cheer. Another thing that open source community should be happy about is what IBM can contribute back. If I am not mistaken IBM BigInsights development team has 4 Apache Hadoop committers. Granted, both Cloudera and Hortonworks have contributed more to the open source but is this such an important criteria? If we were to judge by the sparing between Cloudera and Hortonworks we would conclude that it is. I think that what really matters is bringing legitimacy for Hadoop to the enterprise audiences. So far, a lot of cutting edge Hadoop usage occurred in large-scale internet companies and startups. Enterprise IT views these companies as yahoos (pardon the pun) and approaches the tools they use with a great deal of suspicion. IBM entry in to this market is lending much-needed credibility to Hadoop. 100 years of being a supplier of IT solutions is what IBM brings to the table. I would argue that this is of greater value to the adoption of Hadoop than any technical contribution IBM could have made. IBM did contribute JAQL, a pretty decent scripting and query language. However, as valuable as JAQL is, it pales in comparison to the credibility “contribution”. Enough said.
One of the things we hear as the key impediment to Hadoop adoption is lack of skills. To help solve this we organized a community project called BigDataUniversity.com. This is what I call @your place, @your pace education that is available free of charge. Since we launched it this summer it has become wildly successful with over 8000 people registered and over 1700 people enrolled in the Hadoop Fundamentals course. IBM has been generous enough with the sponsorship of the site donating resources, products, my and my team’s time. IBM sponsored the Hadoop Programming Challenge by picking up all travel and attendance costs for 3 finalists to come to the IOD conference in Las Vegas. Our executive team has been very supportive of the effort and the people who have gone through the course are very appreciative of the level of support BigDataUniversity.com received. Our 3 finalists knew next to nothing about Hadoop only a few weeks ago. They completed the course, passed final test and managed to submit a project that actually does something interesting and valuable. All of that literally in days. Congratulations to Raul Guiu (Spain), Stephen Camera-Murray (USA), and William Cheung (Canada) member of our local Toronto Hadoop User Group. We will see you in Vegas!
Are we done? We are just getting started. We are coming to the Hadoop community and we are the good guys. Cloudera, Hortonworks and others have nothing to fear. Yes, IBM will compete for the business but IBM is also going to expand the opportunity. The Hadoop pie is about to become a lot bigger … no need to fight for the crumbs.