Introduction to Hadoop on Windows Azure
Apache™ Hadoop™-based Services for Windows Azure is a service that deploys and provisions clusters in the cloud, providing a software framework designed to manage, analyze and report on big data.
Data is described as "big data" to indicate that is being collection is in ever escalating volumes, at increasingly high velocities, and for a widening variety of unstructured formats and variable semantic contexts. Big data collection does not provide value to an enterprise. For big data to provide value in the form of actionable intelligence or insight, it must be accessible, cleaned, analyzed, and then presented in a useful way, often in combination with data from various other sources.
Apache Hadoop is a software framework that facilitates big data management and analysis. Apache Hadoop core provides reliable data storage with the Hadoop Distributed File System (HDFS), and a simple MapReduce programming model to process and analyze in parallel the data stored in this distributed system. HDFS uses data replication to address hardware failure issues that arise when deploying such highly distributed systems.
To simplify the complexities of analyzing unstructured data from various sources, the MapReduce programming model provides a core abstraction that provides closure for map and reduce operations. The MapReduce programming model views all of its jobs as computations over key-value pair datasets. So both input and output files must contain such key-value pair datasets. Other Hadoop-related projects such as Pig and Hive are built on top of HDFS and the MapReduce framework, providing higher abstraction levels such as data flow control and querying, as well as additional functionality such as warehousing and mining, required to integrate big data analysis and end-to-end management.
It addition to the available Hadoop-related ecosystem projects, it also provides Open Database Connectivity (ODBC) drivers to integrate Business Intelligence (BI) tools such as Excel, SQL Server Analysis Services, and Reporting Services, facilitating and simplifying end-to-end data analysis.
This topic describes the Hadoop ecosystem on Windows Azure, the main scenarios for Hadoop on Windows Azure cases, and provides a tour around the Hadoop on Windows Azure portal. It contains the following sections:
Big Data: Volume, Velocity, Variety and Variability. - The qualities of big data that render it best managed by NoSQL systems like Hadoop, rather than by conventional Relational Database Management System (RDBMS).
The Hadoop Ecosystem on Windows Azure - Hadoop on Windows Azure provides Pig, Hive, Mahout, Pegasus, Sqoop, and Flume implementations, and supports other BI tools such as Excel, SQL Server Analysis Services and Reporting Services that are integrated with HDFS and the MapReduce framework.
Big Data Scenarios for Hadoop on Windows Azure - The types of jobs appropriate for using Hadoop on Windows Azure.
Getting Started with Microsoft Hadoop on Windows Azure - Get Community Technical Preview (CTP) access and an introduction to the Apache™ Hadoop™-based Services for Windows Azure Portal.
Resources for Hadoop on Windows Azure - Where to find resources with additional information.
Big data: volume, velocity, variety, and variability
You cannot manage or process big data by conventional RDBMS because big data volumes are too large, or because the data arrives at too high a velocity, or because the data structures variety and semantic variability do not fit relational database architectures.
The Hadoop big data solution is a response to two divergent trends. On the one hand, because the capacity of hard drives has continued to increase dramatically over the last 20 years, vast amounts of new data generated by web sites and by new device and instrumentation generations connected to the Internet, can be stored. In addition, there is automated tracking of everyone's online behavior. On the other hand, data access speeds on these larger capacity drives have not kept pace, so reading from and writing to very large disks is too slow.
The solution for this network bandwidth bottleneck has two principal features. First, HDFS provides a type of distributed architecture that stores data on multiple disks with enabled parallel disk reading. Second, move any data processing computational requirements to the data-storing node, enabling access to the data as local as possible. The enhanced MapReduce performance depends on this design principle known as data locality. The idea saves bandwidth by moving programs to the data, rather than data to programs, resulting in the MapReduce programming model scaling linearly with the data set size. For an increase in the cluster size proportionately with the data processed volume, the job executes in more or less the same amount of time.
The rate at which data is becoming available to organizations has followed a trend very similar to the previously described escalating volume of data, and is being driven by increased ecommerce clickstream consumer behavior logging and by data associated social networking such as Facebook and Twitter. Smartphones and tablets device proliferation has dramatically increased the online data generation rate. Online gaming and scientific instrumentation are also generating streams of data at velocities with which traditional RDBMS are not able to cope. Insuring a competitive advantage in commercial and gaming activities requires quick responses as well as quick data analysis results. These high velocity data streams with tight feedback loops require a NoSQL approach like Hadoop's optimized for fast storage and retrieval.
Most generated data is messy. Diverse data sources do not provide a static structure enabling traditional RDBMS timely management. Social networking data, for example, is typically text-based taking a wide variety of forms that may not remain fixed over time. Data from images and sensors feeds present similar challenges. This sort of unstructured data requires a flexible NoSQL system like Hadoop that enables providing sufficient structure to incoming data, storing it without requiring an exact schema. Cleaning up unstructured data is a significant processing part required to prepare unstructured data for use in an application. To make clean high-quality data more readily available, data marketplaces are competing and specializing in providing this service.
Larger issues in the interpretation of big data can also arise. The term variability when applied to big data tends to refer specifically to the wide possible variance in meaning that can be encountered. Finding the most appropriate semantic context within which to interpret unstructured data can introduce significant complexities into the analysis.
The Hadoop ecosystem on Windows Azure
Hadoop on Windows Azure offers a framework implementing Microsoft cloud-based solution for handling big data. This federated ecosystem manages and analyses large data amounts while exploiting parallel processing capabilities, other HDFS architecture optimizations, and the MapReduce programming model. Technologies such as Sqoop and Flume integrate HDFS with relational data stores and log files. Hive and Pig integrate data processing and warehousing capabilities. Pegasus provides graph-mining capabilities. Microsoft Big Data solution integrates with Microsoft BI tools, including SQL Server Analysis Services, Reporting Services, PowerPivot and Excel. Microsoft BI tools enable you to perform a straightforward BI on data stored and managed by the Hadoop on Windows Azure ecosystem. The Apache-compatible technologies and sister technologies are part of this ecosystem built to run on top of Hadoop clusters are itemized and briefly described in this section.
Hive is a distributed data warehouse managing data stored in an HDFS. It is the Hadoop query engine. Hive is for analysts with strong SQL skills providing an SQL-like interface and a relational data model. Hive uses a language called HiveQL; a dialect of SQL. Hive, like Pig, is an abstraction on top of MapReduce and when run, Hive translates queries into a series of MapReduce jobs. Scenarios for Hive are closer in concept to those for RDBMS, and so are appropriate for use with more structured data. For unstructured data, Pig is better choice. Hadoop on Windows Azure includes an ODBC driver for Hive, which provides direct real-time querying from business intelligence tools such as Excel into Hadoop. For more information, see Welcome to Apache Hive!
Mahout is an open source machine-learning library facilitating building scalable matching learning libraries. Using the map/reduce paradigm, algorithms for clustering, classification, and batch-based collaborative filtering developed for Mahout are implemented on top of Apache Hadoop. For more information, see What is Apache Mahout.
Pegasus is a peta-scale graph mining system running on Hadoop. Graph mining is data mining used to find the patterns, rules, and anomalies characterizing graphs. A graph in this context is a set of objects with links that exist between any two objects in the set. This structure type characterizes networks everywhere, including pages linked on the Web, computer and social networks (FaceBook, Twitter), and many biological and physical systems. Before Pegasus, the maximum graph size that could be mined incorporated millions of objects. By developing algorithms that run in parallel on top of a Hadoop cluster, Pegasus develops algorithms to mine graphs containing billions of objects. For more information, see the Project Pegasus Web site.
Sqoop is tool that transfers bulk data between Hadoop and relational databases such a SQL, or other structured data stores, as efficiently as possible. Use Sqoop to import data from external structured data stores into the HDFS or related systems like Hive. Sqoop can also extract data from Hadoop and export the extracted data to external relational databases, enterprise data warehouses, or any other structured data store type. For more information, see the Apache Sqoop Web site.
Flume Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large log data amounts to HDFS. Flume's architecture is streaming data flow based. It is robust and fault tolerant with tunable and reliability mechanisms with many failover and recovery mechanisms. It has a simple extensible data model enabling online analytical applications. For more information, see the Flume incubation site.
Business intelligence tools
Familiar Business Intelligence (BI) tools such as Excel, PowerPivot, SQL Server Analysis Services and Reporting Services retrieves, analyzes, and reports data integrated with Hadoop on Windows Azure using ODBC drivers. The Hive ODBC driver and Hive Add-in for Excel are available for download on the Hadoop on Windows Azure portal How To Connect Excel to Hadoop on Windows Azure via HiveODBC.
* For information Analysis Services, see SQL Server 2012 Analysis Services.
* For information Reporting Services, see SQL Server 2012 Reporting.
Big data scenarios for Hadoop on Windows Azure
An exemplary scenario that provides a case for an Hadoop on Windows Azure application is an ad hoc analysis, in batch fashion, on an entire unstructured dataset stored on Windows Azure nodes, which do not require frequent updates.
These conditions apply to a wide variety of activities in business, science, and governance. These conditions include, for example, monitoring supply chains in retail, suspicious trading patterns in finance, demand patterns for public utilities and services, air and water quality from arrays of environmental sensors, or crime patterns in metropolitan areas.
Hadoop is most suitable for handling a large amount of logged or archived data that does not require frequent updating once it is written, and that is read often, typically to do a full analysis. This scenario is complementary to data more suitably handled by a RDBMS that require lesser amounts of data (Gigabytes instead of Petabytes), and that must be continually updated or queried for specific data points within the full dataset. RDBMS work best with structured data organized and stored according to a fixed schema. MapReduce works well with unstructured data with no predefined schema because it interprets data when being processed.
Getting started with Microsoft Hadoop on Windows Azure
The Hadoop on Windows Azure CTP
The Hadoop on Windows Azure service is available by invitation only during this Community Technical Preview (CTP). The CTP purpose is for you to test Hadoop-based service on Windows Azure, become more familiar with it, and provide feedback. The process for gaining access is outlined below.
The portal used by Apache Hadoop-based services for Windows Azure
The Microsoft implementation of Hadoop on Windows Azure uses a Portal to provision new Apache Hadoop clusters. Clusters provisioned on the portal are temporary and expire after several days. When there is less than six hours left on the clock, an expiration time extension is allowed. These clusters run jobs that process data either on the cluster or located elsewhere. For example, the data could reside in a Windows Azure account or be transferred to the cluster over FTP.
The advantage of using a temporary cluster is that there is no cost to maintain the hardware needed for the MapReduce parallel processing jobs. You use the cluster and then release it or allow it to expire. Apache Hadoop deployment solutions are also available for deploying Apache Hadoop to a Windows Azure account, or on-premise hardware that you manage.
Getting access to the CTP and deploying an Hadoop cluster
The steps for getting access to the Hadoop on Windows Azure, logging into the site and deploying your first Hadoop cluster on Windows Azure are as follows.
- Provide information about yourself, business domain, and technical scenario by completing the Apache Hadoop for Windows Azure & Windows Server Connect profile. You need a LiveID (for example, Hotmail.com, Live.com) to complete and submit the Connect profile. This may require some preliminary steps.
- Once you receive an invite code, go to the HadoopOnAzure.com site and click Sign In.
If it is your first time, you are asked to allow HadoopOnAzure.com access to your LiveID profile information. Click Yes. The Got a code dialog box opens.
In the Got a code dialog box, enter invite code, and then click Enroll.
- For a new account, your first task is to create a Hadoop on Windows Azure cluster. Your options are as follows: Your options are:
DNS Name: Choose a name for your cluster. Here "samples8" is used.
Cluster Size: Select what size of cluster to test against. Currently, the Small 3 node cluster must be used.
Cluster Login: Provide a Username for the cluster and Password for logging into it. Here "someUserName" the default value, is used.
- Click Request cluster (right side bar, under the green bar)
- At this point, the Hadoop on Windows Azure service begins creating the new cluster, which takes up to tens of minutes, depending on the number of nodes and the number of clusters.
Tour of the portal
Once logged in with a deployed cluster, your account page opens.
The Hadoop on Windows Azure Portal
There are three sections on the portal grouping the icons linked to Hadoop on Windows Azure management features:
- Your Tasks: This section enables you to create new jobs and keep track of jobs you have previously created or run.
- Manage your Account: This section tracks Job History on your account, gives access to the Samples that come ready to run with Hadoop on Windows Azure, and provides Downloads of client utilities for Hadoop on Windows Azure, such as the Hive ODBC driver and Hive Add-in for Excel. (The Billing History can be ignored during the developer preview.)
To explore Portal functionality, click any icon. For use of the New Jobs and Interactive Console functionality, see the Running Jobs with Hadoop on Windows Azure topic. For walkthroughs on running the Samples, see the Samples topics. You can use the Hadoop file system commands from the command window on the head node, which is available in the Remote Desktop.
Resources for Hadoop on Windows Azure
Microsoft: Hadoop on Windows Azure
Microsoft: Windows and SQL Database
Microsoft: Business Intelligence
- Apache Hadoop - software library providing a framework that allows for the distributed processing of large data sets across clusters of computers.
- HDFS - Hadoop Distributed File System (HDFS) is the primary storage system used by Hadoop applications.
- Map Reduce - a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes.