HBaseball

http://www.cloudera.com/resource/intorduction-hbase-todd-lipcon/

HBase : open source, distributed, sparse (no strict schema), column oriented (control how stored on disk) sorted map data store, modeled after Big Table.

Usage scenario: alot of data, very high write throughput (i.e. sequential writes), easy to scale (distribute across machines), data layout efficient(key look up disk seeks efficient and cost transparent some rows filled in some not still efficient)

column oriented – every row with same column, transaction on single row basis, not full acid, hbase no sql except with hive, but not realtime,

get : single row get columns

put : put row these columns

scan row x in sorted order until row y, some filtering

indexing – primary key only

clustered indexes – primary key is clustered key index

Hbase built on top of Hadoop, requires HDFS, work with map reduce but built on top of it.

HDFS sequential writes , cant update in middle of file, provides streaming I/O

HDFS lacks random read/write capabilities

HBase random read and write in middle of file

converts random writes to writes into log, merge log back into table

log structure merge trees 89

http://www.cloudera.com/resource/chicago_data_summit_apache_hbase_an_introduction_todd_lipcon/

Failover handled with Zookeeper

Sorted Map Datastore:

Not a relational database

Tables consists of rows and primary key

Each row any number of cols

rows stored in sorted order

Have primary key and columns have attributes (column families) that can store many different things

logical view (row key : data) :

– info and roles are column families

– Can have versions of the data

cutting (row key): info {height: ‘9ft’, state : ‘CA’}, roles {ASF : ‘director’, Hadoop : ‘Founder’}

tlipcon (row key): info {height: ‘5ft7’, state : ‘CA’}, roles {Hadoop : ‘Committer’@ts=2010,Hadoop :’PMC’@ts=2011, Hive: ‘Contributor’}

physical view (for each column family, row key (primary key), col key (what data), time stamp, value) :

– column families stored separately on disk

info column family:

cutting (row key) , info.height (col key) , 12345678910 (timestamp), 9ft (value)

cutting (row key) , info.state (col key) , 12345678911 (timestamp), CA (value)

tlipcon(row key) , info.height (col key) , 12345678912 (timestamp), 5ft7 (value)

tlipcon (row key) , info.state (col key) , 12345678912 (timestamp), CA (value)

– roles column family:

cutting (row key) , roles.ASF (col key) , 12345678910 (timestamp), Director (value)

cutting (row key) , roles.Hadoop (col key) , 12345678911 (timestamp), Founder (value)

tlipcon(row key) , roles.Hadoop(col key) , 12345678912 (timestamp), PMC (value)

tlipcon (row key) ,roles.Hadoop(col key) , 12345678912 (timestamp), Committer (value)

tlipcon (row key) ,roles.Hive(col key) , 12345678912 (timestamp), Contributer (value)

– Sorted in ascending order by row key and column key, and descending order by timestamp

– column families:

each column may have different access patterns or properties

can configure compression, cache priority, #versions properties

– Java API, Rest Calls, Apache Thrift, Hive (sql)/Pig (hybrid) integrate

get(row)

put (row,Map)

scan (key-range, filter)

http://jimbojw.com/wiki/index.php?title=Understanding_Hbase_and_BigTable

HBASE Architecture

” HBase handles basically two kinds of file types. One is used for the write-ahead log and the other for the actual data storage. The files are primarily handled by the HRegionServer‘s.”

You may also notice that the actual files are in fact divided up into smaller blocks when stored within the Hadoop Distributed Filesystem (HDFS).

The general flow is that

a new client contacts the Zookeeper quorum (a separate cluster of Zookeeper nodes) first to find a particular row key. It does so by retrieving the server name (i.e. host name) that hosts the -ROOT- region from Zookeeper.

With that information it can query that server to get the server that hosts the .META. table. Both of these two details are cached and only looked up once.

Lastly it can query the .META. server and retrieve the server that has the row the client is looking for.

http://hbase.apache.org/book/quickstart.html

http://hbase.apache.org/book/zookeeper.html

A distributed HBase depends on a running ZooKeeper cluster. All participating nodes and clients need to be able to access the running ZooKeeper ensemble. HBase by default manages a ZooKeeper “cluster” for you.

http://hbase.apache.org/book/datamodel.html

http://hbase.apache.org/book/data_model_operations.html

http://hbase.apache.org/book/schema.html

http://hbase.apache.org/book/mapreduce.html

http://hbase.apache.org/book/architecture.html

http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html

http://ria101.wordpress.com/2010/02/24/hbase-vs-cassandra-why-we-moved/ :

” I will write about why I think we will see a seismic shift from SQL to NOSQL over the coming years, which will be just as important as the move to cloud computing. ”

“HBase and its required supporting systems are derived from what is known of the original Google BigTable and Google File System designs (as known from the Google File System paper Google published in 2003, and the BigTable paper published in 2006″

” Cassandra on the other hand is a recent open source fork of a standalone database system initially coded by Facebook, which while implementing the BigTable data model, uses a system inspired by Amazon’s Dynamo for storing data (in fact much of the initial development work on Cassandra was performed by two Dynamo engineers recruited to Facebook from Amazon).”

HBase being more suitable for data warehousing, and large scale data processing and analysis (for example, such as that involved when indexing the Web)

Cassandra being more suitable for real time transaction processing and the serving of interactive data.

where platforms are perceived as similar, people tend to aggregate around the platform that is going to offer the best supporting ecosystem in the long term

When starting with HBase, my impression then was that it had the greatest community momentum behind it, but I now believe that Cassandra is coming through much stronger.

HBase vs Cassandra: NoSQL Battle!” link more , or HBASE vs Big Table

ccelerating momentum behind Cassandra. You might also take note of the big names coming on board, such as Twitter, where they plan broad usage (see here).

“CAP Theorem, and was developed by Professor Eric Brewer, Co-founder and Chief Scientist of Inktomi.

The theorem states, that a distributed (or “shared data”) system design, can offer at most two out of three desirable properties –Consistency, Availability and tolerance to network Partitions.

Very basically, “consistency” means that if someone writes a value to a database, thereafter other users will immediately be able to read the same value back,

“availability” means that if some number of nodes fail in your cluster the distributed system can remain operational, and

“tolerance to partitions” means that if the nodes in your cluster are divided into two groups that can no longer communicate by a network failure, again the system remains operational.

a complete HBase solution is really comprised of several parts: you have the database process itself, which may run in several modes, a properly configured and operational hadoop HDFS distributed file system setup, and a Zookeeper system to coordinate the different HBase processes

HBase in pseudo distributed mode on a single server is difficult – so difficult in fact that I did my best to write a guide that takes you past all the various gotchas in the minimum time (see http://ria101.wordpress.com/2010/01/28/setup-hbase-in-pseudo-distributed-mode-and-connect-java-client/ if you wish to try it). As you will see from that guide, getting HBase up and running in this mode actually involves setting up two different system systems manually: first hadoop HDFS, then HBase itself.”