Percolator appears to be a transformative system.

Percolator, Dremel & Pregel is the next evolution for large data structure processing and analytics. It is better and faster than Hadoop. It may be cheaper.

Funny that just as Hadoop is creating a good ecosystem it may be replaced by a new technology.

You can say same for Hadoop and many systems.

Yet what is overkill today may be right in near future. Sooner than you think considering the exxplosion in data growth, and variety.

Google is now using Percolator, Dremel and Pregel.

There is a buzz about Hadoop's days being numbered. The king is dead, long live the king!

Percolator, Dremel & Pregel is simply an alternative to Hadoop - not a replacement.

Apache Giraph is an other implementation of Pregel.

http://giraph.apache.org/

Giraph follows the bulk-synchronous parallel model relative to graphs where vertices can send messages to other vertices during a given superstep. Checkpoints are initiated by the Giraph infrastructure at user-defined intervals and are used for automatic application restarts when any worker in the application fails.

Any worker in the application can act as the application coordinator and one will automatically take over if the current application coordinator fails.

GraphLab - http://graphlab.org/ - is a graph-based, high performance, distributed computation framework written in C++.

While GraphLab was originally developed for Machine Learning tasks, it has found great success at a broad range of other data-mining tasks; out-performing other abstractions by orders of magnitude.

GraphLab Features:

A unified multicore and distributed API: write once run efficiently in both shared and distributed memory systems

Tuned for performance: optimized C++ execution engine leverages extensive multi-threading and asynchronous IO

Scalable: GraphLab intelligently places data and computation using sophisticated new algorithms

HDFS Integration: Access your data directly from HDFS

Powerful Machine Learning Toolkits: Turn BigData into actionable knowledge with ease

Take a look at LevelDB - a fast and lightweight key/value database library.

It provides an ordered mapping from string keys to string values.

http://code.google.com/p/leveldb/

Tenzing is good news to improve speed of Hadoop ecosystem. The biggest performance opportunities for Hive currently lie in making the execution engine more CPU efficient.

Tenzing is a SQL Implementation on the MapReduce Framework. When compared with other similar solutions (Sawzall, Flume-Java, Pig, Hive„ HadoopDB), Tenzing’s advantage is low latency.

Tenzing is a query engine built on top of MapReduce for ad hoc analysis of data. Tenzing supports a mostly complete SQL implementation (with several extensions) combined with several key characteristics such as heterogeneity, high performance, scalability, reliability, metadata awareness, low latency, support for columnar storage and structured data, and easy extensibility.

Learn more: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf

http://www.vldb.org/pvldb/vol4/p1318-chattopadhyay.pdf

It appears they manipulated MapReduce to get better performance for SQL:

+Workerpools: These are essentially long running processes that do various parts of the MapReduce job (map task, reduce task, job coordinator, etc). Having a pool of running processes makes latencies lower than they would be if you had to launch a binary for each task in the job. This is certainly the case with JVM launches in Hadoop. Hadoop gets part of this done with reusable JVMs. The tradeoff, of course, is that fault isolation becomes a messier proposition.

+Streaming and In-Memory Chaining: Allows two MapReduce jobs to communicate without temping to disk (GFS). I wonder if this can be done easily with just some InputFormat/OutputFormat magic ... I suspect this is do-able with some thought. Memory-chaining allows a mapper and a reducer to be co-located in the same process. This is probably going to be a bit harder to do in Hadoop.

+Sort Avoidance: This feature allowed you to tell MapReduce to shuffle, but not sort. I've seen the need for this in many applications. Again, makes perfect sense for Hadoop also.

+Block Shuffle: For smaller rows, when sorting is not needed, the block shuffle reduces overheads in the shuffle phase. This is a performance opportunity opened up by sort avoidance.

NuoDB is a NewSQL cloud database. It looks and behaves like a traditional SQL database from the outside but under the covers it's a revolutionary database solution. It is a new class of database for a new class of datacenter.

NuoDB is the first database that is 100% SQL compliant, guarantees ACID transactions, and scales out elastically on cloud-based computing resources.

Download Beta 7: http://www.nuodb.com/

Highlights include:

Performance and scalability — Beta 7 now supports up to 50 nodes enabling the database to elastically scale to tens of thousands of transactions per second on Amazon EC2 or local commodity servers.

Product hardening — Beta 7 provides additional hardening and increased stability.

SQL — Beta 7 now supports SQL 92 with 99 extensions as well as incorporating several querying and indexing enhancements and enhances SQL Standards compliance to cover a majority of application requirements.

NuoDB Administrative Console — Functional enhancements to graphical console, which was introduced in Beta 6.

New language drivers — Incorporates new Ruby, PHP/PDO, and Perl drivers to address growing Web development needs. These drivers were developed by the NuoDB Github community.