The Internet of Things (IoT) promises to turn the current boom in business data into a supernova explosion. According to IBM, 2.5 quintillion bytes of data are created every single day!

But what do we do with it? It’s all going to have to be stored somewhere, be retrieved somehow and then mined for market information.

Just to make it more interesting for people in the database business, a vast proportion of this data is unstructured. In fact, the BBC reported that 75% of it is unstructured. This means that the relational database system (RDBMS) model is no longer the best one to handle these new super data-sets.

[T]he number of gadgets recording and transmitting data, from smartphones to intelligent fridges, industrial sensors to CCTV cameras, has also proliferated globally, leading to an explosion in the volume of data. These data sets are now so large and complex that we need new tools and approaches to make the most of them.

Matthew Wall, BBC, 4 March 2014.

The Open Source movement have responded with new technologies like NoSQL, Casandra, Hadoop and MongoDB. FoundationDB has been working on their own solution for the last 6 years focussing on the area of NoSQL. They released their latest product, Version 3.0, on 10th December 2014.

While solutions like NoSQL can provide faster writes to disc and faster data retrieval the perennial problem has been maintaining data consistency.

The problem runs like this: in these alternative, non-relational database models the data is distributed across multiple drives, often in different physical locations; and with multiple parts to the whole process, the chances of data errors and inconsistency are increased; ACID (Atomicity, Consistent, Isolated and Durable) is a framework for assessing how successful database performance is.

But business in 2015 is different now: many tech firms have to use distributed databases because they need to offer a service that’s always available and can scale rapidly as demand for the service can shoot up unexpectedly anywhere in the world.

Ultimately, these distributed platforms have to trade-off between meeting the demands on service and compliance with ACID principles like data consistency.

FoundationDB, set-up by Nick Lavezzo, Dave Rosenthal and Dave Schrerer in 2009, say that they are squaring the circle. Their new NoSQL platform, Key Value Store, released late last December, was launched into a frenzied market with a ferocious appetite for super-scalable db engines that can support ultra-heavy workloads but also demands consistency and reliability.

In their offering, Lavezzo and co. use the Key Value Store, as the objective is to make their platform ACID compliant but support distributed databases as well, which will boost read/write speed and scalability.

And FoundationDB are pretty confident about their new release: one of the reasons for this is its new architecture. Previously, the transaction engine that performs the writes was limited to one “master” machine. This created what co-founder Dave Rosenthal called a bottleneck in the process and prevented version 2.0 from fully meeting its performance goals. It achieved 400,000 random writes/second. Compare this to contemporary performance by Netflix’s Cassandra of ~1.1 million random writes per second and Google achieving 1 million writes per second.

The 3.0 release has a new transactional engine model: it compares transactions against each other; executes them and completes the rapid operation of all writes.

It has three components: the Proxies manage incoming transactions and execution queues. The Resolvers manage the isolation principle in ACID and keep track of the keys “modification history” and the Transaction Logs log and deal with incoming data mutations; this ensures data consistency compliance with ACID. The transaction log then reports a successful transaction to the client and then sends updates to the storage nodes.

A major selling point for version 3.0 is its linear scaling which, FoundationDB says, stacks up well against the competition in its ability to limit latency growth when workload is increased substantially.

In September FoundationDB also added a SQL engine to their platform so it can compete with NoSQL and the latest RDBMS. They’ve compared it to Google’s F1 (the database that supports Google Adwords) and  they’re eager to point out that both its SQL layer and Google F1 offer unprecedented levels of scalability.

This scalability and high performance comes with some increase in latency, it hasn’t been eliminated completely, but then again according to Ori Herrnstadt, it offers “almost unlimited bandwidth and scale”. One difference between FoundationDB’s SQL layer and F1  is that Google keeps F1 for Googlers but FoundationDB’s have their new layer on general release.

The future belongs to database systems that can manage unstructured data without increase in latency or decrease in consistency.  FoundationDB have a clear strategy in achieving the ultimate aim of frictionless scaling.