How it Works

Datera Elastic Data Fabric is the application-driven, intelligent storage that simplifies and automates the provisioning of data infrastructure, giving you flexibility to add capacity and performance when and where it’s needed. Datera allows enterprises and service providers alike to cost-effectively build large-scale, fully automated private and public clouds for faster application delivery and shorter time to market.

Multiple Generations and Types of Hardware

 

Data Plane: High Performance and Low Latency

Each Datera node is built on a direct-on-disk, log-structured key/value store that was designed from ground up to support the raw power of next-generation flash technologies on any standard x86 server. This key/value store forms multiple storage media tiers: high performance flash (NVMe), capacity flash (SAS/SATA) or capacity drives (HDDs), variously into all-flash or hybrid nodes.

To deliver low write latencies, each Datera key/value store incorporates persistent memory (NVDIMMs, NVRAM, and soon 3D Xpoint) that absorbs writes “instantly,” coalesces them and then de-stages them sequentially to the flash tier. On hybrid nodes, the flash tier itself serves as a write-back cache, further coalescing writes, and streaming them down to the capacity tier.

Datera pools the various data tiers in mix-and-match nodes via a novel lockless scale-out coherency protocol. This protocol can multi-path data across any number of nodes in the data center, thereby aggregating massive amounts of read/write bandwidth to support the scale and speed needed for modern real-time data processing applications.

The unique Datera design combines the best of both worlds: the operational flexibility of software-defined scale-out storage on commodity x86 servers, and the excellent performance and latencies of traditional storage arrays. On hybrid nodes, it even does it at the cost of capacity drives (HDDs).

The Datera data plane has the full enterprise feature set expected today:

  • Thin provisioning and live volume resizing: efficient under-provisioning of data volumes.
  • Snapshots and clones: “instant” space-efficient snapshots and clones support the exponentially growing amount of data in today’s modern data center.
  • Data reduction: efficient inline compression today, and inline dedup shortly.
  • Replication: 1-5 data replicas per volume to support a wide range of applications, from traditional ones that rely on storage to provide redundancy, to cloud native ones that may have their own built-in replication strategy.
  • Quality of Service: on bandwidth and IOPS for dynamic provisioning of performance and workload isolation, to reliably meet specific workload needs.
  • Data-at-rest encryption: to keep all assets safe on all storage devices.
  • Role-based multi-tenancy: to operationalize, consolidate and isolate a spectrum of workloads across a range of tenants and use cases, data is auto-placed and auto-pathed under full policy control.

Datera is a major contributor to the Linux kernel, with a keen focus on IO optimizations across the data path to drive low latency for a broad range of standard x86 servers. For instance, Datera authored the industry-standard Linux-IO (LIO) Target Driver stack that adds a full suite of block storage protocols to Linux, including iSCSI, iSER, Fibre Channel, FCoE, and soon NVMe-OF.

This allows Datera to connect through its own industry-standard block storage protocols to any traditional or cloud native workloads, with zero host-side footprint.

Control Plane: Intelligent Scalability

Datera has decoupled scalability from performance with its a fully symmetrically distributed control plane. Its simplified deployment and operational model, in conjunction with standards-based data protocols, eliminates the need of any external control or data elements, such as separate metadata or controller nodes, external protocol gateways, or host-side proxies.

The Datera data infrastructure can seamlessly scale up or down with the addition or removal of nodes. Deployments can start small and inexpensively, and gradually grow with evolving needs by adding mix-and-match nodes , including rolling technology upgrades.

Datera continuously re-balances all system elements and automatically migrates all data, to guarantee continuous application access with zero downtime.

 

Management Plane: Intelligent Automation

At the core of the Datera system is a policy engine that automatically optimizes all elements of the data infrastructure by continuously adapting to changing application requirements, hardware capabilities and operational constraints in the data center.

  • Application preferences are codified as intents in templates (or profiles or manifests), which the policy engine translates into dynamic policy sets (performance, cost, QoS, availability, replication, snapshots, security, access, authentication, etc.), and adapts them according to their current tenancy and roles at any given point in time.
  • The diverse capabilities of each node are abstracted and dynamically exported through idempotent service agents that run on each node.
  • The policy engine is network-aware to learn the changing operational constraints in the datacenter, such as network topology, availability zones, power domains, latency cliffs, migrating application endpoints, etc.

Based on its understanding of applications, node capabilities and data center constraints, the Datera policy engine continuously optimizes all elements in the data infrastructure, across all applications, tenants and environments. As intents are invariant, they bring seamless portability, scalability and composability to stateful applications across data centers and clouds.

The powerful Datera policy engine can easily fit data to applications across the data center, rather than forcing it into rigid handcrafted application silos, always place it on the most cost-efficient nodes (incl. flash, hybrid or any combination), and coherently manage its full lifecycle. The result is ground-breaking scalable automation and price/performance elasticity that brings the data agility that is key to any successful cloud project.

Every aspect of the system is fully programmable through a rich RESTful API. Based on its REST API, Datera provides multiple language API bindings, including a python-base SDK and a powerful command line interface (CLI), to simplify integration and accelerate adoption.

Datera seamlessly integrates with all relevant workload orchestration frameworks: VMWare, Openstack, Docker, Kubernetes, Mesosphere/DC-OS and Cloudstack.

The Datera data infrastructure dynamically composes all its resources on the fly to deliver a rich spectrum of application needs: performance, quality of service, security, availability, and reliability. As businesses adapt to evolving needs, Datera provides a single data infrastructure that can support them all.