Stateful Applications with Kubernetes 1.3 Using Datera Elastic Data Fabric

As container adoption evolves beyond stateless application to more stateful workloads such as MySQL, Kafka, Cassandra and Couchbase, the support for persistent volumes in Kubernetes becomes foundational. Datera has integrated with Kubernetes to simplify persistent storage provisioning and consumption.

Based on first principles of containers, Datera allows application resource provisioning to be decoupled from the underlying physical infrastructure. This enables clean contracts (aka, no dependency, or direct knowledge of the underlying physical infrastructure), declarative formats, and eventually portability for stateful applications. 

While Kubernetes allows for great flexibility to define the underlying application infrastructure through yml configurations, Datera allows for that configuration to be passed to the storage infrastructure to provide persistence. Through the notion of Datera AppTemplates, in a Kubernetes environment, stateful applications can be automated to scale.

Deploying Persistent Storage

Persistent storage is defined using the Kubernetes PersistentVolume subsystem. PersistentVolumes are volume plugins and define volumes that live independently of the lifecycle of the pod that is using it. They are implemented as NFS, iSCSI, or by cloud provider specific storage system. Datera has developed a volume plugin for PersistentVolumes that can provision iSCSI block storage on the Datera Data Fabric for Kubernetes pods.

The Datera volume plugin gets invoked by kubelets on minion nodes and relays the calls to the Datera Data Fabric over its REST API. Below is a sample deployment of a PersistentVolume with the Datera plugin:

img-blog1

This manifest defines a PersistentVolume of 100 GB to be provisioned in the Datera Data Fabric, should a pod request the persistence storage.

img-blog2

Configuration

The Datera PersistenceVolume plugin is installed on all minion nodes. When a pod lands on a minion node with a valid claim bound to the persistent storage provisioned earlier, the Datera plugin forwards the request to create the volume on the Datera Data Fabric. All the options that are specified in the PersistentVolume manifest are sent to the plugin upon the provisioning request.

Once a volume is provisioned in the Datera Data Fabric, volumes are presented as iSCSI block device to the minion node, and kubelet mounts this device for the containers (in the pod) to access it.

Using Persistent Storage

Kubernetes PersistentVolumes are used along with a pod using PersistentVolume Claims. Once a claim is defined, it is bound to a PersistentVolume matching the claim’s specification. A typical claim for the PersistentVolume defined above would look like below:

img-blog3

When this claim is defined and it is bound to a PersistentVolume, resources can be used with the pod specification:

img-blog4

A pod can use a PersistentVolume Claim like below:

img-blog5

The result is a pod using a PersistentVolume Claim as a volume. It in-turn sends the request to the Datera volume plugin to provision storage in the Datera Data Fabric.

img-blog6

The persistent volume is presented as iSCSI device at minion node (tlx243 in this case):

img-blog7

Containers running in the pod see this device mounted at /data as specified in the manifest:

img-blog8
Using Pet Sets

Typically, pods are treated as stateless units, so if one of them is unhealthy or gets superseded, Kubernetes just disposes it. In contrast, a PetSet is a group of stateful pods that has a stronger notion of identity. The goal of a PetSet is to decouple this dependency by assigning identities to individual instances of an application that are not anchored to the underlying physical infrastructure.

A PetSet requires {0..n-1} Pets. Each Pet has a deterministic name, PetSetName-Ordinal, and a unique identity. Each Pet has at most one pod, and each PetSet has at most one Pet with a given identity. A PetSet ensures that a specified number of “pets” with unique identities are running at any given time. The identity of a Pet is comprised of:

  • a stable hostname, available in DNS
  • an ordinal index
  • stable storage: linked to the ordinal & hostname

A typical PetSet definition using a PersistentVolume Claim looks like below:

img-blog9

We have the following PersistentVolume Claims available:

img-blog10

When this PetSet is provisioned, two pods get instantiated:

img-blog11

Here is how the PetSet test-petset instantiated earlier looks like:

img-blog12

Once a PetSet is instantiated, such as test-petset below, upon increasing the number of replicas (i.e. the number of pods started with that PetSet), more pods get instantiated and more PersistentVolume Claims get bound to new pods:

img-blog13

Now the PetSet is running 3 pods after patch application.

When the above PetSet definition is patched to have one more replica, it introduces one more pod in the system. This in turn results in one more volume getting provisioned on the Datera Data Fabric. So volumes get dynamically provisioned and attached to a pod upon the PetSet scaling up.

To support the notion of durability and consistency, if a pod moves from one minion to another, volumes do get attached (mounted) to the new minion node and detached (unmounted) from the old minion to maintain persistent access to the data.

Conclusion

This demonstrates Kubernetes with Pet Sets orchestrating stateful and stateless workloads. While the Kubernetes community is working on expanding the FlexVolume framework’s capabilities, we are excited that this solution makes it possible for Kubernetes to be run more widely in the datacenters.

Shaliesh Mittal, Software Architect & Ashok Rajagopalan, Sr. Director, Products