Exploring Different Storage Options at Exoscale

Choosing the right storage is critical for cloud-native applications and scaling your infrastructure. In this blog, we’ll break down Exoscale’s storage options to help you find the best fit for your Kubernetes and Compute workloads.

If you’re short on time, you can skip ahead to the overview table at the end for a summary of the basic storage options.

Overview of Storage Options

We begin with a broad overview: these storage options are versatile and cater to a wide range of setups, from traditional compute instances to modern containerized Kubernetes environments.

Local NVMe Storage
Directly attached to the physical host where a compute instance runs, offering ultra-low latency and high IOPS performance. At Exoscale, every instance comes with at least 10 GiB of local storage, used for the operating system.

Block Storage
Designed as a reliable and scalable solution for persistent storage. It is externally attached to compute instances and operates independently of the instance lifecycle. Additionally, two replicas are always stored across the chosen zone for resiliency. While Block Storage may not meet the needs of databases with strict requirements for remote-attached NVMe storage, it provides reliable performance for most workloads and common databases while keeping flexibility.

Object Storage
Ideal for unstructured data such as backups and media files, Exoscale SOS offers scalability and cost-effectiveness.
One great aspect of Exoscale SOS is that it supports object versioning, object lock, and legal hold, as well as conditional writes. These features help protect your data against ransomware, accidental deletion, and ensure compliance with legal requirements. However, since S3 is an HTTP API, it is not suitable for database backends requiring high amounts of IOPS.

Exoscale Managed Databases
Fully managed, high-availability solution for databases, eliminating the need for manual intervention in managing storage, backups, and scaling.

Unlike self-managed installations, the Exoscale Managed Database Service takes care of all aspects of database administration, including automated backups, replication, and failover.

Choosing the Right Storage for Kubernetes Workloads

When running Kubernetes workloads, storage needs vary based on the nature of the application. Below, we discuss scenarios where each storage type excels and any limitations to consider.

Local NVMe Storage in Kubernetes

Local NVMe Storage provides unmatched performance for high-throughput applications. It is directly attached to the physical host running your virtual machine.

Advantages

Exceptional performance; suitable for temporary data or caching.

Limitations

• Kubernetes node cycling: Because nodes need to be updated or replaced from time to time, data on the local disk won’t survive node changes if not replicated or backed up elsewhere.
• No built-in durability: Local NVMe storage does not provide infrastructure-level durability. While RAID is used to mitigate hardware failures, it does not protect against VM termination or broader system failures. If you need data protection or high availability, your application or database must replicate data.

Local storage requires careful orchestration and is lost if the VM is terminated. It is best suited for temporary or caching volumes in Kubernetes environments. For most Kubernetes-native workloads, managed databases or block storage offer a more streamlined approach.

Another use case is distributed databases like Yugabyte, CockroachDB, or FoundationDB, which can leverage local NVMe performance while replicating data across nodes for durability.

Block Storage in Kubernetes

Block Storage is a natural fit for Kubernetes, especially for applications requiring persistent storage across node cycles. It supports scenarios where pods are rescheduled or nodes are replaced, ensuring data remains accessible.

Advantages

• Seamless integration with Kubernetes via PersistentVolumeClaims (PVCs)
• Data durability across node lifecycles
• Ideal for stateful applications like content management systems (CMS) or application logs or simple databases

Limitations

• Access-mode is ReadWriteOnce, meaning only a single node can mount a given volume at a time.
• Some advanced databases that require low-latency attached storage (e.g., local NVMe) may not perform optimally on remote block storage.
• While using StatefulSets with block storage is common, you should still plan carefully for backup, restore, and failover.

Generally speaking, having stateful workloads is more complex than managing stateless ones, so be prepared for additional planning.

Ideal for use cases like WordPress sites, application logs, or relational databases that require persistent storage.
Our documentation has simple examples.

For applications with high IOPS demands or those that require distributed database capabilities, relying on block storage and managing statefulsets yourself may not be the most efficient or reliable solution.
In such cases, Exoscale’s Managed Database Service offers a better alternative. Allowing you to potentially keep your Kubernetes cluster completely stateless while benefiting from high performance and ease of management.

For mounting Volumes on multiple nodes or Pods at the same time (ReadWriteMany), you could either use Object Storage (see below) or install an NFS server on top of a local- or block storage volume. Having a managed service for that is in our roadmap.

Exoscale Managed Database Service (DBaaS)

Managed Databases offer high availability, automated backups, scaling, and failover, simplifying database management. Unlike self-managed installations, the Exoscale Managed Database Service takes care of all aspects of database administration, including automated backups, replication, and failover.

Key Benefits

• Fully Managed: Exoscale handles all database operations, including provisioning, scaling, patching, and maintenance.
• High Availability: With automatic failover and replication, your database is always available, even in the event of a failure.
• Performance: Optimized for high-performance use cases, ensuring your database runs smoothly without the complexities of managing persistent storage manually.
• Multi-Zone: Use read replicas to also create replicas in other Exoscale zones. Increasing availability and performance.
• Scale any time: Change the plan any time - either up or down.

Use Cases

• High-availability database needs with automated backup and scaling
• Databases with strict performance requirements (e.g., transactional workloads)
• Applications that require ease of management and operational efficiency without the need to manage underlying storage or infrastructure.

Make sure to check out our detailed guide on deploying a fully stateless WordPress instance using Terraform and SKS. In this setup, static files are seamlessly stored in Object Storage, while the MySQL database connects effortlessly to a Managed Database with no setup required—just provide its URL and you’re ready to go!

You can find all supported databases here.

Object Storage in Kubernetes

Since Object Storage operates via an API, it’s best used directly within applications. However, Kubernetes can integrate it using AWS Mountpoint, enabling its use as Kubernetes Volumes.

Since Object Storage operates via an API, it’s best used directly within applications. Kubernetes can also integrate it via AWS Mountpoint to make buckets accessible as volumes.

Object Storage can support high read IOPS and scales well for read-heavy workloads. However, it does not support in-place updates: writing changes requires uploading a new object. For many applications optimized for the S3 protocol (for example, storing large media files or analytics datasets), this is not a problem and can deliver performance at scale.

Still, Object Storage must not be used for traditional databases requiring low-latency writes or frequent small updates.

Using Object Storage with AWS Mountpoint

Mountpoint is an open source S3 client by AWS that you can use to access the objects in your bucket like files on a filesystem and as Kubernetes Volumes. The ideal use case is for read-heavy applications that require large amounts of data (e.g. data lakes, machine learning training data, ETL).

First create a secret with a key which is allowed to access your bucket:

kubectl create secret generic aws-secret \
    --namespace kube-system \
    --from-literal "key_id=YOURKEY" \
    --from-literal "access_key=YOURSECRET"

Then install mountpoint (documentation):

kubectl apply -k "github.com/awslabs/mountpoint-s3-csi-driver/deploy/kubernetes/overlays/stable/"

If you apply the API key secret after installing it, you will have to restart all Mountpoint Pods.

Now you can start using it.
Here is an example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3-pv
spec:
  capacity:
    storage: 1200Gi # ignored, required
  accessModes:
    - ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
  mountOptions:
    - allow-delete
    - endpoint-url https://sos-de-fra-1.exo.io # Adapt the zone
    - prefix / # You can also choose a subfolder of the bucket
  csi:
    driver: s3.csi.aws.com # required
    volumeHandle: s3-csi-driver-volume
    volumeAttributes:
      bucketName: YOURBUCKETNAME # Adapt the bucket
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3-claim
spec:
  accessModes:
    - ReadWriteMany # supported options: ReadWriteMany / ReadOnlyMany
  storageClassName: "" # required for static provisioning
  resources:
    requests:
      storage: 1200Gi # ignored, required
  volumeName: s3-pv
---
apiVersion: v1
kind: Pod
metadata:
  name: s3-app
spec:
  containers:
    - name: app
      image: centos
      command: ["/bin/sh"]
      args: ["-c", "echo 'Hello from the container!' >> /data/$(date -u).txt; tail -f /dev/null"]
      volumeMounts:
        - name: persistent-storage
          mountPath: /data
  volumes:
    - name: persistent-storage

After deployment you should see that a folder date was created with a filename with the current date!

As its ReadWriteMany you can also mount the volume on multiple nodes and pods. Just remember that its not suitable for high-IOPS applications.

Guidelines for Kubernetes Storage Selection

Conclusion

Whether you’re looking for ultra-fast Local NVMe Storage, scalable Block Storage, reliable Object Storage, or the simplicity of Managed Databases, Exoscale has the right solution for you.

Need help choosing the best option for your workloads? Feel free to reach out to our sales team at sales@exoscale.com or contact me directly at denis.arnst@exoscale.com — I’m happy to assist!