How to use ephemeral disk from EC2 instance in EKS

EKS could mount EBS as Persistent Volumes in EKS cluster, we could also declear PersistentVolumnClaim to provision PV dynamically , this is great and we could treat EBS as our EKS disk pool to claim almost unlimited disk inside kubernetes pod. EBS also have different volume types, each volume type could support different type of IOPS. some EC2 instance could also have Instance Store Volumes, these volumes are in the same physical server as the ec2 instance, so they usually have better performance then EBS, the draw back of Instance Store Volumes is that when EC2 instance get rebooted or shutdown, ec2 instnace may move to other host machine and the Instance Store Volumes will be lost. if we are ok about this draw back, the Instance Store Volumes could give us best IO and no additional charge of EBS.

I have a use case which Instance Store Volumes fits us very well, I will need to launch a search server in a pod, the search server will first download the indexed file into local disk and servie online query, this is very IO sensitive service, since we could place the index file into a safe place, e.g: s3 and download it into local disk, I could use Instance Store Volumes to be mount into my pod to keep the index copy and the searchd could also access Instance Store Volumes to do real time query with the best IO performance. below is the steps how I provision the Instance Store Volumes and use it in my pod. Different EC2 instance will have different Instance Store Volumes, I will use r3.2xlarge as an example

• Create an ASG Launch Configuration
  I've already described how to use ASG to manage the worker nodes dynamically , when we create an ASG Launch Configuration , in the userdata section we need some command to config the ec2 nodes

sbin/mkfs -t ext4 /dev/xvdb
/bin/mkdir /media/ephemeral0
/bin/mount /dev/xvdb /media/ephemeral0
/bin/chown root:root -R /media/ephemeral0

these commands should be placed above /etc/eks/bootstrap.sh

• Create ASG based on ASG Launch Configuration see this article for details
• when the new ec2 instance managed by the defined ASG join EKS cluster, we could already have the Instance Store Volumes ready to use

lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   20G  0 disk
└─xvda1 202:1    0   20G  0 part /
xvdb    202:16   0  150G  0 disk /media/ephemeral0

• use /media/ephemeral0 in yaml file
  we could declear hostPath type of volumes in kubernetes like below to mount /media/ephemeral0 in your pod container on /media/disk0

apiVersion: v1
kind: Pod
metadata:
  name: pv-recycler
  namespace: default
spec:
  restartPolicy: Never
  volumes:
  - name: vol
    hostPath:
      path: /media/ephemeral0
  containers:
  - name: pv-recycler
    image: "k8s.gcr.io/busybox"
    command: ["/bin/sh", "-c", "echo 'hello'"]
    volumeMounts:
    - name: vol
      mountPath: /media/disk0

now in the pod pv-recycler I could write any logic to use disk /media/disk0, it will use Instance Store Volumes under the hood and offer the best IO performance.