How to collect JVM heap dump in Kubernetes

Having trouble collecting JVM heap dump from Kubernetes?

You need to use PV(Persistent Volume) to keep data since volumes are ephemeral in Kubernetes.

If you are using EKS, EFS is a great option since EBS does not support ReadWriteMany which is essential for scalable system. (multiple pods should be able to write to the same storage)

First, EFS should be created. AWS policy should be configured beforehand.

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/docs/iam-policy-example.json

AWS CLI should be installed to use the command below.
Document

aws iam create-policy \
    --policy-name EKS_EFS_CSI_Driver_Policy \
    --policy-document file://iam-policy-example.json

EKSCTL should be installed to use the command below.
Document

eksctl create iamserviceaccount \
     --name efs-csi-controller-sa \
     --namespace kube-system \
     --cluster {Your EKS Cluster Name Here} \
     --attach-policy-arn {AWS Policy ARN created above} \
     --approve \
     --override-existing-serviceaccounts \
     --region {Your region}

vpc_id=$(aws eks describe-cluster \
    --name {Your EKS Cluster Name Here} \
    --query "cluster.resourcesVpcConfig.vpcId" \
    --output text)

cidr_range=$(aws ec2 describe-vpcs \
    --vpc-ids $vpc_id \
    --query "Vpcs[].CidrBlock" \
    --output text)

security_group_id=$(aws ec2 create-security-group \
    --group-name {Your new security group for EFS} \
    --description "My EFS security group" \
    --vpc-id $vpc_id \
    --output text)

aws ec2 authorize-security-group-ingress \
    --group-id $security_group_id \
    --protocol tcp \
    --port 2049 \
    --cidr $cidr_range

file_system_id=$(aws efs create-file-system \
    --region {Your region} \
    --performance-mode generalPurpose \
    --query 'FileSystemId' \
    --output text)

aws efs create-mount-target \
    --file-system-id $file_system_id \
    --subnet-id {Your EKS subnets} \
    --security-groups $security_group_id

Remember to replace with your own EKS name and ARN. (Also, remove curly brackets)

Last command should be repeated for every subnets of your EKS.

The command below will give every subnets of your VPC. (Do not replace curly brackets here)

Only select subnets of your EKS, in most cases only private subnets, and use it for the command above.

aws ec2 describe-subnets \
    --filters "Name=vpc-id,Values=$vpc_id" \
    --query 'Subnets[*].{SubnetId: SubnetId,AvailabilityZone: AvailabilityZone,CidrBlock: CidrBlock}' \
    --output table

Next, it’s time for your EKS.

EFS driver should be installed using helm chart.

helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver/

helm repo update aws-efs-csi-driver

helm upgrade --install aws-efs-csi-driver --namespace kube-system aws-efs-csi-driver/aws-efs-csi-driver \
    --set image.repository=602401143452.dkr.ecr.region-code.amazonaws.com/eks/aws-efs-csi-driver \
    --set controller.serviceAccount.create=false \
    --set controller.serviceAccount.name=efs-csi-controller-sa

The image repository should be changed by your region of your resources.
Document

Now, SC(Storage Class), and PVC(Persistent Volume Claim) should be configured.

For storage class, sc.yaml.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com

For Persistent Volume Claim, pvc.yaml.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
  namespace: Your-Namespace
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

Now, add the volume configuration on your manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: your-app-ns
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: your-app
  template:
    metadata:
      labels:
        app: your-app
    spec:
      containers:
        - image: your-image
          imagePullPolicy: Always
          name: your-container
          volumeMounts:
            - mountPath: /heap_dump
              name: efs-claim

The volumeMounts options is the option you need here.

Please make sure that name of volumeMounts matches with PVC metadata name that you configured above.

The mountPath options indicates the path of your container that will be connected to your EFS.

The heap dump path of your application should be matched with the mount path to save your heap dump file on your EFS.

"-XX:HeapDumpPath=/heap_dump/applicationHeapDump.hprof"

The option should be added on your java application. (The name of the file should be changed for the possibility of duplication. I recommend using combination of date and time as postfix.)

To access your heap dump, you need another server, such as EC2, to access to EFS.

sudo mount -t efs -o tls {Your EFS ID}:/ /mnt

Please replace, including curly brackets, with your EFS ID.

You can replace the mount path above(/mnt) with your choice

You will find out that it is mounted on your EC2 with loopback address.

If you want to test, connect to one of your kubernetes pod and make a file under the mount path.

In this case, /heap_dump

kubectl exec -it {your-pod-name} -n {your-pod-namespace} /bin/sh
cd /heap_dump
touch helloworld.sh

Use the command above to connect your kubernetes pod and make file for test.

You will find out helloworld.sh file on your EFS mount path, “/mnt” in this example.

This article was created by Crocoder7. It is not to be copied without permission.

References

• Github Document
• AWS Document