1

Just to give a bit of background - I have a 3 node GlusterFS cluster running, and I made sure to test it works by mounting a few directories on some servers. I also tested a mount on the Kubernetes cluster (currently just 1 node), to make sure it's not an issue there, and that the gluster-client package is installed.

Been using Rancher as the management overlay for Kubernetes. I manually created a Persistent Volume and selected Gluster plugin. For the endpoint, I had to create custom Endpoints and Service objects.

Here's the error I have been getting:

Events:
  Type     Reason       Age                  From                  Message
  ----     ------       ----                 ----                  -------
  Warning  FailedMount  7m8s (x34 over 86m)  kubelet, ssk8smstd01  Unable to mount volumes for pod "hello-7f66d8f588-whxwz_default(6a45cd5e-7259-11e9-ab24-005056aa5a40)": timeout expired waiting for volumes to attach or mount for pod "default"/"hello-7f66d8f588-whxwz". list of unmounted volumes=[pvc1]. list of unattached volumes=[pvc1 default-token-n7tlv]
  Warning  FailedMount  3m2s (x44 over 86m)  kubelet, ssk8smstd01  (combined from similar events): MountVolume.SetUp failed for volume "gluster1" : mount failed: mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t glusterfs -o backup-volfile-servers=ip1:ip2:ip3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster1/hello-7f66d8f588-whxwz-glusterfs.log,log-level=ERROR ip1:gv0 /var/lib/kubelet/pods/6a45cd5e-7259-11e9-ab24-005056aa5a40/volumes/kubernetes.io~glusterfs/gluster1
Output: Mount failed. Please check the log file for more details.

 the following error information was pulled from the glusterfs log to help diagnose this issue: 
[2019-05-09 14:19:04.005668] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-05-09 14:19:04.005900] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x5589853fc38e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x5589853f6dc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7fa9c305268b] ) 0-graph: invalid argument: graph [Invalid argument]

Has anyone seen something similar to this or have any ideas?

2

I found the issue, after having a chat with the tech guys at Rancher.

The issue is that Hyperkube (wrapper for K8s binaries), comes pre-packaged with quite an old version of GlusterFS, so I ended up having to downgrade mine to version 5, which fixed it.

|improve this answer|||||
0

This might be helpful: https://github.com/gluster/glusterfs/issues/658

In brief, you should use the same version of GlusterFS on clients and servers.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.