GlusterFS problems

[Rick Schneider]

Hello,

I'm experiencing an issue with GlusterFS. Some of the files within the gluster file system are not visible to the clients. For example this is a subdirectory mounted on one of the gluster clients. It appears empty from the client side:
[root@cms wsi-tickers]# ls -l
total 0

However, when I log into any of the gluster nodes, I can see that there are files in the directory:
[root@nfs1 wsi-tickers]# ls -l
total 20
-rw-r--r-- 2 rschneider rschneider 4 Jan 17 10:22 abq_current_temp.txt
-rw-r--r-- 2 rschneider rschneider 6 Jan 17 10:22 temperature.txt
-rw-r--r-- 2 rschneider rschneider 1677 Jan 17 10:22 ticker_morning.txt
-rw-r--r-- 1 rschneider rschneider 1478 Jan 17 10:22 ticker_weather.txt
-rw-r--r-- 2 rschneider rschneider 1992 Jan 17 10:22 Today.txt

If, from the client side, I specify a file directly, I can see that it is there:
[root@cms wsi-tickers]# ls -l Today.txt
-rw-r--r-- 1 rschneider rschneider 1992 Jan 17 10:22 Today.txt

So it appears to be a problem with Gluster metadata, which I can confirm by issuing this command:
[root@nfs1 wsi-tickers]# gluster volume heal gv_html info
Brick gluster1:/var/gluster/gv_html/vol
<gfid:ff102ac6-48f4-4f35-8260-9bfa84a89efe>
<gfid:ea859f31-7974-4a08-89fb-2803d7af163f>
<gfid:a0e911e2-81dc-4287-bb2e-4809c81c3a9b>
<gfid:a836c3ab-29f0-4b5b-a498-0779e5348f77>
............................

So I try to issue a heal command:
[root@nfs1 wsi-tickers]# gluster volume heal gv_html full
Launching heal operation to perform full self heal on volume gv_html has been unsuccessful:
Commit failed on nfs2. Please check log file for details.

So I check the log:
[root@nfs1 wsi-tickers]# tail /var/log/glusterfs/glfsheal-gv_html.log

And what I discover is there are a bunch of entries like this:
The message "W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-gv_html-client-0: remote operation failed. Path: <gfid:48029792-a4c7-48b3-a062-2bf6291713b5> (48029792-a4c7-48b3-a062-2bf6291713b5) [No such file or directory]"

Does anyone here have an idea of what I need to do to resolve this?

Thanks in advance.

 

[wizardfromoz]

G'day Rick and welcome to linux.org

I'm moving this Thread to General Server where it may get you the audience you need.

Good luck

Chris Turner
wizardfromoz