Rick Schneider
New Member
Hello,
I'm experiencing an issue with GlusterFS. Some of the files within the gluster file system are not visible to the clients. For example this is a subdirectory mounted on one of the gluster clients. It appears empty from the client side:
[root@cms wsi-tickers]# ls -l
total 0
However, when I log into any of the gluster nodes, I can see that there are files in the directory:
[root@nfs1 wsi-tickers]# ls -l
total 20
-rw-r--r-- 2 rschneider rschneider 4 Jan 17 10:22 abq_current_temp.txt
-rw-r--r-- 2 rschneider rschneider 6 Jan 17 10:22 temperature.txt
-rw-r--r-- 2 rschneider rschneider 1677 Jan 17 10:22 ticker_morning.txt
-rw-r--r-- 1 rschneider rschneider 1478 Jan 17 10:22 ticker_weather.txt
-rw-r--r-- 2 rschneider rschneider 1992 Jan 17 10:22 Today.txt
If, from the client side, I specify a file directly, I can see that it is there:
[root@cms wsi-tickers]# ls -l Today.txt
-rw-r--r-- 1 rschneider rschneider 1992 Jan 17 10:22 Today.txt
So it appears to be a problem with Gluster metadata, which I can confirm by issuing this command:
[root@nfs1 wsi-tickers]# gluster volume heal gv_html info
Brick gluster1:/var/gluster/gv_html/vol
<gfid:ff102ac6-48f4-4f35-8260-9bfa84a89efe>
<gfid:ea859f31-7974-4a08-89fb-2803d7af163f>
<gfid:a0e911e2-81dc-4287-bb2e-4809c81c3a9b>
<gfid:a836c3ab-29f0-4b5b-a498-0779e5348f77>
............................
So I try to issue a heal command:
[root@nfs1 wsi-tickers]# gluster volume heal gv_html full
Launching heal operation to perform full self heal on volume gv_html has been unsuccessful:
Commit failed on nfs2. Please check log file for details.
So I check the log:
[root@nfs1 wsi-tickers]# tail /var/log/glusterfs/glfsheal-gv_html.log
And what I discover is there are a bunch of entries like this:
The message "W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-gv_html-client-0: remote operation failed. Path: <gfid:48029792-a4c7-48b3-a062-2bf6291713b5> (48029792-a4c7-48b3-a062-2bf6291713b5) [No such file or directory]"
Does anyone here have an idea of what I need to do to resolve this?
Thanks in advance.
I'm experiencing an issue with GlusterFS. Some of the files within the gluster file system are not visible to the clients. For example this is a subdirectory mounted on one of the gluster clients. It appears empty from the client side:
[root@cms wsi-tickers]# ls -l
total 0
However, when I log into any of the gluster nodes, I can see that there are files in the directory:
[root@nfs1 wsi-tickers]# ls -l
total 20
-rw-r--r-- 2 rschneider rschneider 4 Jan 17 10:22 abq_current_temp.txt
-rw-r--r-- 2 rschneider rschneider 6 Jan 17 10:22 temperature.txt
-rw-r--r-- 2 rschneider rschneider 1677 Jan 17 10:22 ticker_morning.txt
-rw-r--r-- 1 rschneider rschneider 1478 Jan 17 10:22 ticker_weather.txt
-rw-r--r-- 2 rschneider rschneider 1992 Jan 17 10:22 Today.txt
If, from the client side, I specify a file directly, I can see that it is there:
[root@cms wsi-tickers]# ls -l Today.txt
-rw-r--r-- 1 rschneider rschneider 1992 Jan 17 10:22 Today.txt
So it appears to be a problem with Gluster metadata, which I can confirm by issuing this command:
[root@nfs1 wsi-tickers]# gluster volume heal gv_html info
Brick gluster1:/var/gluster/gv_html/vol
<gfid:ff102ac6-48f4-4f35-8260-9bfa84a89efe>
<gfid:ea859f31-7974-4a08-89fb-2803d7af163f>
<gfid:a0e911e2-81dc-4287-bb2e-4809c81c3a9b>
<gfid:a836c3ab-29f0-4b5b-a498-0779e5348f77>
............................
So I try to issue a heal command:
[root@nfs1 wsi-tickers]# gluster volume heal gv_html full
Launching heal operation to perform full self heal on volume gv_html has been unsuccessful:
Commit failed on nfs2. Please check log file for details.
So I check the log:
[root@nfs1 wsi-tickers]# tail /var/log/glusterfs/glfsheal-gv_html.log
And what I discover is there are a bunch of entries like this:
The message "W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-gv_html-client-0: remote operation failed. Path: <gfid:48029792-a4c7-48b3-a062-2bf6291713b5> (48029792-a4c7-48b3-a062-2bf6291713b5) [No such file or directory]"
Does anyone here have an idea of what I need to do to resolve this?
Thanks in advance.