3

Problem:

I have a git repository mounted via sshfs and cannot commit changes with the following error message:

fatal: cannot update the ref 'HEAD': unable to append to '.git/logs/HEAD': Permission denied

Note that I can

cp -a .git/logs/HEAD .git/logs/HEAD.bu
printf foo > .git/logs/HEAD
mv .git/logs/HEAD.bu .git/logs/HEAD

without a problem, but

printf foo >> .git/logs/HEAD

gives me the 'Permission denied' as well.

Question:

What do I need to change about my configuration to be able to commit from my local machine to the remote repository?

What I tried:

Given the above symptoms, I assume the issue lies with appending to a file. I found Git repository on SSHFS: unable to append to '.git/logs/HEAD': Invalid argument which refers to https://github.com/libfuse/sshfs/issues/82 suggesting the issue (note the slighty differerent error message) could be solved by mounting the remote file system with writeback_cache=no. The latter source quotes the man page referencing the following caveat/workaround:

CAVEATS / WORKAROUNDS
[...]
   O_APPEND
       When  writeback  caching is enabled, SSHFS cannot reliably support the O_APPEND open
       flag and thus signals an error on open.  To enable support for  unreliable  O_APPEND
       (which  may  overwrite  data if the file changes on the server at a bad time), mount
       the file system with -o unreliable_append.

However, this section is not in my man page:

sshfs -V
SSHFS version 3.7.0
FUSE library version 3.9.1
using FUSE kernel interface version 7.31
fusermount3 version: 3.9.1

I found that the writeback-cache feature that I tried to disable, was actually removed (after being disabled and re-enabled more than once before). So I guess I should be good but clearly there (still) is a problem.

A further complication that I should probably mention, is that my user name and ID on the remote system do not match the local one, so I need to used the idmap feature.

Here is the corresponding fstab entry:

<remote-user>@<remote-machine>: /mnt/ssh/<remote-machine>  sshfs  _netdev,user,idmap=user,allow_other  0 0

Also, my /etc/fuse.conf contains

user_allow_other

Background:

To avoid answer just telling me not to do this:

  • I know how git works.
  • I know I can clone the repository locally, commit there, and push to the remote one over ssh.

Why I don't do it? - Because I track code that can only be tested on the remote machine and I do want to test it before committing. So to some extent this is 'just' a convenience issue to avoid having to:

  1. Edit code on the local copy.
  2. Commit the changes to the local copy.
  3. Push to the remote copy.
  4. SSH to the remote machine (or switch terminals).
  5. Test the code on the remote machine.
  6. Checkout another branch (to allow force-pushing).
  7. End the SSH sessions (or switch [back] terminals).
  8. Edit the code.
  9. Amend the previous commit on the local copy.
  10. Force-push to the remote copy.
  11. SSH to the remote machine (or switch terminals).
  12. Checkout the force-pushed branch.
  13. Repeat steps 5 - 11 (seven steps!) until I'm happy.

Instead, I want to:

  1. SSH to the remote machine (or switch terminals).
  2. Edit code on the remote copy from the remote machine.
  3. Test the code on the remote machine.
  4. Edit code on the remote copy from the remote machine.
  5. Repeat steps 3 - 4 (two steps!) until I'm happy.
  6. End the SSH sessions (or switch [back] terminals).
  7. Commit the changes to the remote copy from the local machine.

Why don't I simply commit from the remote machine? - Because I want to sign my commits but can't entrust the remote machine with the private key. So the best alternative I could come up with is:

  1. SSH to the remote machine (or switch terminals).
  2. Edit code on the remote copy from the remote machine.
  3. Test the code on the remote machine.
  4. Edit code on the remote copy from the remote machine.
  5. Repeat steps 3 - 4 (two steps!) until I'm happy.
  6. Commit the changes to the remote copy from the remote machine.
  7. Checkout another branch (to allow force-pushing).
  8. End the SSH sessions (or switch [back] terminals).
  9. Pull from the remote copy.
  10. Amend (sign) the previous commit on the local copy.
  11. Force-push to the remote copy.
  12. SSH to the remote machine (or switch terminals).
  13. Checkout the force-pushed branch.

So on the one hand, I'd like to get rid of these extra steps (things get more complicated when adding feature branches as those need to be properly checked out on both copies and configured for proper tracking), on the other I want to understand why it doesn't 'just work'(tm).


Update:

Following up on a comment by @tukan, I reproduced the error with debug output:

  1. Mount remote with debug output:
 mount -o sshfs_debug MOUNTPOINT
SSHFS version 3.7.0
executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-2> <USER@SERVER> <-s> <sftp>
USER@SERVER's password:
Server version: 3
Extension: versions <2,3,4,5,6>
Extension: fsync@openssh.com <1>
Extension: posix-rename@openssh.com <1>
Extension: statvfs@openssh.com <2>
Extension: fstatvfs@openssh.com <2>
Extension: hardlink@openssh.com <1>
remote_uid = 0
  1. In a different terminal, access the mounted share:
cd MOUNTPOINT/DIR_WITH_WRITE_PERMISSIONS
[00002] LSTAT
  [00002]          ATTRS       45bytes (188ms)
  1. Verify regular writing works:
echo foo > foobar
[00003] LSTAT
  [00003]         STATUS       38bytes (46ms)
[00004] LSTAT
  [00004]         STATUS       38bytes (32ms)
[00005] LSTAT
  [00005]          ATTRS       45bytes (242ms)
[00006] OPENDIR
  [00006]         HANDLE       29bytes (31ms)
[00007] READDIR
[00008] READDIR
  [00007]           NAME      668bytes (58ms)
[00009] READDIR
[00010] READDIR
  [00008]           NAME      483bytes (65ms)
[00011] READDIR
[00012] READDIR
  [00009]         STATUS       37bytes (27ms)
  [00010]         STATUS       37bytes (27ms)
[00013] CLOSE
[00014] LSTAT
  [00011]         STATUS       37bytes (27ms)
  [00012]         STATUS       37bytes (27ms)
  [00013]         STATUS       28bytes (26ms)
  [00014]         STATUS       38bytes (31ms)
[00015] OPEN
[00016] LSTAT
  [00015]         HANDLE       29bytes (153ms)
  [00016]          ATTRS       45bytes (158ms)
[00017] FSTAT
  [00017]          ATTRS       45bytes (29ms)
[00018] WRITE
  [00018]         STATUS       28bytes (28ms)
[00019] CLOSE
  [00019]         STATUS       28bytes (28ms)
  1. Trigger error by attempting to append:
echo bar >> foobar
[00020] LSTAT
  [00020]         STATUS       38bytes (74ms)
[00021] LSTAT
  [00021]         STATUS       38bytes (57ms)
[00022] LSTAT
  [00022]          ATTRS       45bytes (52ms)
[00023] OPENDIR
  [00023]         HANDLE       29bytes (53ms)
[00024] READDIR
[00025] READDIR
  [00024]           NAME      668bytes (68ms)
[00026] READDIR
[00027] READDIR
  [00025]           NAME      597bytes (77ms)
[00028] READDIR
[00029] READDIR
  [00026]         STATUS       37bytes (47ms)
[00030] CLOSE
  [00027]         STATUS       37bytes (47ms)
[00031] OPEN
[00032] LSTAT
  [00028]         STATUS       37bytes (47ms)
  [00029]         STATUS       37bytes (47ms)
  [00030]         STATUS       28bytes (26ms)
  [00031]         STATUS       43bytes (28ms)
  [00032]          ATTRS       45bytes (29ms)
zsh: permission denied: foobar

Hope this helps to find the root cause of my problem.


Note: Based on the answer by @Devidas (and the lack of a solution even after a desparate attention-seeking bounty week), I cross-posted this to the corresponding GitHub issue.

  • Can you read the file? cat .git/logs/HEAD – tukan Jun 25 at 7:42
  • @tukan: Of course! As I wrote, I can also write to it (be it via mv, as shown above, or using nvim), As I wrote, > redirection works but >> fails. So I suspect this to be an issue with sshfs, not Git (thus no Git tag on this question). Git probably just tries to open that file in write-append mode causing the faulty permission error. – mschilli Jun 25 at 9:25
  • The problem is not writing, that works for you. The problem, as you specified, is with using >> (append) is that you need to read the contents first or at least check where is the EOF and then you append the contents. The question is what really happens with the git there (as you noted). Could you show us your sshfs debug log (-o sshfs_debug)? – tukan Jun 25 at 13:38
  • @tukan: Thx for following up. I updated the question to include a simpler reproduction (no Git) with -o sshfs_debug output. – mschilli Jun 26 at 8:04
  • You are welcome does the directory MOUNTPOINT/DIR_WITH_WRITE_PERMISSIONS has also read permissions? Could you show us the ls -la? Who is the owner? – tukan Jun 26 at 12:23
1
+100

such a big and detailed question. Lets solve this step by step. Error is "Permission denied"

Linux error code
EACCES          13      /* Permission denied */

when I searched for EACCES in sshfs repo I found only two instances in file [sshfs.c][1]

one is about file permission in local context. the one you demonstrated.

other is SSH_FX_PERMISSION_DENIED error from ssh permission denied.

From data I have I can say for almost certainty. that As you have permission in local machine

Cases

printf foo >> .git/logs/HEAD

reason this gives permission denied and not

printf foo > .git/logs/HEAD

either you don't have permission on remote machine or remote servers that don't support O_APPEND refer to issue 117

you can verify it using strace.

this is why part. How to solve depends on your reply. which case is it?

Do let me know so that I can help you further.

feel free to comment if you disagree.

| improve this answer | |
  • Thx for the response. The remote machine actually is an SFTP server is this case. I could try another one were I have full interactive SSH access that mounts the same filesystem (though it is explicitily not intended for data transfer) just to make sure it is server-related. How would I use strace verify 'it' (what exactly)? – mschilli Jun 26 at 7:42
  • Note that I updated the question based on feedback by @tukan. Maybe this helps you to further help me. ;-) – mschilli Jun 26 at 8:08
  • I did award this question the bounty as it was the only attempt of at least trying to help me. Seems my best hope is to get a over two-year old enhancement issue worked on, included in a new release, and that getting stable enough to propagate to my distro's repositories...thx anyways for the pointer. – mschilli Jun 30 at 5:49
  • @mschilli strace utility helps tracing system calls. So you will see exactly which syscall is failing and with which error code. If you compare strace for both cases you will see the difference. Hence solving the mistry why you could write to that file and why you can't append. – Devidas Jul 1 at 11:54
  • I did not ask 'What does strace do and how would it help to better understand the issue?' but 'How would I use it (in this particular case)?'. ;-) But thank you anyways for your help. At least you pointed me to the correct bug and it appears there is no proper solution nor a better workaround. So I'm going to accept this answer (and +1 obviously) as it looks like this is the best answer I could hope for and you made the effort to help. – mschilli Jul 1 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.