Scheduled service maintenance on November 22


On Friday, November 22, 2024, between 06:00 CET and 18:00 CET, GIN services will undergo planned maintenance. Extended service interruptions should be expected. We will try to keep downtimes to a minimum, but recommend that users avoid critical tasks, large data uploads, or DOI requests during this time.

We apologize for any inconvenience.

#70 Separate ssh key necessary for using git-annex?

Open
opened 4 months ago by rherikstad · 6 comments

This is probably more of a clarification on how to use git-annex directly. If I manually add a file under a gin repository using git-annex, i.e.

git annex add mylargefile.bin

I was unable to automatically sync it using

gin upload .

After some digging, it seems that the ssh-keys that gin manages under the hood are not forwarded to the underlying git-annex call. Only after explicitly adding new keys using ssh-keygen and then adding those to my ssh-agent did I manage to upload the files.

I should note that this is using a private G-node instance running in Docker on my local computer (purely for testing purposes).

This is probably more of a clarification on how to use git-annex directly. If I manually add a file under a gin repository using git-annex, i.e. ```bash git annex add mylargefile.bin ``` I was unable to automatically sync it using ```bash gin upload . ``` After some digging, it seems that the ssh-keys that gin manages under the hood are not forwarded to the underlying git-annex call. Only after explicitly adding new keys using ssh-keygen and then adding those to my ssh-agent did I manage to upload the files. I should note that this is using a private G-node instance running in Docker on my local computer (purely for testing purposes).

gin upload will (or should) use the ssh keys that were generated when you ran gin login for your particular server. If you look at the log file, do you see if it's trying to use the key or not?

`gin upload` will (or should) use the ssh keys that were generated when you ran `gin login` for your particular server. If you look at the log file, do you see if it's trying to use the key or not?
Roger Herikstad commented 4 months ago
Poster

Yes, it looks like it's trying to use the appropriate key. Here are the relevant lines from the log

2024/07/17 07:52:31 env GIT_SSH_COMMAND=ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o 'UserKnownHostsFile="/Users/roger/Library/Application Support/g-node/gin/known_hosts"'
2024/07/17 07:52:31 Running shell command (Dir: /Volumes/Drive1/data/MyTestData): git annex sync --verbose --no-pull --no-commit origin
2024/07/17 07:52:31 env GIT_SSH_COMMAND=ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o 'UserKnownHostsFile="/Users/roger/Library/Application Support/g-node/gin/known_hosts"'
2024/07/17 07:52:31 Running shell command (Dir: /Volumes/Drive1/data/MyTestData): git annex whereis --json .
2024/07/17 07:52:31 env GIT_SSH_COMMAND=ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o 'UserKnownHostsFile="/Users/roger/Library/Application Support/g-node/gin/known_hosts"'
2024/07/17 07:52:31 Running shell command (Dir: /Volumes/Drive1/data/MyTestData): git annex copy --json-progress --to=origin .
2024/07/17 07:52:32 Error during AnnexPush
2024/07/17 07:52:32
2024/07/17 07:52:32 The following error occured:
2024/07/17 07:52:32 Exiting with ERROR message: 1 operation failed

If I manually add the diskstationback.key key to the ssh agent, I can upload the file using git annex.

Yes, it looks like it's trying to use the appropriate key. Here are the relevant lines from the log ``` 2024/07/17 07:52:31 env GIT_SSH_COMMAND=ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o 'UserKnownHostsFile="/Users/roger/Library/Application Support/g-node/gin/known_hosts"' 2024/07/17 07:52:31 Running shell command (Dir: /Volumes/Drive1/data/MyTestData): git annex sync --verbose --no-pull --no-commit origin 2024/07/17 07:52:31 env GIT_SSH_COMMAND=ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o 'UserKnownHostsFile="/Users/roger/Library/Application Support/g-node/gin/known_hosts"' 2024/07/17 07:52:31 Running shell command (Dir: /Volumes/Drive1/data/MyTestData): git annex whereis --json . 2024/07/17 07:52:31 env GIT_SSH_COMMAND=ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o 'UserKnownHostsFile="/Users/roger/Library/Application Support/g-node/gin/known_hosts"' 2024/07/17 07:52:31 Running shell command (Dir: /Volumes/Drive1/data/MyTestData): git annex copy --json-progress --to=origin . 2024/07/17 07:52:32 Error during AnnexPush 2024/07/17 07:52:32 2024/07/17 07:52:32 The following error occured: 2024/07/17 07:52:32 Exiting with ERROR message: 1 operation failed ``` If I manually add the `diskstationback.key` key to the ssh agent, I can upload the file using git annex.

I have an idea of what might be wrong:

env GIT_SSH_COMMAND=ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key ...

It's setting two keys in the ssh command, diskstationback.key and localserver.key. This should work fine in most cases but it will fail if both keys are configured on the same server but for different users. In that case, it will depend on which key is used first and if that user who owns the key has permission to write to the repository.

Are those two keys for the same server?

This is really an oversight in the gin client. It should only be using the key for the specific server and user.

I have an idea of what might be wrong: ``` env GIT_SSH_COMMAND=ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key ... ``` It's setting two keys in the ssh command, `diskstationback.key` and `localserver.key`. This should work fine in most cases but it will fail if both keys are configured on the same server but for different users. In that case, it will depend on which key is used first and if that user who owns the key has permission to write to the repository. Are those two keys for the same server? This is really an oversight in the gin client. It should only be using the key for the specific server and user.
Roger Herikstad commented 4 months ago
Poster

Hm, they are actually for two different servers. Is it possible that specifying two different keys confuses ssh somehow?

Hm, they are actually for two different servers. Is it possible that specifying two different keys confuses ssh somehow?

Possibly. We can debug the thing by replicating what gin-cli does:

  • Set the GIT_SSH_COMMAND=ssh ... like seen in the log, but also add -vvv, so export GIT_SSH_COMMAND="ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key -o IdentitiesOnly=yes"
  • Run the git-annex command that failed: git annex copy --json-progress --to=origin .

The verbose output will show which key it tried and why it failed. You can then remove the wrong key from the GIT_SSH_COMMAND and try again to see if it works.

Possibly. We can debug the thing by replicating what gin-cli does: - Set the `GIT_SSH_COMMAND=ssh ...` like seen in the log, but also add `-vvv`, so `export GIT_SSH_COMMAND="ssh -i /Users/roger/Library/Application\ Support/g-node/gin/diskstationback.key -i /Users/roger/Library/Application\ Support/g-node/gin/localserver.key -o IdentitiesOnly=yes"` - Run the git-annex command that failed: `git annex copy --json-progress --to=origin .` The verbose output will show which key it tried and why it failed. You can then remove the wrong key from the `GIT_SSH_COMMAND` and try again to see if it works.
Roger Herikstad commented 4 months ago
Poster

Hm, this is weird. According to the output, it is not even trying to use the key I specify in the -i argument. I think that suggests there is something wrong with my ssh config somehow. I'll have to do some investigations.

Hm, this is weird. According to the output, it is not even trying to use the key I specify in the -i argument. I think that suggests there is something wrong with my ssh config somehow. I'll have to do some investigations.
Sign in to join this conversation.
No Milestone
No assignee
2 Participants
Loading...
Cancel
Save
There is no content yet.