Dealing_with_Gin/gin-client/troubleshooting.md

Gin-CLI: troubleshooting

• The gin command not found
• Gin-CLI hangs when adding / committing changes
• Uploading fails with an error
• The content of a large file is not uploaded

The gin command not found

Symptoms

Your terminal emulator complains that gin command is not found.

But you can see your gin program on your PC.

Diagnosis

In short: it is very likely that the PATH environment variable is not set (properly) for your Gin-CLI.

The terminal emulator looks for commands by visiting the directories listed in its settings. The settings for the terminal emulator is called the "environment variables", and the variable for the list of the command-lookup directories is called PATH.

In many cases, you have to manually add to PATH the directory where the command is located.

Solutions

• If there has been some time since you downloaded Gin-CLI, downloading the latest binary will prevent any further unexpected problems.
• For Gin-CLI on Windows, there is a nice file set-global.bat to do it for you. Just double-click this file.

Even when you use set-global.bat, sometimes the directory is not added properly because there are already lots of directories listed in PATH and it cannot add any more due to the character-number limitation(!). It can occur when you have already installed lots of programs on your computer.

In such cases, the possibilities are:

• Check PATH one by one, and try to reduce the number of characters in it: defining the PATH environment variable is much like defining the HTTP_PROXY variable on this section, so please refer to it.
• Use gin-shell.bat as an alternative terminal emulator.

Gin-CLI hangs when adding / committing changes

Symptoms

• You run either gin commit or gin upload to add/commit changes.
• Gin-CLI lists up the files that changed, but the progress stops at some point onwards.
• You can abort the procedure by hitting Ctrl + C, but you end up failing to add changes.

Diagnosis

Although this situation has not been fully resolved, there are two factors that may be involved: large file-sizes, and a large number of files.

• large file size: adding a large file via gin takes time (increases linearly with the file size). Probably you see in the Process Manager that Gin-CLI consumes some CPU power until it is done. Please wait for some time (up to several minutes).
• many files:
  • I suspect that gin has some issues when adding too many changes all at once.
  • Git is slow when you commit a lot of changes all at once (takes up to several minutes, hopefully).
  • Git for Windows cannot handle a commit with more than 2 GB (excluding the sizes of "large files").

Solutions (or tips)

Below are some tips to avoid the situation:

• add, commit and upload separately.
  1. to add changes: gin annex add <files or directories>
  2. to add deletions: gin git add --update
  3. to commit changes: gin git commit (optionally appended by a custom message: -m "<message>")
  4. to upload: gin upload (note: without a dot . at the end)
• try to add / commit large directories separately: individual processes will take (at least) much shorter time, and thereby keeping you feel safer and more comfortable.
• register all "data files" as "large" (annexed) files: to do so, under the repository-root directory (or optionally under the individual directories separately) where you keep data files, add and edit the .gitattributes file. IMPORTANT: if you want gin to take this settings into account, you must use gin annex add (instead of gin commit).
  1. Create a plain-text file, and rename it to .gitattributes (without any extension e.g. ".txt").
  2. Open it from your text editor, and add lines as below:
     • The setting in general: a line * annex.largefiles=largerthan=100kb will add all files into annex if they have more than 100 kilobytes in size.
     • The extension-specific setting: for example, a line *.mat annex.largefiles=anything will add all files into annex if they have the ".mat" extension.
     • The file-specific un-setting: for example, a line *.ipynb annex.largefiles=nothing will add the ".ipynb" file normally (i.e. without being annexed) no matter the file size.
     • You can add a comment line, by starting it with a hash #.

The example file is as below (you can copy and paste it if you like):

# general settings
*       annex.largefiles=largerthan=100kb
LICENSE annex.largefiles=nothing

# data files
*.dat   annex.largefiles=anything
*.mat   annex.largefiles=anything
*.ibw   annex.largefiles=anything
*.ifn   annex.largefiles=anything
*.npy   annex.largefiles=anything
*.npz   annex.largefiles=anything
*.tif   annex.largefiles=anything
*.tiff  annex.largefiles=anything
*.png   annex.largefiles=anything # it depends, but safer this way
*.jpg   annex.largefiles=anything # it depends, but safer this way
*.jpeg  annex.largefiles=anything # it depends, but safer this way
*.bmp   annex.largefiles=anything
*.avi   annex.largefiles=anything
*.mp4   annex.largefiles=anything
*.json  annex.largefiles=anything # it depends, but safer this way
*.csv   annex.largefiles=anything # it depends, but safer this way
*.tsv   annex.largefiles=anything # it depends, but safer this way
*.xml   annex.largefiles=anything # it depends, but safer this way

# binary document files
*.pdf   annex.largefiles=anything
*.docx  annex.largefiles=anything
*.pptx  annex.largefiles=anything
*.xlsx  annex.largefiles=anything

# code and text files
*.md    annex.largefiles=nothing
*.txt   annex.largefiles=nothing
*.yml   annex.largefiles=nothing
*.yaml  annex.largefiles=nothing
*.m     annex.largefiles=nothing
*.r     annex.largefiles=nothing
*.py    annex.largefiles=nothing
*.ipynb annex.largefiles=nothing
*.htm   annex.largefiles=nothing
*.html  annex.largefiles=nothing

Uploading fails with an error

Uploading involves a lot of uncertainties, but gin may only say upload failed (if it says upload OK, then please refer to another section).

Please check the points below, one by one:

1. Is the Gin server alive?
   • Visit https://gin.g-node.org/ and check if there are any maintenance etc. going on.
2. Have you changed your repository name recently? It changes the URL to communicate, so the local repositories that were downloaded before renaming will fail to upload/download changes.
   • In your local repository: run gin git remote set-url origin <URL> The <URL> must be in the form: ssh://git@gin.g-node.org:22/<organization>/<repository-name> . Make sure that you do not have .git after the repository name!
3. Do you use the wired connection from Charité?
   • Please refer to this page for problems related to this point.
   • In case you have not done any proxy settings on the command line, try connecting via a Wi-Fi.
   • If the error says there is a problem in connecting to larkumlab-data-micro.charite.de, then it is the problem with the SSH proxy server. Ask someone for restarting the proxy service.
   • Please do note that some other problems (like this) may persist even after solving this problem...

If the problem does not fall into one of the points above, please contact someone who knows more.

The content of a large file is not uploaded

Symptoms

• You run gin upload, and it says uploading OK.
• The directory organization becomes up-to-date on the server after gin upload.
• Yet, if you look at the large file on the server, it says "no content available".

Diagnosis

Below are the points to our current interpretation:

• gin has experienced a problem (likely with SSH) some time before, when it tries to upload contents of large data files.
• and then gin erranously considered "this server is not compatible with uploading large data files".
• since then, gin does not attempt to upload file contents anymore.

Solution

In short: gin (or git annex) uses the git config remote.origin.annex-ignore to remember whether the server has the annex backend. All you have to do is to unset this config.

In reality, it requires a bit of complexity to solve the problem:

1. In your repository, run gin git config --list. It shows all the repository-specific Git configuration. (listed in the alphabetical order)
2. Find the line starting with remote.origin.annex-ignore= Make sure that is says true (Otherwise it must be due to some other reasons).
3. Run: gin git config --unset remote.origin.annex-ignore It should return without any output.
4. You can run gin git config --list to make sure that the property is gone.