#49 Problem uploading files

Hi,

I am trying to upload some files using gin-cli and I get 'operation failed'. The status of the known files to fail is just listed as 'Locally modified (not uploaded):' when running 'gin ls'. I've tried uploading them separately but this also fails...

How can I solve this issue?

Thanks for a great product!

Hi Sara!

From experience this particular message has been tied to trying to upload many small files (individual size < 10MB) with a total size of >4GiB that have been added in one single commit, which is something git does not handle well. If this is the case, please try splitting the commit into a couple of commits with fewer files in each commit and try uploading again.

If this was not the case or does not solve the issue, could you tell us on which operating system you work and which version of the gin client you are using (run gin version)?

Could you check the gin client logfile? You should be able to find it at

• Windows: c:\users\{user}\appdata\local\g-node\gin\gin.log
• Linux: /home/{user}/.cache/g-node/gin
• MacOS: /Users/<User>/Library/Caches/g-node/gin/gin.log

Please check the log before running the offending command, run the upload command on one of the files causing the issue. Then please post the content of the gin client log that has been added for this upload so we can look at the issues in more detail.

Hi Michael,

Thank you so much for your fast reply. I tried both a folder and single files, both resulted in the same error message.

When checking the gin version, I get this version for when uploading from this computer: c273f1c5 (I have it also installed on my laptop, which has been working to upload files)

I tried again and attaching the error message from the logfile.

Thank you so much for helping me!

Dear Michael,

Did you get a chance to have a look at the issue I was facing?

Thank you very much for your help!

Hi Sara,
can you describe your issue more?
On what OS you are using gin-cli?
What gin-cli version are you using? (command "gin --version")
can you share the logfile that Michael mentioned above?

Hi Jiri,

my issue is mention in the first post but in short, I cannot upload my files. When I try single files it just says 'operation fails'. I'm using Windows as my OS.

When running 'gin --version' I recieve: GIN command line client 1.11 Build 001459 (103e923620) git: 2.28.0.windows.1 git-annex: 8.20200330-g971791563

You can find the log file for the recent upload attempt in the attachment.

Thank you very much for your help!

Hi Sara, can you please try to donwload and use latest version gin-cli 1.12 that fixes some issue on Windows with bundled git and git annex and path. Are you using Windows 10 or older version? Didnt you forget to attach the log file?

So "gin commit" works without any error and only "gin upload" gives you error?

Apologies, I tried to attach it as a wordpad doc but it seems I cannot upload word files, text files or pdf files.... What format does it need to be?

Yes, I am using Windows 10 and gin commit works fine it seems (although no changes recorded) so it seems to be only to upload the content.

Hi Sara, I am not sure, but maybe only pictures are allowed as attachement. Can you put last 20 lines of log, after commint attempt, here in post?

I see, I converted them now to PNG and this hopefully works.

Thank you for your help!

Hi Sara, it looks like issue with git annex files. I presume that files that are not uploaded (Locally modified (not uploaded)) are larger than 10 MB? Can you run inside the gin repository folder (with gin-cli) command "gin sync" and again put here the log output for this command?

Exactly!

Here is a the gin sync log output.

Thank you

Can you please try the "gin upload ." again amd check the files status with "gin ls" I thought that gin sync will give us more specific error with git annex, but it finished without any error...

Could be the issue when git annex failed some time before, and the annex connection becomes disabled in git config. I don’t remember the exact name, but you have to re-enable it.

I tried again and will now upload the gin sync and the gin ls output

Thanks Keisuke I'm not sure... how do I re-enable the annex connection? It could be because it's only the larger files that failed.

Sara can you put here the git config file from repository? It is in hidden folder ".git" in repository

Ok, here is the config file,

[core] repositoryformatversion = 0 filemode = false bare = false logallrefupdates = true symlinks = false ignorecase = true quotepath = false [remote "origin"] url = ssh://git@gin.g-node.org:22/larkumlab/sara_moberg_association_formation fetch = +refs/heads/:refs/remotes/origin/ annex-ignore = true [branch "master"] remote = origin merge = refs/heads/master [user] name = Sara Moberg email = [annex] backends = MD5 addunlocked = true uuid = 50247185-5737-41d3-a668-4fa6dee100 sshcaching = false crippledfilesystem = true version = 8 [filter "annex"] smudge = git-annex smudge -- %f clean = git-annex smudge --clean -- %f [gin] remote = origin ~ ~ ~ ~ ~ ~ config unix 1,1 All"config" [unix] 30L, 673C

I hope that removing the parametr "annex-ignore = true" will help. But just to be sure, make backup of the old config file before making changes.

Ok, I don't know how to edit this file in the prompt (I got help to redirect and show the file). Can you help me edit it?

So a friend is helping me and we commented out that line with a #

I tried uploading again and got this error message:

  0 [main] ssh (332) C:\Program Files\Git\usr\bin\ssh.exe: *** fatal error - cygheap base mismatch detected - 0x180317408/0x180348408.

This problem is probably due to using incompatible versions of the cygwin DLL. Search for cygwin1.dll using the Windows Start->Find/Search facility and delete all but the most recent version. The most recent version should reside in x:\cygwin\bin, where 'x' is the drive on which you have installed the cygwin distribution. Rebooting is also suggested if you are unable to find another cygwin DLL.

[error] 1 operation failed

Did you download the latest gin-cli version (1.12)? The GIN is using bundled Git and Git Annex to resolve the different versions issues. It should use git and git annex in gin-cli folder and the latest version (https://gin.g-node.org/G-Node/Info/wiki/GIN+CLI+Setup#windows) of gin-cli should fix that.

Ah, it seems that I have the 1.11 version of GIN. Is there no way I can update it in the prompt instead of downloading the poackage?

I am not aware of any "automatic" update that can be done... If you can run powershell you can download the lastest version in powershell with commands:
(new-object System.Net.WebClient).DownloadFile("https://gin.g-node.org/G-Node/gin-cli-releases/raw/master/gin-cli-latest-windows64.zip","C:\Temp\gin-cli.zip")
Expand-Archive -Path C:\Temp\gin-cli.zip -DestinationPath C:\Temp\gin-cli112 < br/>

I tried downloading, also getting the repository from scratch and tried again. Still failed but it was connected to the network via cable so I dissconnected it and tried wireless with eduroam. It still failed (see pngs for both with(first attempt) and without (second attempt)). I checked the version and it was still 1.11 on the newly downloaded one....

Could be the issue when git annex failed some time before, and the annex connection becomes disabled in git config. I don’t remember the exact name, but you have to re-enable it.

The setting for this is called annex.ignore and it may be causing this.

The easiest way to check for this is to open the git config file in the repository (.git\config) using a simple text editor and checking if there is any line that reads: annex.ignore = true.

If you find one, delete the line, or change it to false

Thank you Achilleas!

The error-message doesn't show up anymore but now I don't have the choice to upload the bigger files. They are in the folder on the desktop but don't show up in the prompt....

That's strange. Does running gin upload . do anything now?

What's the output of the following?

• gin ls
• git status
• gin annex info

Hi,

gin upload . only uploads the smaller files and when I check gin ls I can only see the smaller files and not the bigger ones (they're not even listed as local copies)

gin status shows the same.

gin annex info shows this:

D:\gin\sara_moberg_association_formation\10_rawdata\MLA12736>gin annex info trusted repositories: 0 semitrusted repositories: 17

    00000000-0000-0000-0000-000000000001 -- web
    00000000-0000-0000-0000-000000000002 -- bittorrent
    1efac426-6c58-4f77-8e1c-549bb02d9451 -- git@c3d83a1cb62f:/data/tmp/appdata/tmp/local-repo/1619
    2e1f5d5c-e353-4859-9ddc-057c49aca41f -- git@c3d83a1cb62f:/data/tmp/appdata/tmp/local-repo/1619
    41b64453-a082-412a-8bac-a93cfbd587dd -- git@c3d83a1cb62f:/data/tmp/appdata/tmp/local-repo/1619
    4d4a2581-9083-4b21-a7fd-0246333b51c4 -- git@def205aad250:/data/tmp/appdata/tmp/local-repo/1619
    50247185-5737-41d3-a668-4fa6dee10069 -- moberg@DESKTOP-TSB8H9C [here]
    5a085806-dec1-455d-9f48-36f76c8b9b16 -- git@c3d83a1cb62f:/data/tmp/appdata/tmp/local-repo/1619
    704f5231-e7d6-48d1-926b-1c0ad2594a5d -- git@b71abff825ef:/data/tmp/appdata/tmp/local-repo/1619
    82213ca4-d8c6-418b-8df0-ab41e5dfc54a -- git@c3d83a1cb62f:/data/tmp/appdata/tmp/local-repo/1619
    91322776-5641-4ff3-bad8-2c943bd8f736 -- git@c3d83a1cb62f:/data/tmp/appdata/tmp/local-repo/1619
    972210d7-bd11-4a30-b63c-ec029a1d2388 -- Moberg@DESKTOP-A8EI0PU
    ba79057c-cb4b-4c87-8114-ad3318e490ec -- git@def205aad250:/data/repos/larkumlab/sara_moberg_association_formation.git [origin]
    bd766067-a0d8-4f5e-8644-6566bbf3e447 -- git@c3d83a1cb62f:/data/tmp/appdata/tmp/local-repo/1619
    d52987bb-6233-4f28-8aa7-38a9f5b21bbc -- Moberg@DESKTOP-A8EI0PU
    ec89f990-ed7a-4a26-a306-04c889edb5a4 -- git@c3d83a1cb62f:/data/tmp/appdata/tmp/local-repo/1619
    faab4d47-047e-469e-91c9-905ef73f42a5 -- colombj@MacBook-Pro-4:~/github_repo/07_larkumlabrepos/sara_moberg_association_formation

untrusted repositories: 0 transfers in progress: none available local disk space: 1.47 terabytes (+1 megabyte reserved) local annex keys: 6 local annex size: 3.22 gigabytes annexed files in working tree: 725 size of annexed files in working tree: 31.25 gigabytes bloom filter size: 32 mebibytes (0% full) backend usage:

    MD5: 725

The output shows that there are 6 annex files locally. Does this match what you expect?

Another thing that should help shed light here is running:

gin annex whereis .

This will list all annexed files with their locations (locally, server, or both). Could you please paste the output from that?

whereis suite2p/plane0/Fall.mat (2 copies)

    50247185-5737-41d3-a668-4fa6dee10069 -- moberg@DESKTOP-TSB8H9C [here]
    ba79057c-cb4b-4c87-8114-ad3318e490ec -- git@def205aad250:/data/repos/larkumlab/sara_moberg_association_formation.git [origin]

ok

but are these the two copies are were already uploaded? I have in total 8 files, two are uploaded and I'm having issues with the residual 6, you're right.

This only shows one file having two copies (one local and one on the server). Were there no other files in the list?

What happens when you run gin annex whereis and name one of the files that seem to be ignored?

Hi Achilleas,

Sorry I have been away and just got back. I tried this but since I think we disabled the annex (annex.ignore = false). Since then the bigger files do not show in the prompt and the gin annex whereis for the bigger file cannot be found... Should I change it back so annex.ignore = true ?

Setting annex.ignore = false shouldn't stop annex from working, quite the opposite (it disables ignore). I wonder if there's something wrong with the syntax. Please try removing the annex.ignore line instead, which should set it to the default behaviour.

Hi Achilleas,

I stopped trying now because I don't know how to navigate in to the config file, edit it or save it or anything.... Thank you so much for helping me though.

I have a separate issue now... I am now uploading files from my laptop where it all works fine and then downloading on the work desktop. The problem now is the files I download do not show up in the gin folder.... When I do 'gin ls' they show up as green and OK but they're not in mz local folder. I tried both 'gin download' and 'gin get-content' but neither work. Can someone help me with this?

Thanks in advance!

This is very strange. I don't know how the files could be showing up in the output of gin ls and not be in the directory. If you gin get the repository into a new directory, do the files show up? The large files should be empty pointer-files, but they should all be there.

Are the files possibly marked as hidden in the folder view?

Hi Achilleas, zes I also find it very strange... I now deleted and re-synced the repo and all the files are now there.

Thank you for your help!

I have a new issue... I am using gin on my laptop and now am trying to use gin on my desktop computer (where all the issues are arising from).

The previous issue got solved but now I made some changes on some code on desktop computer and uploaded the changes. When I tried to download the changes on my laptop I get the following message:

Downloading changes [error] download failed: local modified or untracked files would be overwritten by download: 30_code/behavior analysis code/auroc_compile_and_plot.m

Can you help me with this? I tried closing everything on my laptop and re-opening it but got the same message...

Cheers, Sara

I'm trying to just overrun the changes from my laptop as this synchrony with the desktop is messing things up... But now I cannot upload any changes from the laptop (or download anything for that matter, gin sync doesn't work either). Is there no way to overrun the changes from the local to the remote repo?

I'm trying to just overrun the changes from my laptop as this synchrony with the desktop is messing things up... But now I cannot upload any changes from the laptop (or download anything for that matter, gin sync doesn't work either). Is there no way to overrun the changes from the local to the remote repo?

There are a few ways to get into a conflict and the way to get out depends on the kind of conflict. The help pages have a guide for this that should help you figure out what exactly is going on: https://gin.g-node.org/G-Node/Info/wiki/GIN+CLI+Usage+Tutorial#handling-conflicts

Let me know if the solutions don't work or the instructions are unclear.

Thank you Achilleas! I had a look and tried to moved/rename the conflicted files using "mv" but it still comes up as conflicted files when I tried to download.... I also made sure to delete the old files but didn't help...

One last resort, if you want to override everything local and make your local repository reflect the one on the server, is to do a git reset. Note that this will remove any local changes, so if you have files in the local directory that you might want to keep, copy them outside the repository first:

gin git fetch --all
gin git reset --hard origin/master

origin/master is the default symbolic name for the remote repository and a gin git reset --hard tells git to reset everything to reflect it.

Hi Achilleas, thank you for your help!

It now works!

Hi Achilleas, thank you for your help!

It now works!