Bos55/Hugo

Fork 0

Nextcloud backup never finishes #22

New issue

Closed

opened 2023-06-13 22:46:38 +02:00 by tdpeuter · 6 comments

tdpeuter commented

2023-06-13 22:46:38 +02:00

Owner

The Nextcloud backup task never finishes transfer

Steps already taken to prevent this:

ZFS snapshot before transfer

Logs after forcing running task to stop do not show anything particular

The Nextcloud backup task never finishes transfer ![running task](/attachments/56e3ba42-fa6f-4ecd-a9f9-1f06db566ffb) Steps already taken to prevent this: - ZFS snapshot before transfer Logs after forcing running task to stop do not show anything particular

463512.log

7.6 MiB

afbeelding.png

8.7 KiB

tdpeuter commented

2023-06-17 17:58:29 +02:00

Author

Owner

Some more logs

476477.log

165 KiB

tdpeuter commented

2023-06-20 07:00:07 +02:00

Author

Owner

After running for a couple of days.

It is possible to view the rsync output by tailing the logs of the TrueNAS task (which is just a number).

tail -f /var/log/jobs/485544.log

Lines that seem to occur often (throughout multiple logs) are

Transferring:
 * tdpeuter/files/Video's…he Final Countdown.mp4: 19% /7.104Gi, 2.814Mi/s, 34m34s
 * tdpeuter/files_trashbi…70543Z.iso.d1665762992: 16% /3.772Gi, 2.842Mi/s, 18m58s

I will have to look into those files

![image](/attachments/9f667e7c-28ef-44b9-9003-4613516c6751) After running for a couple of days. It is possible to view the rsync output by tailing the logs of the TrueNAS task (which is just a number). `tail -f /var/log/jobs/485544.log` Lines that seem to occur often (throughout multiple logs) are ``` Transferring: * tdpeuter/files/Video's…he Final Countdown.mp4: 19% /7.104Gi, 2.814Mi/s, 34m34s * tdpeuter/files_trashbi…70543Z.iso.d1665762992: 16% /3.772Gi, 2.842Mi/s, 18m58s ``` I will have to look into those files

image.png

13 KiB

tdpeuter commented

2023-06-20 07:45:07 +02:00

Author

Owner

https://serverfault.com/q/627002

https://bobcares.com/blog/rsync-hangs-on-large-files/

The issue could be due to insufficient RAM in the destination server to handle files larger than say 5GB. Might have to open Nicole up soon to check that. We need to be in there anyways.

I believe, what happens is that Rsync fails because the server hangs. TrueNAS will always try again as long as it has connectivity to the destination. However, every time the transfer fails, it removes what has already been transferring, resulting in a loop where the first part of the file is being transferred before hanging again.

It's also possible that timeout is reached because Nicole (the backup server) takes a while to calculate the checksum of the big file.

https://serverfault.com/q/627002 https://bobcares.com/blog/rsync-hangs-on-large-files/ The issue could be due to insufficient RAM in the destination server to handle files larger than say 5GB. Might have to open Nicole up soon to check that. We need to be in there anyways. I believe, what happens is that Rsync fails because the server hangs. TrueNAS will always try again as long as it has connectivity to the destination. However, every time the transfer fails, it removes what has already been transferring, resulting in a loop where the first part of the file is being transferred before hanging again. It's also possible that timeout is reached because Nicole (the backup server) takes a while to calculate the checksum of the big file.

tdpeuter commented

2023-06-20 17:42:22 +02:00

Author

Owner

<6>INFO  : 
Transferred:      422.353Gi / 425.717 GiByte, 99%, 14.819 KiByte/s, ETA 2d18h7m17s
Errors:                86 (retrying may help)
Checks:            719559 / 719559, 100%
Transferred:          101 / 103, 98%
Elapsed time:   10h1m57.3s
Transferring:
 * tdpeuter/files_trashbi…ectuur.vdi.v1678947093:100% /16.772Gi, 7.218Mi/s, 0s
 * tdpeuter/files_trashbi…mputerArchitectuur.vdi: 79% /16.778Gi, 8.255Mi/s, 6m57s

<6>INFO  : 
Transferred:      422.361Gi / 442.489 GiByte, 95%, 14.819 KiByte/s, ETA 2w2d11h36m57s
Errors:                86 (retrying may help)
Checks:            719559 / 719559, 100%
Transferred:          101 / 103, 98%
Elapsed time:   10h1m58.3s
Transferring:
 * tdpeuter/files_trashbi…ectuur.vdi.v1678947093:  0% /16.772Gi, 0/s, -
 * tdpeuter/files_trashbi…mputerArchitectuur.vdi: 79% /16.778Gi, 8.489Mi/s, 6m44s

In these consecutive lines, you can see that it for some reason retries the finished file. Once again a big file: I will remove this unnecessary file and try again.

``` <6>INFO : Transferred: 422.353Gi / 425.717 GiByte, 99%, 14.819 KiByte/s, ETA 2d18h7m17s Errors: 86 (retrying may help) Checks: 719559 / 719559, 100% Transferred: 101 / 103, 98% Elapsed time: 10h1m57.3s Transferring: * tdpeuter/files_trashbi…ectuur.vdi.v1678947093:100% /16.772Gi, 7.218Mi/s, 0s * tdpeuter/files_trashbi…mputerArchitectuur.vdi: 79% /16.778Gi, 8.255Mi/s, 6m57s <6>INFO : Transferred: 422.361Gi / 442.489 GiByte, 95%, 14.819 KiByte/s, ETA 2w2d11h36m57s Errors: 86 (retrying may help) Checks: 719559 / 719559, 100% Transferred: 101 / 103, 98% Elapsed time: 10h1m58.3s Transferring: * tdpeuter/files_trashbi…ectuur.vdi.v1678947093: 0% /16.772Gi, 0/s, - * tdpeuter/files_trashbi…mputerArchitectuur.vdi: 79% /16.778Gi, 8.489Mi/s, 6m44s ``` In these consecutive lines, you can see that it for some reason retries the finished file. Once again a big file: I will remove this unnecessary file and try again.

486359.log

16 MiB

tdpeuter commented

2023-06-30 22:14:43 +02:00

Author

Owner

Might have to open Nicole up soon to check that.

So opened Nicole up, only to be disappointed by an integrated board with no way to upgrade the 2 Gigs of RAM. We will have to tackle this issue by using the rsync options that allow continuing from stopped transfer.

> Might have to open Nicole up soon to check that. So opened Nicole up, only to be disappointed by an integrated board with no way to upgrade the 2 Gigs of RAM. We will have to tackle this issue by using the rsync options that allow continuing from stopped transfer.