[Cyberduck-trac] [Cyberduck] #10564: Large jobs take a long time after the last transfer to be marked as finished

[Cyberduck]

#10564: Large jobs take a long time after the last transfer to be marked as
finished
------------------------+-------------------------
 Reporter:  sebastian   |         Owner:  dkocher
     Type:  defect      |        Status:  reopened
 Priority:  normal      |     Milestone:
Component:  core        |       Version:  6.8.3
 Severity:  normal      |    Resolution:
 Keywords:              |  Architecture:  Intel
 Platform:  Windows 10  |
------------------------+-------------------------
Changes (by sebastian):

 * status:  closed => reopened
 * resolution:  duplicate =>
 * milestone:  6.9.0 =>


Comment:

 Replying to [comment:1 dkocher]:
 > This is caused by verifying the checksum of the downloaded file. Related
 to #10215.


 Unfortunately this can not be the reason, because (as the ticket states)
 the behaviour occurs also on '''uploads''' where the checksum is
 calculated before the transfer.

 So I did some more testing (on Windows using the Cyberduck GUI and the
 file:// protocol) and can provide more, hopefully helpful insights:
 * The waiting time only occurs when working with cryptomator vaults, not
 when working directly with an unencrypted local folder.
 * The waiting time scales (quadratically) with the '''number of files''',
 not the amount of data:
  * 10.000 MB in 3 files produced no noticeable waiting time.
  * 0,8 MB in 100 files produced about 1 second of waiting time.
  * 1,6 MB in 200 files produced about 5 seconds of waiting time.
  * 3,2 MB in 400 files produced about 20 seconds of waiting time.
  * 6,4 MB in 800 files produced about 82 seconds of waiting time.
  * I did not wait for the job of 80 MB in 10.000 files to finish, but the
 waiting time calculates to about 3,5 hours given the observed quadratic
 growth.

 My assumption is that for each file transferred (either up- or downloaded)
 from/to a cryptomator vault some data structure is left over and has to be
 cleaned up after the transfer is finished and that cleanup is done in an
 inefficient manner.

--
Ticket URL: <https://trac.cyberduck.io/ticket/10564#comment:3>
Cyberduck <https://cyberduck.io>
Libre FTP, SFTP, WebDAV, S3 & OpenStack Swift browser for Mac and Windows