[Cyberduck-trac] [Cyberduck] #11833: Extremely slow S3 downloads of large files

Cyberduck trac at cyberduck.io
Sat Oct 2 16:20:22 UTC 2021


#11833: Extremely slow S3 downloads of large files
-------------------------+------------------------
    Reporter:  allklier  |      Owner:
        Type:  defect    |     Status:  new
    Priority:  normal    |  Milestone:
   Component:  core      |    Version:  7.10.2
    Severity:  normal    |   Keywords:
Architecture:  Intel     |   Platform:  Windows 10
-------------------------+------------------------
 For the second time in a few weeks I've face massive delays in download
 large files from an S3 bucket.

 In the latest case it's a single 220GB video file. In other case it was a
 number of larger video clips totaling 150GB.

 The actual download proceeds at network speed (Gigabit Internet), however
 after the last segment has been fetched Cyberduck sits for hours at 100%
 while it's assembling the segments into a single file. Despite having a
 very fast RAID (800MB/s transfer rates).

 It seems it shouldn't take that long if the code were to simply read each
 segment and concatenate them. I'm assuming there must be some inefficiency
 such as writing the entire file with each segment, or similar as the only
 reason why this exponentially slows down with file size.

 In this last download it created 104 2G segments. The last segment (104)
 completed download at 10:37AM. It's now 12:19PM and the new combined file
 still is only 109GB (about 50% of the total).

 That practically makes Cyberduck unusable for files like this.

--
Ticket URL: <https://trac.cyberduck.io/ticket/11833>
Cyberduck <https://cyberduck.io>
Libre FTP, SFTP, WebDAV, S3 & OpenStack Swift browser for Mac and Windows


More information about the Cyberduck-trac mailing list