[Cyberduck-trac] [Cyberduck] #10087: Large directory on S3 not opening

Cyberduck trac at cyberduck.io
Fri Sep 22 07:41:20 UTC 2017


#10087: Large directory on S3 not opening
-----------------------+-------------------------
    Reporter:  bjinse  |      Owner:
        Type:  defect  |     Status:  new
    Priority:  normal  |  Milestone:
   Component:  s3      |    Version:  6.2.6
    Severity:  normal  |   Keywords:
Architecture:  Intel   |   Platform:  macOS 10.12
-----------------------+-------------------------
 I have an S3 directory with a lot of files. About 400.000
 CD is opening the directory in chunks of 1000 (according to the log). Just
 before the end (might be the last 1000 items), it stops.
 Then I have max processor usage and no network activity.

 Looks like CD retried 10 minutes later or so. Same activity in the log,
 and again stops just before the end.

 First time statusbar message was: Authenticating, this time it is: Login
 succesful (Well in Dutch...)

 Opening this directory used to work (although it took a lot of time, like
 5 minutes or so), but of course the number of items is growing. Did I
 reach a maximum directory items threshold?

 Log ends with:
 GET /?max-keys=1000&versions&prefix=tl_assets%2F400%2F&delimiter=%2F
 &version-id-marker=null&key-
 marker=tl_assets%2F400%2Fyellowcab_yellowcabb_y37090_moss_winter_2016_3d24_400.jpg
 HTTP/1.1
 Date: Fri, 22 Sep 2017 07:16:45 GMT
 x-amz-request-payer: requester
 x-amz-content-sha256:
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 Host: production-publicbucket-xx1az459c85z.s3.amazonaws.com
 x-amz-date: 20170922T071645Z
 Authorization: ********
 Connection: Keep-Alive
 User-Agent: Cyberduck/6.2.6.26405 (Mac OS X/10.12.5) (x86_64)
 HTTP/1.1 200 OK
 x-amz-id-2:
 Su1iVBolRrntKrVrKr+LVWeRS9v/gsOsxw2tysbh8vQ/GWKCl7BVHcqVhBtq85IZZn6tOraBQtc=
 x-amz-request-id: 889DD509E82DFDD9
 Date: Fri, 22 Sep 2017 07:16:46 GMT
 Content-Type: application/xml
 Transfer-Encoding: chunked
 Server: AmazonS3
 GET /?max-keys=1000&versions&prefix=tl_assets%2F400%2F&delimiter=%2F
 &version-id-marker=null&key-
 marker=tl_assets%2F400%2Fzecchinodoro_zecchinodg_a191968_silver_zomer_2016_3d06_400.jpg
 HTTP/1.1
 Date: Fri, 22 Sep 2017 07:16:45 GMT
 x-amz-request-payer: requester
 x-amz-content-sha256:
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 Host: production-publicbucket-xx1az459c85z.s3.amazonaws.com
 x-amz-date: 20170922T071645Z
 Authorization: ********
 Connection: Keep-Alive
 User-Agent: Cyberduck/6.2.6.26405 (Mac OS X/10.12.5) (x86_64)
 HTTP/1.1 200 OK
 x-amz-id-2:
 P+9in7fBELqsSkosVqCdI/fJeIOoYm6FkrJ9LRrORIQnQYwRZ1YE+akLiQtcZx2hJDCeHIt12fw=
 x-amz-request-id: C7EAA78031C5A49D
 Date: Fri, 22 Sep 2017 07:16:46 GMT
 Content-Type: application/xml
 Transfer-Encoding: chunked
 Server: AmazonS3
 GET /?max-keys=1000&versions&prefix=tl_assets%2F400%2F&delimiter=%2F
 &version-id-marker=null&key-
 marker=tl_assets%2F400%2Fzecchinodoro_zecchinodg_n121251_51445041304_zomer_2017_3d33_400.jpg
 HTTP/1.1
 Date: Fri, 22 Sep 2017 07:16:46 GMT
 x-amz-request-payer: requester
 x-amz-content-sha256:
 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 Host: production-publicbucket-xx1az459c85z.s3.amazonaws.com
 x-amz-date: 20170922T071646Z
 Authorization: ********
 Connection: Keep-Alive
 User-Agent: Cyberduck/6.2.6.26405 (Mac OS X/10.12.5) (x86_64)
 HTTP/1.1 200 OK
 x-amz-id-2:
 hLtIZGUT4NCc2LebAKrvazrLiqHTDml17eN4Es5m1IPAn9qB1TfqAD7Pbkl4EZieJg2cUKNQcQ4=
 x-amz-request-id: AED983E421CE06E0
 Date: Fri, 22 Sep 2017 07:16:47 GMT
 Content-Type: application/xml
 Transfer-Encoding: chunked
 Server: AmazonS3
 Connection: close

--
Ticket URL: <https://trac.cyberduck.io/ticket/10087>
Cyberduck <https://cyberduck.io>
Libre FTP, SFTP, WebDAV, S3 & OpenStack Swift browser for Mac and Windows


More information about the Cyberduck-trac mailing list