[Cyberduck-trac] [Cyberduck] #10995: ListObjects for directories over 1000 files doesn't work for S3 compatible storage

[Cyberduck]

#10995: ListObjects for directories over 1000 files doesn't work for S3 compatible
storage
-----------------------+-------------------------
    Reporter:  skier5  |      Owner:
        Type:  defect  |     Status:  new
    Priority:  normal  |  Milestone:
   Component:  s3      |    Version:  7.1
    Severity:  normal  |   Keywords:
Architecture:          |   Platform:  macOS 10.15
-----------------------+-------------------------
 Starting with Cyberduck 7.1.0, I've noticed that when connecting to a
 private S3 compatible storage service, I can no longer list files in a
 directory where there are over 1000 files/keys present.  The UI presents
 itself as if there were no files in the directory.

 In Cyberduck 7.0.2, executing the same scenario results in displaying the
 >1000 list of keys.

 When turning on debugging, I see the following:

 > GET /bucket/?encoding-type=url&max-
 keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F HTTP/1.1
 This returns a 200.

 I also see
 > GET /bucket/?encoding-type=url&max-
 keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F&marker=path%2Fto%2Flarge%2Fdirectory%2Fmarker.json
 HTTP/1.1
 This also returns a 200.

 Then I see the following
 ```
 2020-03-17 13:00:23.199539-0400 0x7427b8   Default     0xaecc6e
 88140  0    Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40
 ch.cyberduck.core.threading.BackgroundCallable] Failure
 java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in
 escape (%) pattern - Error at index 1 in: "ap" running background task
 java.lang.Exception
         at
 ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
         at
 ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
         at
 ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
         at
 ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:517)
         at
 ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3064)
         at
 ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3039)
         at
 ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2705)
         at
 ch.cyberduck.ui.cocoa.controller.BrowserController$AbstractBrowserTableDelegate.tableRowDoubleClicked(BrowserController.java:3597)
         at java.base/jdk.internal.reflect.NativeMethodAccessorI
 2020-03-17 13:00:23.199617-0400 0x7427b8   Info        0xaecc6e
 88140  0    Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40
 ch.cyberduck.core.threading.SessionBackgroundAction] Run alert callback
 ch.cyberduck.ui.cocoa.callback.PromptAlertCallback at 8102284 for failure
 BackgroundException{class=class
 ch.cyberduck.core.exception.ConnectionCanceledException, file=null,
 message='Error', detail='Unknown application error. URLDecoder: Illegal
 hex characters in escape (%) pattern - Error at index 1 in: "ap".',
 cause='java.lang.IllegalArgumentException: URLDecoder: Illegal hex
 characters in escape (%) pattern - Error at index 1 in: "ap"'}
 ```

 I'm having issues building Cyberduck on macOS 10.15.3 (most likely due to
 my own incompetence) so I can't quite debug the exact issue.  If I can
 figure that out, I'd can attempt to create a patch fixing or working
 around my issue.

 Reading through code, I think it may be related to https://github.com
 /iterate-ch/cyberduck/commit/07e0e548684b7a65f2d1bcbc424da639ffeb4396.

 I'm not blocked since I have a work around (using 7.0.2) but figured that
 I might create a ticket for this anyway to track.

--
Ticket URL: <https://trac.cyberduck.io/ticket/10995>
Cyberduck <https://cyberduck.io>
Libre FTP, SFTP, WebDAV, S3 & OpenStack Swift browser for Mac and Windows