[Cyberduck-trac] [Cyberduck] #10995: ListObjects for directories over 1000 files doesn't work for S3 compatible storage
Cyberduck
trac at cyberduck.io
Tue Mar 17 17:28:46 UTC 2020
#10995: ListObjects for directories over 1000 files doesn't work for S3 compatible
storage
-------------------------+--------------------
Reporter: skier5 | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: s3 | Version: 7.1
Severity: normal | Resolution:
Keywords: | Architecture:
Platform: macOS 10.15 |
-------------------------+--------------------
Description changed by skier5:
Old description:
> Starting with Cyberduck 7.1.0, I've noticed that when connecting to a
> private S3 compatible storage service, I can no longer list files in a
> directory where there are over 1000 files/keys present. The UI presents
> itself as if there were no files in the directory.
>
> In Cyberduck 7.0.2, executing the same scenario results in displaying the
> >1000 list of keys.
>
> When turning on debugging, I see the following:
>
> > GET /bucket/?encoding-type=url&max-
> keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F HTTP/1.1
> This returns a 200.
>
> I also see
> > GET /bucket/?encoding-type=url&max-
> keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F&marker=path%2Fto%2Flarge%2Fdirectory%2Fmarker.json
> HTTP/1.1
> This also returns a 200.
>
> Then I see the following
> ```
> 2020-03-17 13:00:23.199539-0400 0x7427b8 Default 0xaecc6e
> 88140 0 Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40
> ch.cyberduck.core.threading.BackgroundCallable] Failure
> java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in
> escape (%) pattern - Error at index 1 in: "ap" running background task
> java.lang.Exception
> at
> ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
> at
> ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
> at
> ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
> at
> ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:517)
> at
> ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3064)
> at
> ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3039)
> at
> ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2705)
> at
> ch.cyberduck.ui.cocoa.controller.BrowserController$AbstractBrowserTableDelegate.tableRowDoubleClicked(BrowserController.java:3597)
> at java.base/jdk.internal.reflect.NativeMethodAccessorI
> 2020-03-17 13:00:23.199617-0400 0x7427b8 Info 0xaecc6e
> 88140 0 Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40
> ch.cyberduck.core.threading.SessionBackgroundAction] Run alert callback
> ch.cyberduck.ui.cocoa.callback.PromptAlertCallback at 8102284 for failure
> BackgroundException{class=class
> ch.cyberduck.core.exception.ConnectionCanceledException, file=null,
> message='Error', detail='Unknown application error. URLDecoder: Illegal
> hex characters in escape (%) pattern - Error at index 1 in: "ap".',
> cause='java.lang.IllegalArgumentException: URLDecoder: Illegal hex
> characters in escape (%) pattern - Error at index 1 in: "ap"'}
> ```
>
> I'm having issues building Cyberduck on macOS 10.15.3 (most likely due to
> my own incompetence) so I can't quite debug the exact issue. If I can
> figure that out, I'd can attempt to create a patch fixing or working
> around my issue.
>
> Reading through code, I think it may be related to https://github.com
> /iterate-ch/cyberduck/commit/07e0e548684b7a65f2d1bcbc424da639ffeb4396.
>
> I'm not blocked since I have a work around (using 7.0.2) but figured that
> I might create a ticket for this anyway to track.
New description:
Starting with Cyberduck 7.1.0, I've noticed that when connecting to a
private S3 compatible storage service, I can no longer list files in a
directory where there are over 1000 files/keys present. The UI presents
itself as if there were no files in the directory.
In Cyberduck 7.0.2, executing the same scenario results in displaying the
>1000 list of keys.
When turning on debugging, I see the following:
> GET /bucket/?encoding-type=url&max-
keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F HTTP/1.1
This returns a 200.
I also see
> GET /bucket/?encoding-type=url&max-
keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F&marker=path%2Fto%2Flarge%2Fdirectory%2Fmarker.json
HTTP/1.1
This also returns a 200.
Then I see the following
```
2020-03-17 13:00:23.199539-0400 0x7427b8 Default 0xaecc6e
88140 0 Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40
ch.cyberduck.core.threading.BackgroundCallable] Failure
java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in
escape (%) pattern - Error at index 1 in: "ap" running background task
java.lang.Exception
at
ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
at
ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
at
ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
at
ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:517)
at
ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3064)
at
ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3039)
at
ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2705)
at
ch.cyberduck.ui.cocoa.controller.BrowserController$AbstractBrowserTableDelegate.tableRowDoubleClicked(BrowserController.java:3597)
at java.base/jdk.internal.reflect.NativeMethodAccessorI
2020-03-17 13:00:23.199617-0400 0x7427b8 Info 0xaecc6e
88140 0 Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40
ch.cyberduck.core.threading.SessionBackgroundAction] Run alert callback
ch.cyberduck.ui.cocoa.callback.PromptAlertCallback at 8102284 for failure
BackgroundException{class=class
ch.cyberduck.core.exception.ConnectionCanceledException, file=null,
message='Error', detail='Unknown application error. URLDecoder: Illegal
hex characters in escape (%) pattern - Error at index 1 in: "ap".',
cause='java.lang.IllegalArgumentException: URLDecoder: Illegal hex
characters in escape (%) pattern - Error at index 1 in: "ap"'}
```
I'm having issues building Cyberduck on macOS 10.15.3 (most likely due to
my own incompetence) so I can't quite debug the exact issue. If I can
figure that out, I can attempt to create a patch fixing or working around
my issue.
Reading through code, I think it may be related to https://github.com
/iterate-ch/cyberduck/commit/07e0e548684b7a65f2d1bcbc424da639ffeb4396.
I'm not blocked since I have a work around (using 7.0.2) but figured that
I might create a ticket for this anyway to track.
--
--
Ticket URL: <https://trac.cyberduck.io/ticket/10995#comment:1>
Cyberduck <https://cyberduck.io>
Libre FTP, SFTP, WebDAV, S3 & OpenStack Swift browser for Mac and Windows
More information about the Cyberduck-trac
mailing list