[Cyberduck-trac] [Cyberduck] #7732: openstack bulk delete can appear to timeout
Cyberduck
trac at trac.cyberduck.io
Sun Jan 12 18:01:44 UTC 2014
#7732: openstack bulk delete can appear to timeout
---------------------------+---------------------------
Reporter: david cole | Owner: dkocher
Type: defect | Status: new
Priority: normal | Milestone: 4.4.4
Component: openstack | Version: 4.4.3
Severity: major | Keywords: bulk
Architecture: Intel | Platform: Mac OS X 10.9
---------------------------+---------------------------
As I understand it, the bulk-delete multi object deletion in openstack
swift takes a list of objects and does not return until the objects have
been removed (assuming success). I don't believe there's any network I/O
while the bulk-delete is working, so if the operation takes more than the
connection timeout value ( preference connection.timeout.seconds) then the
connection is closed locally. I had the default 30second timeout set
when I ran into this:
{{{
$ defaults read ch.sudo.cyberduck connection.timeout.seconds
30
}}}
Here's the log messages I saw when deleting 500 objects:
{{{
2014-01-11 14:40:00,573 [background-1] INFO ch.cyberduck.core.Session -
Deleting 1
2014-01-11 14:40:00,574 [background-1] INFO ch.cyberduck.core.Session -
Deleting 10
....
.... < 500 other files >
....
2014-01-11 14:40:00,685 [background-1] INFO ch.cyberduck.core.Session -
Deleting 99
2014-01-11 14:40:00,685 [background-1] INFO ch.cyberduck.core.Session -
Deleting foo
2014-01-11 14:40:00,685 [background-1] INFO ch.cyberduck.core.Session -
Deleting test_25meg_files
2014-01-11 14:40:00,685 [background-1] WARN
ch.cyberduck.core.openstack.SwiftRegionService - Unknown region null in
authentication context
2014-01-11 14:40:00,685 [background-1] WARN
ch.cyberduck.core.openstack.SwiftRegionService - Fallback to first region
found null
2014-01-11 14:40:00,685 [background-1] WARN
ch.cyberduck.core.openstack.SwiftRegionService - No CDN management
endpoint found for region null
2014-01-11 14:40:00,705 [background-1] DEBUG
org.apache.http.impl.conn.PoolingClientConnectionManager - Connection
request: [route: {s}->https://proxy.example.com][total kept alive: 29;
route allocated: 28 of 2147483647; total allocated: 29 of 2147483647]
2014-01-11 14:40:00,706 [background-1] DEBUG
org.apache.http.impl.conn.PoolingClientConnectionManager - Connection
leased: [id: 28][route: {s}->https://proxy.example.com][total kept alive:
28; route allocated: 28 of 2147483647; total allocated: 29 of 2147483647]
2014-01-11 14:40:00,706 [background-1] DEBUG
org.apache.http.impl.client.DefaultHttpClient - Stale connection check
2014-01-11 14:40:00,707 [background-1] DEBUG
org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected:
best-match
2014-01-11 14:40:00,707 [background-1] DEBUG
org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in
the context
2014-01-11 14:40:00,707 [background-1] DEBUG
org.apache.http.client.protocol.RequestTargetAuthentication - Target auth
state: UNCHALLENGED
2014-01-11 14:40:00,707 [background-1] DEBUG
org.apache.http.client.protocol.RequestProxyAuthentication - Proxy auth
state: UNCHALLENGED
2014-01-11 14:40:00,707 [background-1] INFO ch.cyberduck.core.Session -
DELETE /v1/AUTH_yyyyyyyyyyyyyyyyyyyy/roadtrip?bulk-delete=1 HTTP/1.1
2014-01-11 14:40:00,707 [background-1] INFO ch.cyberduck.core.Session -
Content-Type: text/plain
2014-01-11 14:40:00,708 [background-1] INFO ch.cyberduck.core.Session - X
-Auth-Token: xxxxxxxxxxxx
2014-01-11 14:40:00,708 [background-1] INFO ch.cyberduck.core.Session -
Content-Length: 14606
2014-01-11 14:40:00,708 [background-1] INFO ch.cyberduck.core.Session -
Host: proxy.example.com
2014-01-11 14:40:00,708 [background-1] INFO ch.cyberduck.core.Session -
Connection: Keep-Alive
2014-01-11 14:40:00,708 [background-1] INFO ch.cyberduck.core.Session -
User-Agent: Cyberduck/4.4.3 (Mac OS X/10.9) (x86_64)
2014-01-11 14:40:00,708 [background-1] DEBUG
org.apache.http.impl.client.DefaultHttpClient - Attempt 1 to execute
request
2014-01-11 14:40:00,708 [background-1] DEBUG
org.apache.http.impl.conn.DefaultClientConnection - Sending request:
DELETE /v1/AUTH_yyyyyyyyyyy/roadtrip?bulk-delete=1 HTTP/1.1
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >>
DELETE /v1/AUTH_yyyyyyyyyyyyy/roadtrip?bulk-delete=1 HTTP/1.1
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >>
Content-Type: text/plain
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >>
X-Auth-Token: xxxxxxxxxx
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >>
Content-Length: 14606
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >>
Host: proxy.example.com
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >>
Connection: Keep-Alive
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >>
User-Agent: Cyberduck/4.4.3 (Mac OS X/10.9) (x86_64)
2014-01-11 14:40:00,708 [background-1] DEBUG org.apache.http.headers - >>
Accept-Encoding: gzip,deflate
2014-01-11 14:40:30,711 [background-1] DEBUG
org.apache.http.impl.conn.DefaultClientConnection - Connection
0.0.0.0:56108<->193.203.66.227:443 closed
2014-01-11 14:40:30,711 [background-1] DEBUG
org.apache.http.impl.client.DefaultHttpClient - Closing the connection.
2014-01-11 14:40:30,711 [background-1] DEBUG
org.apache.http.impl.conn.DefaultClientConnection - Connection
0.0.0.0:56108<->193.203.66.227:443 closed
2014-01-11 14:40:30,712 [background-1] DEBUG
org.apache.http.impl.conn.DefaultClientConnection - Connection
0.0.0.0:56108<->193.203.66.227:443 shut down
2014-01-11 14:40:30,712 [background-1] DEBUG
org.apache.http.impl.conn.DefaultClientConnection - Connection
0.0.0.0:56108<->193.203.66.227:443 closed
2014-01-11 14:40:30,712 [background-1] DEBUG
org.apache.http.impl.conn.PoolingClientConnectionManager - Connection
released: [id: 28][route: {s}->https://proxy.example.com][total kept
alive: 28; route allocated: 27 of 2147483647; total allocated: 28 of
2147483647]
2014-01-11 14:40:30,714 [background-1] WARN
ch.cyberduck.core.threading.SessionBackgroundAction - Failure executing
background action: Cannot delete 1 (/roadtrip/test_25meg_files/1). Read
timed out.
2014-01-11 14:40:30,715 [background-1] INFO
ch.cyberduck.core.threading.SessionBackgroundAction - Retry failed
background action
WorkerBackgroundAction{worker=ch.cyberduck.ui.cocoa.BrowserController$20 at 718bdaf8,
result=null}
}}}
Notice it's 30seconds from :[[BR]]
2014-01-11 '''14:40:00''',708 [background-1] DEBUG org.apache.http.headers
- >> Accept-Encoding: gzip,deflate[[BR]]
to[[BR]]
2014-01-11 '''14:40:30''',711 [background-1] DEBUG
org.apache.http.impl.conn.DefaultClientConnection - '''Connection
0.0.0.0:56108<->193.203.66.227:443 closed'''
The 5 retries did not help in this case. It's properly a bad idea to re-
send a duplicate bulk-delete request several times...
Setting connection.timeout.seconds to 600 seconds helped me to delete my
large data set. Would love to get a fix for this so we can delete 50000+
objects and still retain the default 30 sec connection timeout
--
Ticket URL: <https://trac.cyberduck.io/ticket/7732>
Cyberduck <http://cyberduck.io>
Libre FTP, SFTP, WebDAV, S3 & OpenStack Swift browser for Mac and Windows
More information about the Cyberduck-trac
mailing list