[Cyberduck-trac] [Cyberduck] #10392: Out of memory error when uploading larger files

[Cyberduck]

#10392: Out of memory error when uploading larger files
------------------------+------------------------------
 Reporter:  Th3Dan      |         Owner:
     Type:  defect      |        Status:  new
 Priority:  high        |     Milestone:  7.0
Component:  onedrive    |       Version:  Nightly Build
 Severity:  major       |    Resolution:
 Keywords:              |  Architecture:  Intel
 Platform:  Windows 10  |
------------------------+------------------------------

Comment (by Th3Dan):

 I ran two tests:

 Test 1
 4 files with 2x7,1GB, 1x1,8GB and 1x1,7GB => total about ~17GB
 This upload was done during night without errors. Ram usage between
 80-100MB at beginning. When reaching 16 percent, it increased to
 ~140-160MB. Later I went to bed so no data exists. It was surprising but I
 remember that I changed the settings in Filetransfer > General > File
 transfers to "Use connection in browser". This was done several months ago
 to figure out if may wrong settings are the reason why large uploads
 doesn't work.

 Test 2
 29,5GB total in multiple ISO images from 4 to 7GB
 Here the file transfer settings were changed to "Open multiple
 connections", which was the default I think. I started the uplad in the
 morning before I went to work, today in the evening it was aborted. File
 transfer window say "transfer incomplete" at 15,9 from 29,6GB. and in the
 line above "Resolve graph.microsoft.com".

 The cyberduck.log in appdata folder has an entry near 3PM:


 {{{
 2018-11-16 14:54:52,405 [background-11] ERROR AsyncController - Unhandled
 exception during invoke
 cli.System.OutOfMemoryException
 2018-11-16 14:54:52,421 [cbafInh0-transfer-2] ERROR
 ch.cyberduck.core.Resolver - Waiting for resolving of graph.microsoft.com
 java.lang.InterruptedException
         at
 java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1324)
         at
 java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
         at ch.cyberduck.core.Resolver.resolve(Resolver.java:78)
         at
 ch.cyberduck.core.LoginConnectionService.connect(LoginConnectionService.java:133)
         at
 ch.cyberduck.core.LoginConnectionService.check(LoginConnectionService.java:102)
         at
 ch.cyberduck.core.pool.StatelessSessionPool.borrow(StatelessSessionPool.java:71)
         at
 ch.cyberduck.core.worker.ConcurrentTransferWorker.borrow(ConcurrentTransferWorker.java:128)
         at
 ch.cyberduck.core.worker.AbstractTransferWorker$3.call(AbstractTransferWorker.java:381)
         at
 ch.cyberduck.core.worker.AbstractTransferWorker$3.call(AbstractTransferWorker.java:371)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
         at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:512)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
         at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
         at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
         at
 ch.cyberduck.core.threading.NamedThreadFactory$1.run(NamedThreadFactory.java:58)
         at java.lang.Thread.run(Thread.java:955)
 }}}

 Again you can see that I got an OutOfMemoryException diretly before the
 DNS resolve error. This can't be related to my memory since my machine has
 32GB memory. Currently about 22GB of them are usable. BUT since Cyberduck
 is a 32 Bit application, it can't allocate more than ~4GB for a single
 process.

 To be sure, I looked in the logs of my DSL router. There were no entrys in
 the hole day, so the WAN connection seems stable. This doesn't
 automatically mean that there were no outages by the DNS server from the
 ISP. But on the other hand, even if there was a DNS failture: Why does
 this abort the hole upload process? Would expect that in this case
 Cyberduck wait a minute and re-try the upload process.

 And I don't really belive that the DNS issue is the cause since the
 OutOfMemoryException was thrown only milliseconds before. It seems that
 for some reason the app ran out of memory and the DNS error is some
 aftereffect of this.

--
Ticket URL: <https://trac.cyberduck.io/ticket/10392#comment:11>
Cyberduck <https://cyberduck.io>
Libre FTP, SFTP, WebDAV, S3 & OpenStack Swift browser for Mac and Windows