[Cyberduck-trac] [Cyberduck] #9662: Support SHA256 checksums and use checksum verification built into Jargon/iRODS protocol (was: Support SHA256 checksums)

[Cyberduck]

#9662: Support SHA256 checksums and use checksum verification built into
Jargon/iRODS protocol
--------------------------+-------------------------
 Reporter:  carsten.jahn  |         Owner:  dkocher
     Type:  enhancement   |        Status:  assigned
 Priority:  normal        |     Milestone:
Component:  irods         |       Version:  5.0.11
 Severity:  normal        |    Resolution:
 Keywords:                |  Architecture:
 Platform:                |
--------------------------+-------------------------

Comment (by carsten.jahn):

 I wanted to add that I tested Cyberduck 5.1.0 upload with iRODS 4.1.9
 today in more depth, with Cyberduck debug log enabled.

 Cyberduck does this call:
 ```org.irods.jargon.core.pub.CollectionAndDataObjectListAndSearchAOImpl -
 ObjStat ...```
 for retrieving the file size and checksum from iRODS after transfer.

 The iRODS/Jargon response will contain a checksum only if an iRODS server
 rule for calculating the checksum on upload is defined (`acPostProcForPut
 {msiSysChksumDataObj; }`) - otherwise, the returned checksum is empty
 (`...dataId=321827, checksum=, ownerName=...`).  The usage of
 `msiSysChksumDataObj` is not necessary if the iRODS client submits a
 checksum during upload - the client-calculated checksum will be
 transferred along with the data and the server will check it upon receipt.
 However, because Cyberduck does
 `setComputeAndVerifyChecksumAfterTransfer(false)`, the Jargon default
 behavior (as defined in `jargon.properties`) is overridden and Jargon does
 not send a checksum. Thus, when Cyberduck asks for the object checksum, it
 does not receive any.

 The iRODS default rule file does not contain the `msiSysChksumDataObj`
 call, but Cyberduck seems to depend on it, as well as on the MD5 checksum
 algorithm setting, which is SHA256 on modern iRODS servers. Switching back
 to MD5 and including the `msiSysChksumDataObj` rule, I can verify that
 Cyberduck does not complain about the checksum any more.

 There is another dependency on server configuration built in: the ObjStat
 information contains an empty iRODS checksum even if `msiSysChksumDataObj`
 is present, in the case that Cyberduck writes to a 'compound' iRODS
 resource. This may be an iRODS or Jargon bug.

 I think the ideal implementation would be to remove
 `setComputeAndVerifyChecksumAfterTransfer(false)` and
 `setComputeChecksumAfterTransfer(false)`, so that the default
 `transfer.computeandvalidate.checksum=true` from `jargon.properties` is in
 effect. Do not attempt to parse iRODS checksums and compare it to a self-
 computed checksum.  Rather, have Jargon handle this internally. Thus,
 Cyberduck would automatically support all known iRODS checksum algorithms
 and wouldn't depend on a particular server-side rule being present.

 Maybe the Jargon verify feature was turned off for performance reasons -
 then it would be good to have this configurable for people who would like
 to use the Jargon default, maybe as a "hidden option" somewhere.

--
Ticket URL: <https://trac.cyberduck.io/ticket/9662#comment:8>
Cyberduck <https://cyberduck.io>
Libre FTP, SFTP, WebDAV, S3 & OpenStack Swift browser for Mac and Windows