Transferring Large Files

From CCI User Wiki
Jump to: navigation, search

Generate Checksum

 md5sum someLargeFile


 split -b Nm someLargeFile PREFIX

where N is the size of each chunk in MB and PREFIX is the prefix for the created chunks.


 cat PREFIX* > someLargeFile2


Run multiple instances of scp to transfer the chunks. At some point adding more scp instances will not increase the total throughput.


 md5sum someLargeFile2

Compare this output against the original files checksum. It should be exactly the same.