Transferring Large Files

From CCI User Wiki
Jump to: navigation, search

Generate Checksum

 md5sum someLargeFile

Split

 split -b Nm someLargeFile PREFIX

where N is the size of each chunk in MB and PREFIX is the prefix for the created chunks.

Join

 cat PREFIX* > someLargeFile2

Transfer

Run multiple instances of scp to transfer the chunks. At some point adding more scp instances will not increase the total throughput.

Check

 md5sum someLargeFile2

Compare this output against the original files checksum. It should be exactly the same.