Transferring Large Files
Very large files can be transferred by splitting them into multiple files and copying them in parallel.
- Generate Checksum
split -b Nm someLargeFile PREFIXwhere N is the size of each chunk in MB and PREFIX is the prefix for the created chunks.
- Transfer Run multiple instances of scp to transfer the chunks. At some point adding more scp instances will not increase the total throughput.
cat PREFIX* > someLargeFile2
md5sum someLargeFile2Compare this output against the original files checksum. It should be exactly the same.
Last update: May 14, 2020