Parallel Copy from local folders to remote servers at the same time
up vote
1
down vote
favorite
I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist.
Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.
I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.
What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that either prsync or rdist, don't copy multiple files in parallel on single host, they can only copy file by file in parallel on multiple hosts in parallel.So is there any different between the two from performance side?
for rdist, my scripts create a distfile like
HOSTS( user@server user@server2 user@server3 )
RUN:(/var/inputpath/folder)
-> ${HOSTS}
install (/var/outputpath/folder)
then I run rdist like following
rdist-f /dist-file-path -P /pathto ssh
I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.
So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?
rsync gnu-parallel pscp
New contributor
add a comment |
up vote
1
down vote
favorite
I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist.
Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.
I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.
What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that either prsync or rdist, don't copy multiple files in parallel on single host, they can only copy file by file in parallel on multiple hosts in parallel.So is there any different between the two from performance side?
for rdist, my scripts create a distfile like
HOSTS( user@server user@server2 user@server3 )
RUN:(/var/inputpath/folder)
-> ${HOSTS}
install (/var/outputpath/folder)
then I run rdist like following
rdist-f /dist-file-path -P /pathto ssh
I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.
So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?
rsync gnu-parallel pscp
New contributor
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist.
Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.
I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.
What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that either prsync or rdist, don't copy multiple files in parallel on single host, they can only copy file by file in parallel on multiple hosts in parallel.So is there any different between the two from performance side?
for rdist, my scripts create a distfile like
HOSTS( user@server user@server2 user@server3 )
RUN:(/var/inputpath/folder)
-> ${HOSTS}
install (/var/outputpath/folder)
then I run rdist like following
rdist-f /dist-file-path -P /pathto ssh
I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.
So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?
rsync gnu-parallel pscp
New contributor
I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist.
Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.
I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.
What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that either prsync or rdist, don't copy multiple files in parallel on single host, they can only copy file by file in parallel on multiple hosts in parallel.So is there any different between the two from performance side?
for rdist, my scripts create a distfile like
HOSTS( user@server user@server2 user@server3 )
RUN:(/var/inputpath/folder)
-> ${HOSTS}
install (/var/outputpath/folder)
then I run rdist like following
rdist-f /dist-file-path -P /pathto ssh
I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.
So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?
rsync gnu-parallel pscp
rsync gnu-parallel pscp
New contributor
New contributor
edited Nov 18 at 5:37
Rui F Ribeiro
38.2k1475123
38.2k1475123
New contributor
asked Nov 17 at 22:09
Askj
61
61
New contributor
New contributor
add a comment |
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Askj is a new contributor. Be nice, and check out our Code of Conduct.
Askj is a new contributor. Be nice, and check out our Code of Conduct.
Askj is a new contributor. Be nice, and check out our Code of Conduct.
Askj is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f482407%2fparallel-copy-from-local-folders-to-remote-servers-at-the-same-time%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown