Parallel Copy from local folders to remote servers at the same time











up vote
1
down vote

favorite












I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist.
Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.



I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.





  1. What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that either prsync or rdist, don't copy multiple files in parallel on single host, they can only copy file by file in parallel on multiple hosts in parallel.So is there any different between the two from performance side?
    for rdist, my scripts create a distfile like
    HOSTS( user@server user@server2 user@server3 )
    RUN:(/var/inputpath/folder)
    -> ${HOSTS}
    install (/var/outputpath/folder)



    then I run rdist like following
    rdist-f /dist-file-path -P /pathto ssh



  2. I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.



So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?










share|improve this question









New contributor




Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
























    up vote
    1
    down vote

    favorite












    I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist.
    Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.



    I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.





    1. What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that either prsync or rdist, don't copy multiple files in parallel on single host, they can only copy file by file in parallel on multiple hosts in parallel.So is there any different between the two from performance side?
      for rdist, my scripts create a distfile like
      HOSTS( user@server user@server2 user@server3 )
      RUN:(/var/inputpath/folder)
      -> ${HOSTS}
      install (/var/outputpath/folder)



      then I run rdist like following
      rdist-f /dist-file-path -P /pathto ssh



    2. I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.



    So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?










    share|improve this question









    New contributor




    Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.






















      up vote
      1
      down vote

      favorite









      up vote
      1
      down vote

      favorite











      I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist.
      Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.



      I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.





      1. What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that either prsync or rdist, don't copy multiple files in parallel on single host, they can only copy file by file in parallel on multiple hosts in parallel.So is there any different between the two from performance side?
        for rdist, my scripts create a distfile like
        HOSTS( user@server user@server2 user@server3 )
        RUN:(/var/inputpath/folder)
        -> ${HOSTS}
        install (/var/outputpath/folder)



        then I run rdist like following
        rdist-f /dist-file-path -P /pathto ssh



      2. I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.



      So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?










      share|improve this question









      New contributor




      Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist.
      Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.



      I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.





      1. What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that either prsync or rdist, don't copy multiple files in parallel on single host, they can only copy file by file in parallel on multiple hosts in parallel.So is there any different between the two from performance side?
        for rdist, my scripts create a distfile like
        HOSTS( user@server user@server2 user@server3 )
        RUN:(/var/inputpath/folder)
        -> ${HOSTS}
        install (/var/outputpath/folder)



        then I run rdist like following
        rdist-f /dist-file-path -P /pathto ssh



      2. I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.



      So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?







      rsync gnu-parallel pscp






      share|improve this question









      New contributor




      Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      share|improve this question









      New contributor




      Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this question




      share|improve this question








      edited Nov 18 at 5:37









      Rui F Ribeiro

      38.2k1475123




      38.2k1475123






      New contributor




      Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      asked Nov 17 at 22:09









      Askj

      61




      61




      New contributor




      Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      Askj is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.



























          active

          oldest

          votes











          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "106"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });






          Askj is a new contributor. Be nice, and check out our Code of Conduct.










           

          draft saved


          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f482407%2fparallel-copy-from-local-folders-to-remote-servers-at-the-same-time%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown






























          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          Askj is a new contributor. Be nice, and check out our Code of Conduct.










           

          draft saved


          draft discarded


















          Askj is a new contributor. Be nice, and check out our Code of Conduct.













          Askj is a new contributor. Be nice, and check out our Code of Conduct.












          Askj is a new contributor. Be nice, and check out our Code of Conduct.















           


          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f482407%2fparallel-copy-from-local-folders-to-remote-servers-at-the-same-time%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Morgemoulin

          Scott Moir

          Souastre