Move the oldest files into a directory, source directory has more than 10,000 files constantly incoming












0














I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find" result into "ls" or just use the "ls" .. please suggest.



ls -ltr `/bin/find ${in}/* -prune -name "*.txt"  -type f` | head -10 |     
while read -r infile ; do
-move the file
done


or should i just use ls.
the reason i am using find is: i read some online content that ls should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls.
i am worried if find result is too big would it might cause any problem.










share|improve this question
























  • I noticed you tagged ksh and hp-ux; is zsh an option?
    – Jeff Schaller
    Dec 28 '18 at 19:35










  • No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
    – user3342678
    Dec 28 '18 at 20:08










  • unix.stackexchange.com/a/449771/117549 may be useful
    – Jeff Schaller
    Dec 28 '18 at 20:32










  • 10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
    – pbhj
    Dec 29 '18 at 1:22










  • If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
    – Jeff Schaller
    2 days ago
















0














I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find" result into "ls" or just use the "ls" .. please suggest.



ls -ltr `/bin/find ${in}/* -prune -name "*.txt"  -type f` | head -10 |     
while read -r infile ; do
-move the file
done


or should i just use ls.
the reason i am using find is: i read some online content that ls should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls.
i am worried if find result is too big would it might cause any problem.










share|improve this question
























  • I noticed you tagged ksh and hp-ux; is zsh an option?
    – Jeff Schaller
    Dec 28 '18 at 19:35










  • No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
    – user3342678
    Dec 28 '18 at 20:08










  • unix.stackexchange.com/a/449771/117549 may be useful
    – Jeff Schaller
    Dec 28 '18 at 20:32










  • 10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
    – pbhj
    Dec 29 '18 at 1:22










  • If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
    – Jeff Schaller
    2 days ago














0












0








0







I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find" result into "ls" or just use the "ls" .. please suggest.



ls -ltr `/bin/find ${in}/* -prune -name "*.txt"  -type f` | head -10 |     
while read -r infile ; do
-move the file
done


or should i just use ls.
the reason i am using find is: i read some online content that ls should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls.
i am worried if find result is too big would it might cause any problem.










share|improve this question















I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find" result into "ls" or just use the "ls" .. please suggest.



ls -ltr `/bin/find ${in}/* -prune -name "*.txt"  -type f` | head -10 |     
while read -r infile ; do
-move the file
done


or should i just use ls.
the reason i am using find is: i read some online content that ls should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls.
i am worried if find result is too big would it might cause any problem.







shell-script ls ksh mv hp-ux






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Dec 28 '18 at 19:52









Rui F Ribeiro

39.3k1479131




39.3k1479131










asked Dec 28 '18 at 18:55









user3342678user3342678

1




1












  • I noticed you tagged ksh and hp-ux; is zsh an option?
    – Jeff Schaller
    Dec 28 '18 at 19:35










  • No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
    – user3342678
    Dec 28 '18 at 20:08










  • unix.stackexchange.com/a/449771/117549 may be useful
    – Jeff Schaller
    Dec 28 '18 at 20:32










  • 10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
    – pbhj
    Dec 29 '18 at 1:22










  • If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
    – Jeff Schaller
    2 days ago


















  • I noticed you tagged ksh and hp-ux; is zsh an option?
    – Jeff Schaller
    Dec 28 '18 at 19:35










  • No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
    – user3342678
    Dec 28 '18 at 20:08










  • unix.stackexchange.com/a/449771/117549 may be useful
    – Jeff Schaller
    Dec 28 '18 at 20:32










  • 10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
    – pbhj
    Dec 29 '18 at 1:22










  • If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
    – Jeff Schaller
    2 days ago
















I noticed you tagged ksh and hp-ux; is zsh an option?
– Jeff Schaller
Dec 28 '18 at 19:35




I noticed you tagged ksh and hp-ux; is zsh an option?
– Jeff Schaller
Dec 28 '18 at 19:35












No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
– user3342678
Dec 28 '18 at 20:08




No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
– user3342678
Dec 28 '18 at 20:08












unix.stackexchange.com/a/449771/117549 may be useful
– Jeff Schaller
Dec 28 '18 at 20:32




unix.stackexchange.com/a/449771/117549 may be useful
– Jeff Schaller
Dec 28 '18 at 20:32












10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
– pbhj
Dec 29 '18 at 1:22




10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
– pbhj
Dec 29 '18 at 1:22












If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
– Jeff Schaller
2 days ago




If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
– Jeff Schaller
2 days ago










2 Answers
2






active

oldest

votes


















0














While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.



The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *. The inner loop then compares every file's timestamp to see if it is older than (-ot) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.



for((count=0; count < 3; count++))
do
set -- *
oldest=$1
for f in ./*
do
if [ "$f" -ot "$oldest" ]
then
oldest=$f
fi
done
echo mv -- "$oldest" ./.archive
mv -- "$oldest" ./.archive
done





share|improve this answer





















  • Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just ls command , i could just use ls -ltr | head -x then for each file will do the move operation.
    – user3342678
    Dec 28 '18 at 21:52



















0














Python heapq.nsmallest



bash:



find -printf '%T@ %pn' |
python noldest.py 1000 |
xargs -Ixxx mv xxx directory_for_old_files


Here find command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs command takes filenames from stdin one by one and applies mv xxx directory_for_old_files command substituting xxx with them.



noldest.py implementation could be:



import sys
from heapq import nsmallest
from datetime import datetime


n = int(sys.argv[1])
date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy

def parse_date(date_and_filename):
seconds_from_1970 = float(date_and_filename[0])
return datetime.fromtimestamp(int(seconds_from_1970))

for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines


The perfomance depends on implementation of nsmallest algorithm from python standard library.






share|improve this answer





















    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "106"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f491356%2fmove-the-oldest-files-into-a-directory-source-directory-has-more-than-10-000-fi%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.



    The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *. The inner loop then compares every file's timestamp to see if it is older than (-ot) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.



    for((count=0; count < 3; count++))
    do
    set -- *
    oldest=$1
    for f in ./*
    do
    if [ "$f" -ot "$oldest" ]
    then
    oldest=$f
    fi
    done
    echo mv -- "$oldest" ./.archive
    mv -- "$oldest" ./.archive
    done





    share|improve this answer





















    • Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just ls command , i could just use ls -ltr | head -x then for each file will do the move operation.
      – user3342678
      Dec 28 '18 at 21:52
















    0














    While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.



    The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *. The inner loop then compares every file's timestamp to see if it is older than (-ot) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.



    for((count=0; count < 3; count++))
    do
    set -- *
    oldest=$1
    for f in ./*
    do
    if [ "$f" -ot "$oldest" ]
    then
    oldest=$f
    fi
    done
    echo mv -- "$oldest" ./.archive
    mv -- "$oldest" ./.archive
    done





    share|improve this answer





















    • Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just ls command , i could just use ls -ltr | head -x then for each file will do the move operation.
      – user3342678
      Dec 28 '18 at 21:52














    0












    0








    0






    While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.



    The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *. The inner loop then compares every file's timestamp to see if it is older than (-ot) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.



    for((count=0; count < 3; count++))
    do
    set -- *
    oldest=$1
    for f in ./*
    do
    if [ "$f" -ot "$oldest" ]
    then
    oldest=$f
    fi
    done
    echo mv -- "$oldest" ./.archive
    mv -- "$oldest" ./.archive
    done





    share|improve this answer












    While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.



    The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *. The inner loop then compares every file's timestamp to see if it is older than (-ot) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.



    for((count=0; count < 3; count++))
    do
    set -- *
    oldest=$1
    for f in ./*
    do
    if [ "$f" -ot "$oldest" ]
    then
    oldest=$f
    fi
    done
    echo mv -- "$oldest" ./.archive
    mv -- "$oldest" ./.archive
    done






    share|improve this answer












    share|improve this answer



    share|improve this answer










    answered Dec 28 '18 at 20:45









    Jeff SchallerJeff Schaller

    39k1054125




    39k1054125












    • Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just ls command , i could just use ls -ltr | head -x then for each file will do the move operation.
      – user3342678
      Dec 28 '18 at 21:52


















    • Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just ls command , i could just use ls -ltr | head -x then for each file will do the move operation.
      – user3342678
      Dec 28 '18 at 21:52
















    Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just ls command , i could just use ls -ltr | head -x then for each file will do the move operation.
    – user3342678
    Dec 28 '18 at 21:52




    Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just ls command , i could just use ls -ltr | head -x then for each file will do the move operation.
    – user3342678
    Dec 28 '18 at 21:52













    0














    Python heapq.nsmallest



    bash:



    find -printf '%T@ %pn' |
    python noldest.py 1000 |
    xargs -Ixxx mv xxx directory_for_old_files


    Here find command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs command takes filenames from stdin one by one and applies mv xxx directory_for_old_files command substituting xxx with them.



    noldest.py implementation could be:



    import sys
    from heapq import nsmallest
    from datetime import datetime


    n = int(sys.argv[1])
    date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy

    def parse_date(date_and_filename):
    seconds_from_1970 = float(date_and_filename[0])
    return datetime.fromtimestamp(int(seconds_from_1970))

    for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
    print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines


    The perfomance depends on implementation of nsmallest algorithm from python standard library.






    share|improve this answer


























      0














      Python heapq.nsmallest



      bash:



      find -printf '%T@ %pn' |
      python noldest.py 1000 |
      xargs -Ixxx mv xxx directory_for_old_files


      Here find command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs command takes filenames from stdin one by one and applies mv xxx directory_for_old_files command substituting xxx with them.



      noldest.py implementation could be:



      import sys
      from heapq import nsmallest
      from datetime import datetime


      n = int(sys.argv[1])
      date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy

      def parse_date(date_and_filename):
      seconds_from_1970 = float(date_and_filename[0])
      return datetime.fromtimestamp(int(seconds_from_1970))

      for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
      print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines


      The perfomance depends on implementation of nsmallest algorithm from python standard library.






      share|improve this answer
























        0












        0








        0






        Python heapq.nsmallest



        bash:



        find -printf '%T@ %pn' |
        python noldest.py 1000 |
        xargs -Ixxx mv xxx directory_for_old_files


        Here find command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs command takes filenames from stdin one by one and applies mv xxx directory_for_old_files command substituting xxx with them.



        noldest.py implementation could be:



        import sys
        from heapq import nsmallest
        from datetime import datetime


        n = int(sys.argv[1])
        date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy

        def parse_date(date_and_filename):
        seconds_from_1970 = float(date_and_filename[0])
        return datetime.fromtimestamp(int(seconds_from_1970))

        for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
        print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines


        The perfomance depends on implementation of nsmallest algorithm from python standard library.






        share|improve this answer












        Python heapq.nsmallest



        bash:



        find -printf '%T@ %pn' |
        python noldest.py 1000 |
        xargs -Ixxx mv xxx directory_for_old_files


        Here find command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs command takes filenames from stdin one by one and applies mv xxx directory_for_old_files command substituting xxx with them.



        noldest.py implementation could be:



        import sys
        from heapq import nsmallest
        from datetime import datetime


        n = int(sys.argv[1])
        date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy

        def parse_date(date_and_filename):
        seconds_from_1970 = float(date_and_filename[0])
        return datetime.fromtimestamp(int(seconds_from_1970))

        for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
        print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines


        The perfomance depends on implementation of nsmallest algorithm from python standard library.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Dec 29 '18 at 1:07









        belkkabelkka

        1307




        1307






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Unix & Linux Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f491356%2fmove-the-oldest-files-into-a-directory-source-directory-has-more-than-10-000-fi%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Morgemoulin

            Scott Moir

            Souastre