redirect and log script output












8














I am trying to tidy up the following snippets, design goals are to log all output from a script, and should not be a wrapper. Less lines are better.



I don't care about user inputs (at this stage), target scripts are run non-interactively.



The snippet needs to




  • output stdout to log, and always echo to console

  • output stderr to log, and echo to console iff debugging is enabled

  • stderr messages should be prefixed with time stamps and other usefulness


At the moment I have the following which only tests out under recent versions of bash (4.2+?) like in Ubuntu precise, but misbehaves on CentOS6.



DEBUG_LOG="${0##*/}.log"

# copy stdout to log always and echo to console
exec > >(tee -a ${DEBUG_LOG})

# copy stderr to log only, unless debugging is enabled
[ $DEBUG_TEST = "true" ]
&& exec 2> >(tee -a ${DEBUG_LOG} >&2)
|| exec 2>> ${DEBUG_LOG}


Then this...



# Expand escaped characters, wrap at 70 chars on spaces, 
# and indent wrapped lines
msg_log() {
echo -e "$(date +%T) ${0##*/}: $1"
| fold -w70 -s | sed '2~1s/^/ /' >&2;
}
msg_con() {
if [ "${DEBUG_TEST}" = "true" ]; then
msg_log "$1"
else
echo -e "$1" | fold -w70 -s | sed '2~1s/^/ /';
fi
}


Instead of echo I can call one of those msg procedures, e.g., msg_con "hello world".

Also script output will then go to stderr by setting as an environment variable at call time, e.g., DEBUG_TEST=true myscript.



I have read that exec may not work in some shells such as busybox. There is a mkfifo and fork combination at https://stackoverflow.com/a/5200754 that does something similar but I'd rather not use fork unless absolutely needed.



Prefer bash examples please, but something that works under sh or is more portable would be nice. Any ideas?










share|improve this question





























    8














    I am trying to tidy up the following snippets, design goals are to log all output from a script, and should not be a wrapper. Less lines are better.



    I don't care about user inputs (at this stage), target scripts are run non-interactively.



    The snippet needs to




    • output stdout to log, and always echo to console

    • output stderr to log, and echo to console iff debugging is enabled

    • stderr messages should be prefixed with time stamps and other usefulness


    At the moment I have the following which only tests out under recent versions of bash (4.2+?) like in Ubuntu precise, but misbehaves on CentOS6.



    DEBUG_LOG="${0##*/}.log"

    # copy stdout to log always and echo to console
    exec > >(tee -a ${DEBUG_LOG})

    # copy stderr to log only, unless debugging is enabled
    [ $DEBUG_TEST = "true" ]
    && exec 2> >(tee -a ${DEBUG_LOG} >&2)
    || exec 2>> ${DEBUG_LOG}


    Then this...



    # Expand escaped characters, wrap at 70 chars on spaces, 
    # and indent wrapped lines
    msg_log() {
    echo -e "$(date +%T) ${0##*/}: $1"
    | fold -w70 -s | sed '2~1s/^/ /' >&2;
    }
    msg_con() {
    if [ "${DEBUG_TEST}" = "true" ]; then
    msg_log "$1"
    else
    echo -e "$1" | fold -w70 -s | sed '2~1s/^/ /';
    fi
    }


    Instead of echo I can call one of those msg procedures, e.g., msg_con "hello world".

    Also script output will then go to stderr by setting as an environment variable at call time, e.g., DEBUG_TEST=true myscript.



    I have read that exec may not work in some shells such as busybox. There is a mkfifo and fork combination at https://stackoverflow.com/a/5200754 that does something similar but I'd rather not use fork unless absolutely needed.



    Prefer bash examples please, but something that works under sh or is more portable would be nice. Any ideas?










    share|improve this question



























      8












      8








      8


      2





      I am trying to tidy up the following snippets, design goals are to log all output from a script, and should not be a wrapper. Less lines are better.



      I don't care about user inputs (at this stage), target scripts are run non-interactively.



      The snippet needs to




      • output stdout to log, and always echo to console

      • output stderr to log, and echo to console iff debugging is enabled

      • stderr messages should be prefixed with time stamps and other usefulness


      At the moment I have the following which only tests out under recent versions of bash (4.2+?) like in Ubuntu precise, but misbehaves on CentOS6.



      DEBUG_LOG="${0##*/}.log"

      # copy stdout to log always and echo to console
      exec > >(tee -a ${DEBUG_LOG})

      # copy stderr to log only, unless debugging is enabled
      [ $DEBUG_TEST = "true" ]
      && exec 2> >(tee -a ${DEBUG_LOG} >&2)
      || exec 2>> ${DEBUG_LOG}


      Then this...



      # Expand escaped characters, wrap at 70 chars on spaces, 
      # and indent wrapped lines
      msg_log() {
      echo -e "$(date +%T) ${0##*/}: $1"
      | fold -w70 -s | sed '2~1s/^/ /' >&2;
      }
      msg_con() {
      if [ "${DEBUG_TEST}" = "true" ]; then
      msg_log "$1"
      else
      echo -e "$1" | fold -w70 -s | sed '2~1s/^/ /';
      fi
      }


      Instead of echo I can call one of those msg procedures, e.g., msg_con "hello world".

      Also script output will then go to stderr by setting as an environment variable at call time, e.g., DEBUG_TEST=true myscript.



      I have read that exec may not work in some shells such as busybox. There is a mkfifo and fork combination at https://stackoverflow.com/a/5200754 that does something similar but I'd rather not use fork unless absolutely needed.



      Prefer bash examples please, but something that works under sh or is more portable would be nice. Any ideas?










      share|improve this question















      I am trying to tidy up the following snippets, design goals are to log all output from a script, and should not be a wrapper. Less lines are better.



      I don't care about user inputs (at this stage), target scripts are run non-interactively.



      The snippet needs to




      • output stdout to log, and always echo to console

      • output stderr to log, and echo to console iff debugging is enabled

      • stderr messages should be prefixed with time stamps and other usefulness


      At the moment I have the following which only tests out under recent versions of bash (4.2+?) like in Ubuntu precise, but misbehaves on CentOS6.



      DEBUG_LOG="${0##*/}.log"

      # copy stdout to log always and echo to console
      exec > >(tee -a ${DEBUG_LOG})

      # copy stderr to log only, unless debugging is enabled
      [ $DEBUG_TEST = "true" ]
      && exec 2> >(tee -a ${DEBUG_LOG} >&2)
      || exec 2>> ${DEBUG_LOG}


      Then this...



      # Expand escaped characters, wrap at 70 chars on spaces, 
      # and indent wrapped lines
      msg_log() {
      echo -e "$(date +%T) ${0##*/}: $1"
      | fold -w70 -s | sed '2~1s/^/ /' >&2;
      }
      msg_con() {
      if [ "${DEBUG_TEST}" = "true" ]; then
      msg_log "$1"
      else
      echo -e "$1" | fold -w70 -s | sed '2~1s/^/ /';
      fi
      }


      Instead of echo I can call one of those msg procedures, e.g., msg_con "hello world".

      Also script output will then go to stderr by setting as an environment variable at call time, e.g., DEBUG_TEST=true myscript.



      I have read that exec may not work in some shells such as busybox. There is a mkfifo and fork combination at https://stackoverflow.com/a/5200754 that does something similar but I'd rather not use fork unless absolutely needed.



      Prefer bash examples please, but something that works under sh or is more portable would be nice. Any ideas?







      bash shell-script logs exec






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Dec 19 '18 at 6:26









      Rui F Ribeiro

      39k1479130




      39k1479130










      asked Feb 22 '13 at 8:24









      Glenn

      6317




      6317






















          4 Answers
          4






          active

          oldest

          votes


















          1














          function startLogging {
          exec > >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' | tee -a $logfile)
          [ ! -z "$DEBUG" ] && exec 2>&1 || exec 2> >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' >>$logfile)
          echo "=== Log started for $$ at $(date +%F-%T) ==="
          }


          You need to have $logfile set to something






          share|improve this answer























          • This is kinda cool, the way I'm reading it you use gawk to add the message headers. That will have the additional side effect of adding those to command outputs as well.
            – Glenn
            Oct 5 '13 at 9:14










          • The reason to use gawk to add a timestamp (and process id) to each log line is because it will get executed each time output is written and therefore update the timestamp. Also it's important to use gawk and not awk because I think the strftime function is a GNU extension.
            – Angelo
            Oct 9 '13 at 17:41



















          0














          exec > filename should work in sh, and it actually works in busybox v1.15.3 (Nov 2011). But the process substitution >(command) is unportable since it's bash extension. Just avoid using it in scripts. Why >> isn't enough for you?



          exec 1>>${DEBUG_LOG}
          exec 2>>${DEBUG_LOG}




          Another solution is to specify redirection outside of your scripts. When your script is invoked in background (by cron, or system script, etc.), they should be called like this



          ./my_script 1>>${DEBUG_LOG} 2>>${DEBUG_LOG}


          When you invoke the script manually and you want to see the output, just call it without redirections.






          share|improve this answer

















          • 1




            The question asker wants the output of the script to go to both the console and the log file.
            – user26112
            May 4 '13 at 11:29





















          0














          These two examples will do what your stated objectives are



          echo -n $(date) >> $DEBUG_LOG
          command 2>&1 | tee -a $DEBUG_LOG


          or



          echo -n $(date) >> $DEBUG_LOG
          command >> $DEBUG_LOG 2>&1





          share|improve this answer





























            0














            You could use tee command or script command both are really useful.






            share|improve this answer





















            • script looks cool, but does not really do what I need, the behavior has to modifiable during runtime.
              – Glenn
              Oct 5 '13 at 8:54













            Your Answer








            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "106"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f65718%2fredirect-and-log-script-output%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            4 Answers
            4






            active

            oldest

            votes








            4 Answers
            4






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            1














            function startLogging {
            exec > >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' | tee -a $logfile)
            [ ! -z "$DEBUG" ] && exec 2>&1 || exec 2> >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' >>$logfile)
            echo "=== Log started for $$ at $(date +%F-%T) ==="
            }


            You need to have $logfile set to something






            share|improve this answer























            • This is kinda cool, the way I'm reading it you use gawk to add the message headers. That will have the additional side effect of adding those to command outputs as well.
              – Glenn
              Oct 5 '13 at 9:14










            • The reason to use gawk to add a timestamp (and process id) to each log line is because it will get executed each time output is written and therefore update the timestamp. Also it's important to use gawk and not awk because I think the strftime function is a GNU extension.
              – Angelo
              Oct 9 '13 at 17:41
















            1














            function startLogging {
            exec > >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' | tee -a $logfile)
            [ ! -z "$DEBUG" ] && exec 2>&1 || exec 2> >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' >>$logfile)
            echo "=== Log started for $$ at $(date +%F-%T) ==="
            }


            You need to have $logfile set to something






            share|improve this answer























            • This is kinda cool, the way I'm reading it you use gawk to add the message headers. That will have the additional side effect of adding those to command outputs as well.
              – Glenn
              Oct 5 '13 at 9:14










            • The reason to use gawk to add a timestamp (and process id) to each log line is because it will get executed each time output is written and therefore update the timestamp. Also it's important to use gawk and not awk because I think the strftime function is a GNU extension.
              – Angelo
              Oct 9 '13 at 17:41














            1












            1








            1






            function startLogging {
            exec > >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' | tee -a $logfile)
            [ ! -z "$DEBUG" ] && exec 2>&1 || exec 2> >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' >>$logfile)
            echo "=== Log started for $$ at $(date +%F-%T) ==="
            }


            You need to have $logfile set to something






            share|improve this answer














            function startLogging {
            exec > >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' | tee -a $logfile)
            [ ! -z "$DEBUG" ] && exec 2>&1 || exec 2> >(gawk -v pid=$$ '{ print strftime("%F-%T"),pid,$0; fflush(); }' >>$logfile)
            echo "=== Log started for $$ at $(date +%F-%T) ==="
            }


            You need to have $logfile set to something







            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Sep 18 '13 at 3:00

























            answered Sep 17 '13 at 21:01









            Angelo

            9331718




            9331718












            • This is kinda cool, the way I'm reading it you use gawk to add the message headers. That will have the additional side effect of adding those to command outputs as well.
              – Glenn
              Oct 5 '13 at 9:14










            • The reason to use gawk to add a timestamp (and process id) to each log line is because it will get executed each time output is written and therefore update the timestamp. Also it's important to use gawk and not awk because I think the strftime function is a GNU extension.
              – Angelo
              Oct 9 '13 at 17:41


















            • This is kinda cool, the way I'm reading it you use gawk to add the message headers. That will have the additional side effect of adding those to command outputs as well.
              – Glenn
              Oct 5 '13 at 9:14










            • The reason to use gawk to add a timestamp (and process id) to each log line is because it will get executed each time output is written and therefore update the timestamp. Also it's important to use gawk and not awk because I think the strftime function is a GNU extension.
              – Angelo
              Oct 9 '13 at 17:41
















            This is kinda cool, the way I'm reading it you use gawk to add the message headers. That will have the additional side effect of adding those to command outputs as well.
            – Glenn
            Oct 5 '13 at 9:14




            This is kinda cool, the way I'm reading it you use gawk to add the message headers. That will have the additional side effect of adding those to command outputs as well.
            – Glenn
            Oct 5 '13 at 9:14












            The reason to use gawk to add a timestamp (and process id) to each log line is because it will get executed each time output is written and therefore update the timestamp. Also it's important to use gawk and not awk because I think the strftime function is a GNU extension.
            – Angelo
            Oct 9 '13 at 17:41




            The reason to use gawk to add a timestamp (and process id) to each log line is because it will get executed each time output is written and therefore update the timestamp. Also it's important to use gawk and not awk because I think the strftime function is a GNU extension.
            – Angelo
            Oct 9 '13 at 17:41













            0














            exec > filename should work in sh, and it actually works in busybox v1.15.3 (Nov 2011). But the process substitution >(command) is unportable since it's bash extension. Just avoid using it in scripts. Why >> isn't enough for you?



            exec 1>>${DEBUG_LOG}
            exec 2>>${DEBUG_LOG}




            Another solution is to specify redirection outside of your scripts. When your script is invoked in background (by cron, or system script, etc.), they should be called like this



            ./my_script 1>>${DEBUG_LOG} 2>>${DEBUG_LOG}


            When you invoke the script manually and you want to see the output, just call it without redirections.






            share|improve this answer

















            • 1




              The question asker wants the output of the script to go to both the console and the log file.
              – user26112
              May 4 '13 at 11:29


















            0














            exec > filename should work in sh, and it actually works in busybox v1.15.3 (Nov 2011). But the process substitution >(command) is unportable since it's bash extension. Just avoid using it in scripts. Why >> isn't enough for you?



            exec 1>>${DEBUG_LOG}
            exec 2>>${DEBUG_LOG}




            Another solution is to specify redirection outside of your scripts. When your script is invoked in background (by cron, or system script, etc.), they should be called like this



            ./my_script 1>>${DEBUG_LOG} 2>>${DEBUG_LOG}


            When you invoke the script manually and you want to see the output, just call it without redirections.






            share|improve this answer

















            • 1




              The question asker wants the output of the script to go to both the console and the log file.
              – user26112
              May 4 '13 at 11:29
















            0












            0








            0






            exec > filename should work in sh, and it actually works in busybox v1.15.3 (Nov 2011). But the process substitution >(command) is unportable since it's bash extension. Just avoid using it in scripts. Why >> isn't enough for you?



            exec 1>>${DEBUG_LOG}
            exec 2>>${DEBUG_LOG}




            Another solution is to specify redirection outside of your scripts. When your script is invoked in background (by cron, or system script, etc.), they should be called like this



            ./my_script 1>>${DEBUG_LOG} 2>>${DEBUG_LOG}


            When you invoke the script manually and you want to see the output, just call it without redirections.






            share|improve this answer












            exec > filename should work in sh, and it actually works in busybox v1.15.3 (Nov 2011). But the process substitution >(command) is unportable since it's bash extension. Just avoid using it in scripts. Why >> isn't enough for you?



            exec 1>>${DEBUG_LOG}
            exec 2>>${DEBUG_LOG}




            Another solution is to specify redirection outside of your scripts. When your script is invoked in background (by cron, or system script, etc.), they should be called like this



            ./my_script 1>>${DEBUG_LOG} 2>>${DEBUG_LOG}


            When you invoke the script manually and you want to see the output, just call it without redirections.







            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered May 4 '13 at 9:03









            kirikaza

            1012




            1012








            • 1




              The question asker wants the output of the script to go to both the console and the log file.
              – user26112
              May 4 '13 at 11:29
















            • 1




              The question asker wants the output of the script to go to both the console and the log file.
              – user26112
              May 4 '13 at 11:29










            1




            1




            The question asker wants the output of the script to go to both the console and the log file.
            – user26112
            May 4 '13 at 11:29






            The question asker wants the output of the script to go to both the console and the log file.
            – user26112
            May 4 '13 at 11:29













            0














            These two examples will do what your stated objectives are



            echo -n $(date) >> $DEBUG_LOG
            command 2>&1 | tee -a $DEBUG_LOG


            or



            echo -n $(date) >> $DEBUG_LOG
            command >> $DEBUG_LOG 2>&1





            share|improve this answer


























              0














              These two examples will do what your stated objectives are



              echo -n $(date) >> $DEBUG_LOG
              command 2>&1 | tee -a $DEBUG_LOG


              or



              echo -n $(date) >> $DEBUG_LOG
              command >> $DEBUG_LOG 2>&1





              share|improve this answer
























                0












                0








                0






                These two examples will do what your stated objectives are



                echo -n $(date) >> $DEBUG_LOG
                command 2>&1 | tee -a $DEBUG_LOG


                or



                echo -n $(date) >> $DEBUG_LOG
                command >> $DEBUG_LOG 2>&1





                share|improve this answer












                These two examples will do what your stated objectives are



                echo -n $(date) >> $DEBUG_LOG
                command 2>&1 | tee -a $DEBUG_LOG


                or



                echo -n $(date) >> $DEBUG_LOG
                command >> $DEBUG_LOG 2>&1






                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Jul 25 '13 at 20:53









                Luke

                873




                873























                    0














                    You could use tee command or script command both are really useful.






                    share|improve this answer





















                    • script looks cool, but does not really do what I need, the behavior has to modifiable during runtime.
                      – Glenn
                      Oct 5 '13 at 8:54


















                    0














                    You could use tee command or script command both are really useful.






                    share|improve this answer





















                    • script looks cool, but does not really do what I need, the behavior has to modifiable during runtime.
                      – Glenn
                      Oct 5 '13 at 8:54
















                    0












                    0








                    0






                    You could use tee command or script command both are really useful.






                    share|improve this answer












                    You could use tee command or script command both are really useful.







                    share|improve this answer












                    share|improve this answer



                    share|improve this answer










                    answered Sep 12 '13 at 10:57









                    Falk

                    1315




                    1315












                    • script looks cool, but does not really do what I need, the behavior has to modifiable during runtime.
                      – Glenn
                      Oct 5 '13 at 8:54




















                    • script looks cool, but does not really do what I need, the behavior has to modifiable during runtime.
                      – Glenn
                      Oct 5 '13 at 8:54


















                    script looks cool, but does not really do what I need, the behavior has to modifiable during runtime.
                    – Glenn
                    Oct 5 '13 at 8:54






                    script looks cool, but does not really do what I need, the behavior has to modifiable during runtime.
                    – Glenn
                    Oct 5 '13 at 8:54




















                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Unix & Linux Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.





                    Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                    Please pay close attention to the following guidance:


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f65718%2fredirect-and-log-script-output%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    Morgemoulin

                    Scott Moir

                    Souastre