Shell script required for collection/compiling of data












-1















I wanted to collect some stats in a specific format (of previous day) from my server log files using a cronjob everyday.



Below is the sample list of log files:-



-rw-rw-r-- 1 lpt  tkg  2.0M **Jan  3** 15:15 FTM.FC108.**20190103**103505.20190103104500
-rw-rw-r-- 1 lpt tkg 1.9M Jan 3 15:15 FTM.FC107.20190103103504.20190103104500
-rw-rw-r-- 1 lpt tkg 2.8M Jan 3 15:15 FTM.FC108.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 2.9M Jan 3 15:15 FTM.FC107.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 1.4M Jan 3 15:15 FTM.FC108.20190103104503.20190103104504


each file is containing following log entries apart from many others:



FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:51.512 **FAIL** **DROP**: Too many resend tries failed Failed for request id: 12194518 **Cause:** **equipmentFailure** Info: <undef> Code: 34,USSD RequestId=12194518 OriginalId=12194456 EventCorrelationId="0b97104124" CreationTime="20190101041411" ResendCount=1 Timestamp=1546299951308 (Tue Jan 01 04:15:51 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100306252 UssdText=Last event was charged MPM 1200 Minutes 1.00 minutes, Remaining MPM 1200 Minutes 556.00 min1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:06.002 FAIL RETRY: Failed for request id: 12194431 **Cause:** **timeout Info**: <undef> Code: <undef>,USSD RequestId=12194431 OriginalId=12194431 EventCorrelationId="1211005491|244828886||1546299669000" CreationTime="20190101041325" ResendCount=0 Timestamp=1546299805999 (Tue Jan 01 04:13:25 AFT 2019) State=STATE_SENT **SubscriberNumber**=91106347299 UssdText=Last event was charged 2.33 MB from FB Monthly, Remaining data 15.61 MB (Exp 22.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

FTM.FC103.20181231234501.20190101000000:2019-01-01 04:20:02.564 FAIL RETRY: Failed for request id: 12194657 **Cause:** **unknown Info**: <undef> Code: <undef>,USSD RequestId=12194657 OriginalId=12194657 EventCorrelationId="0165905324" CreationTime="20190101042001" ResendCount=0 Timestamp=1546300201043 (Tue Jan 01 04:20:01 AFT 2019) State=STATE_SENT SubscriberNumber=91104919244 UssdText=Last event was charged 0.50 AFN, Duration 0:01:09, Remaining balance 13.28 AFN and will expire 27.12.2020.1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:04.908 FAIL RETRY: Failed for request id: 12194487 **Cause:** **userNotReachable** Info: <undef> Code: 27,USSD RequestId=12194487 OriginalId=12194487 EventCorrelationId="-6700793294474983013" CreationTime="20190101041504" ResendCount=0 Timestamp=1546299904761 (Tue Jan 01 04:15:04 AFT 2019) State=STATE_SENT **SubscriberNumber**=91102615905 UssdText=Account update on 01.01.2019: Your new balance is 33.30 AFN and expires on 25.12.2020. Cost of last event was 1.00 AFN. Last recharge: 45.00 AFN on 26.12.2018. NumberingPlan=1 Nadi=4 UssdFormat=2

FTM.FC103.20190101074505.20190101075031:2019-01-01 12:18:07.133 FAIL RETRY: Failed for request id: 12973754 **Cause:** **userUnavailable** Info: <undef> Code: 9,USSD RequestId=12973754 OriginalId=12973754 EventCorrelationId="615-493|-1909431042||1546328251000" CreationTime="20190101121807" ResendCount=0 Timestamp=1546328887082 (Tue Jan 01 12:18:07 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100012120 UssdText=Last event was charged 0.62 MB from 3GB Monthly, Remaining data 3,717.45 MB (Exp 25.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2


Now I wanted to extract all the SubscriberNumber having the following counter values in separate output files together with their duplicate occurrences for each SubscriberNumber of previous day.



Cause: equipmentFailure
Cause: timeout Info
Cause: unknown Info
Cause: userNotReachable
Cause: userUnavailable


For example:



Sample Output file = equipmentFailure.out

Dupl_Occur SubscriberNumber
3 93700000022
2 93700000030
3 93700000048
2 93700000065
2 93700000066
19 93700000165









share|improve this question





























    -1















    I wanted to collect some stats in a specific format (of previous day) from my server log files using a cronjob everyday.



    Below is the sample list of log files:-



    -rw-rw-r-- 1 lpt  tkg  2.0M **Jan  3** 15:15 FTM.FC108.**20190103**103505.20190103104500
    -rw-rw-r-- 1 lpt tkg 1.9M Jan 3 15:15 FTM.FC107.20190103103504.20190103104500
    -rw-rw-r-- 1 lpt tkg 2.8M Jan 3 15:15 FTM.FC108.20190103104501.20190103104502
    -rw-rw-r-- 1 lpt tkg 2.9M Jan 3 15:15 FTM.FC107.20190103104501.20190103104502
    -rw-rw-r-- 1 lpt tkg 1.4M Jan 3 15:15 FTM.FC108.20190103104503.20190103104504


    each file is containing following log entries apart from many others:



    FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:51.512 **FAIL** **DROP**: Too many resend tries failed Failed for request id: 12194518 **Cause:** **equipmentFailure** Info: <undef> Code: 34,USSD RequestId=12194518 OriginalId=12194456 EventCorrelationId="0b97104124" CreationTime="20190101041411" ResendCount=1 Timestamp=1546299951308 (Tue Jan 01 04:15:51 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100306252 UssdText=Last event was charged MPM 1200 Minutes 1.00 minutes, Remaining MPM 1200 Minutes 556.00 min1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

    FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:06.002 FAIL RETRY: Failed for request id: 12194431 **Cause:** **timeout Info**: <undef> Code: <undef>,USSD RequestId=12194431 OriginalId=12194431 EventCorrelationId="1211005491|244828886||1546299669000" CreationTime="20190101041325" ResendCount=0 Timestamp=1546299805999 (Tue Jan 01 04:13:25 AFT 2019) State=STATE_SENT **SubscriberNumber**=91106347299 UssdText=Last event was charged 2.33 MB from FB Monthly, Remaining data 15.61 MB (Exp 22.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

    FTM.FC103.20181231234501.20190101000000:2019-01-01 04:20:02.564 FAIL RETRY: Failed for request id: 12194657 **Cause:** **unknown Info**: <undef> Code: <undef>,USSD RequestId=12194657 OriginalId=12194657 EventCorrelationId="0165905324" CreationTime="20190101042001" ResendCount=0 Timestamp=1546300201043 (Tue Jan 01 04:20:01 AFT 2019) State=STATE_SENT SubscriberNumber=91104919244 UssdText=Last event was charged 0.50 AFN, Duration 0:01:09, Remaining balance 13.28 AFN and will expire 27.12.2020.1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

    FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:04.908 FAIL RETRY: Failed for request id: 12194487 **Cause:** **userNotReachable** Info: <undef> Code: 27,USSD RequestId=12194487 OriginalId=12194487 EventCorrelationId="-6700793294474983013" CreationTime="20190101041504" ResendCount=0 Timestamp=1546299904761 (Tue Jan 01 04:15:04 AFT 2019) State=STATE_SENT **SubscriberNumber**=91102615905 UssdText=Account update on 01.01.2019: Your new balance is 33.30 AFN and expires on 25.12.2020. Cost of last event was 1.00 AFN. Last recharge: 45.00 AFN on 26.12.2018. NumberingPlan=1 Nadi=4 UssdFormat=2

    FTM.FC103.20190101074505.20190101075031:2019-01-01 12:18:07.133 FAIL RETRY: Failed for request id: 12973754 **Cause:** **userUnavailable** Info: <undef> Code: 9,USSD RequestId=12973754 OriginalId=12973754 EventCorrelationId="615-493|-1909431042||1546328251000" CreationTime="20190101121807" ResendCount=0 Timestamp=1546328887082 (Tue Jan 01 12:18:07 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100012120 UssdText=Last event was charged 0.62 MB from 3GB Monthly, Remaining data 3,717.45 MB (Exp 25.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2


    Now I wanted to extract all the SubscriberNumber having the following counter values in separate output files together with their duplicate occurrences for each SubscriberNumber of previous day.



    Cause: equipmentFailure
    Cause: timeout Info
    Cause: unknown Info
    Cause: userNotReachable
    Cause: userUnavailable


    For example:



    Sample Output file = equipmentFailure.out

    Dupl_Occur SubscriberNumber
    3 93700000022
    2 93700000030
    3 93700000048
    2 93700000065
    2 93700000066
    19 93700000165









    share|improve this question



























      -1












      -1








      -1








      I wanted to collect some stats in a specific format (of previous day) from my server log files using a cronjob everyday.



      Below is the sample list of log files:-



      -rw-rw-r-- 1 lpt  tkg  2.0M **Jan  3** 15:15 FTM.FC108.**20190103**103505.20190103104500
      -rw-rw-r-- 1 lpt tkg 1.9M Jan 3 15:15 FTM.FC107.20190103103504.20190103104500
      -rw-rw-r-- 1 lpt tkg 2.8M Jan 3 15:15 FTM.FC108.20190103104501.20190103104502
      -rw-rw-r-- 1 lpt tkg 2.9M Jan 3 15:15 FTM.FC107.20190103104501.20190103104502
      -rw-rw-r-- 1 lpt tkg 1.4M Jan 3 15:15 FTM.FC108.20190103104503.20190103104504


      each file is containing following log entries apart from many others:



      FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:51.512 **FAIL** **DROP**: Too many resend tries failed Failed for request id: 12194518 **Cause:** **equipmentFailure** Info: <undef> Code: 34,USSD RequestId=12194518 OriginalId=12194456 EventCorrelationId="0b97104124" CreationTime="20190101041411" ResendCount=1 Timestamp=1546299951308 (Tue Jan 01 04:15:51 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100306252 UssdText=Last event was charged MPM 1200 Minutes 1.00 minutes, Remaining MPM 1200 Minutes 556.00 min1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

      FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:06.002 FAIL RETRY: Failed for request id: 12194431 **Cause:** **timeout Info**: <undef> Code: <undef>,USSD RequestId=12194431 OriginalId=12194431 EventCorrelationId="1211005491|244828886||1546299669000" CreationTime="20190101041325" ResendCount=0 Timestamp=1546299805999 (Tue Jan 01 04:13:25 AFT 2019) State=STATE_SENT **SubscriberNumber**=91106347299 UssdText=Last event was charged 2.33 MB from FB Monthly, Remaining data 15.61 MB (Exp 22.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

      FTM.FC103.20181231234501.20190101000000:2019-01-01 04:20:02.564 FAIL RETRY: Failed for request id: 12194657 **Cause:** **unknown Info**: <undef> Code: <undef>,USSD RequestId=12194657 OriginalId=12194657 EventCorrelationId="0165905324" CreationTime="20190101042001" ResendCount=0 Timestamp=1546300201043 (Tue Jan 01 04:20:01 AFT 2019) State=STATE_SENT SubscriberNumber=91104919244 UssdText=Last event was charged 0.50 AFN, Duration 0:01:09, Remaining balance 13.28 AFN and will expire 27.12.2020.1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

      FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:04.908 FAIL RETRY: Failed for request id: 12194487 **Cause:** **userNotReachable** Info: <undef> Code: 27,USSD RequestId=12194487 OriginalId=12194487 EventCorrelationId="-6700793294474983013" CreationTime="20190101041504" ResendCount=0 Timestamp=1546299904761 (Tue Jan 01 04:15:04 AFT 2019) State=STATE_SENT **SubscriberNumber**=91102615905 UssdText=Account update on 01.01.2019: Your new balance is 33.30 AFN and expires on 25.12.2020. Cost of last event was 1.00 AFN. Last recharge: 45.00 AFN on 26.12.2018. NumberingPlan=1 Nadi=4 UssdFormat=2

      FTM.FC103.20190101074505.20190101075031:2019-01-01 12:18:07.133 FAIL RETRY: Failed for request id: 12973754 **Cause:** **userUnavailable** Info: <undef> Code: 9,USSD RequestId=12973754 OriginalId=12973754 EventCorrelationId="615-493|-1909431042||1546328251000" CreationTime="20190101121807" ResendCount=0 Timestamp=1546328887082 (Tue Jan 01 12:18:07 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100012120 UssdText=Last event was charged 0.62 MB from 3GB Monthly, Remaining data 3,717.45 MB (Exp 25.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2


      Now I wanted to extract all the SubscriberNumber having the following counter values in separate output files together with their duplicate occurrences for each SubscriberNumber of previous day.



      Cause: equipmentFailure
      Cause: timeout Info
      Cause: unknown Info
      Cause: userNotReachable
      Cause: userUnavailable


      For example:



      Sample Output file = equipmentFailure.out

      Dupl_Occur SubscriberNumber
      3 93700000022
      2 93700000030
      3 93700000048
      2 93700000065
      2 93700000066
      19 93700000165









      share|improve this question
















      I wanted to collect some stats in a specific format (of previous day) from my server log files using a cronjob everyday.



      Below is the sample list of log files:-



      -rw-rw-r-- 1 lpt  tkg  2.0M **Jan  3** 15:15 FTM.FC108.**20190103**103505.20190103104500
      -rw-rw-r-- 1 lpt tkg 1.9M Jan 3 15:15 FTM.FC107.20190103103504.20190103104500
      -rw-rw-r-- 1 lpt tkg 2.8M Jan 3 15:15 FTM.FC108.20190103104501.20190103104502
      -rw-rw-r-- 1 lpt tkg 2.9M Jan 3 15:15 FTM.FC107.20190103104501.20190103104502
      -rw-rw-r-- 1 lpt tkg 1.4M Jan 3 15:15 FTM.FC108.20190103104503.20190103104504


      each file is containing following log entries apart from many others:



      FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:51.512 **FAIL** **DROP**: Too many resend tries failed Failed for request id: 12194518 **Cause:** **equipmentFailure** Info: <undef> Code: 34,USSD RequestId=12194518 OriginalId=12194456 EventCorrelationId="0b97104124" CreationTime="20190101041411" ResendCount=1 Timestamp=1546299951308 (Tue Jan 01 04:15:51 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100306252 UssdText=Last event was charged MPM 1200 Minutes 1.00 minutes, Remaining MPM 1200 Minutes 556.00 min1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

      FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:06.002 FAIL RETRY: Failed for request id: 12194431 **Cause:** **timeout Info**: <undef> Code: <undef>,USSD RequestId=12194431 OriginalId=12194431 EventCorrelationId="1211005491|244828886||1546299669000" CreationTime="20190101041325" ResendCount=0 Timestamp=1546299805999 (Tue Jan 01 04:13:25 AFT 2019) State=STATE_SENT **SubscriberNumber**=91106347299 UssdText=Last event was charged 2.33 MB from FB Monthly, Remaining data 15.61 MB (Exp 22.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

      FTM.FC103.20181231234501.20190101000000:2019-01-01 04:20:02.564 FAIL RETRY: Failed for request id: 12194657 **Cause:** **unknown Info**: <undef> Code: <undef>,USSD RequestId=12194657 OriginalId=12194657 EventCorrelationId="0165905324" CreationTime="20190101042001" ResendCount=0 Timestamp=1546300201043 (Tue Jan 01 04:20:01 AFT 2019) State=STATE_SENT SubscriberNumber=91104919244 UssdText=Last event was charged 0.50 AFN, Duration 0:01:09, Remaining balance 13.28 AFN and will expire 27.12.2020.1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2

      FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:04.908 FAIL RETRY: Failed for request id: 12194487 **Cause:** **userNotReachable** Info: <undef> Code: 27,USSD RequestId=12194487 OriginalId=12194487 EventCorrelationId="-6700793294474983013" CreationTime="20190101041504" ResendCount=0 Timestamp=1546299904761 (Tue Jan 01 04:15:04 AFT 2019) State=STATE_SENT **SubscriberNumber**=91102615905 UssdText=Account update on 01.01.2019: Your new balance is 33.30 AFN and expires on 25.12.2020. Cost of last event was 1.00 AFN. Last recharge: 45.00 AFN on 26.12.2018. NumberingPlan=1 Nadi=4 UssdFormat=2

      FTM.FC103.20190101074505.20190101075031:2019-01-01 12:18:07.133 FAIL RETRY: Failed for request id: 12973754 **Cause:** **userUnavailable** Info: <undef> Code: 9,USSD RequestId=12973754 OriginalId=12973754 EventCorrelationId="615-493|-1909431042||1546328251000" CreationTime="20190101121807" ResendCount=0 Timestamp=1546328887082 (Tue Jan 01 12:18:07 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100012120 UssdText=Last event was charged 0.62 MB from 3GB Monthly, Remaining data 3,717.45 MB (Exp 25.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2


      Now I wanted to extract all the SubscriberNumber having the following counter values in separate output files together with their duplicate occurrences for each SubscriberNumber of previous day.



      Cause: equipmentFailure
      Cause: timeout Info
      Cause: unknown Info
      Cause: userNotReachable
      Cause: userUnavailable


      For example:



      Sample Output file = equipmentFailure.out

      Dupl_Occur SubscriberNumber
      3 93700000022
      2 93700000030
      3 93700000048
      2 93700000065
      2 93700000066
      19 93700000165






      shell text-processing scripting






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Jan 3 at 14:00









      Rui F Ribeiro

      39.4k1479131




      39.4k1479131










      asked Jan 3 at 11:48









      Jack AndersonJack Anderson

      1




      1






















          0






          active

          oldest

          votes











          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "106"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492219%2fshell-script-required-for-collection-compiling-of-data%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Unix & Linux Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492219%2fshell-script-required-for-collection-compiling-of-data%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Morgemoulin

          Scott Moir

          Souastre