Shell script required for collection/compiling of data
I wanted to collect some stats in a specific format (of previous day) from my server log files using a cronjob everyday.
Below is the sample list of log files:-
-rw-rw-r-- 1 lpt tkg 2.0M **Jan 3** 15:15 FTM.FC108.**20190103**103505.20190103104500
-rw-rw-r-- 1 lpt tkg 1.9M Jan 3 15:15 FTM.FC107.20190103103504.20190103104500
-rw-rw-r-- 1 lpt tkg 2.8M Jan 3 15:15 FTM.FC108.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 2.9M Jan 3 15:15 FTM.FC107.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 1.4M Jan 3 15:15 FTM.FC108.20190103104503.20190103104504
each file is containing following log entries apart from many others:
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:51.512 **FAIL** **DROP**: Too many resend tries failed Failed for request id: 12194518 **Cause:** **equipmentFailure** Info: <undef> Code: 34,USSD RequestId=12194518 OriginalId=12194456 EventCorrelationId="0b97104124" CreationTime="20190101041411" ResendCount=1 Timestamp=1546299951308 (Tue Jan 01 04:15:51 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100306252 UssdText=Last event was charged MPM 1200 Minutes 1.00 minutes, Remaining MPM 1200 Minutes 556.00 min1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:06.002 FAIL RETRY: Failed for request id: 12194431 **Cause:** **timeout Info**: <undef> Code: <undef>,USSD RequestId=12194431 OriginalId=12194431 EventCorrelationId="1211005491|244828886||1546299669000" CreationTime="20190101041325" ResendCount=0 Timestamp=1546299805999 (Tue Jan 01 04:13:25 AFT 2019) State=STATE_SENT **SubscriberNumber**=91106347299 UssdText=Last event was charged 2.33 MB from FB Monthly, Remaining data 15.61 MB (Exp 22.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:20:02.564 FAIL RETRY: Failed for request id: 12194657 **Cause:** **unknown Info**: <undef> Code: <undef>,USSD RequestId=12194657 OriginalId=12194657 EventCorrelationId="0165905324" CreationTime="20190101042001" ResendCount=0 Timestamp=1546300201043 (Tue Jan 01 04:20:01 AFT 2019) State=STATE_SENT SubscriberNumber=91104919244 UssdText=Last event was charged 0.50 AFN, Duration 0:01:09, Remaining balance 13.28 AFN and will expire 27.12.2020.1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:04.908 FAIL RETRY: Failed for request id: 12194487 **Cause:** **userNotReachable** Info: <undef> Code: 27,USSD RequestId=12194487 OriginalId=12194487 EventCorrelationId="-6700793294474983013" CreationTime="20190101041504" ResendCount=0 Timestamp=1546299904761 (Tue Jan 01 04:15:04 AFT 2019) State=STATE_SENT **SubscriberNumber**=91102615905 UssdText=Account update on 01.01.2019: Your new balance is 33.30 AFN and expires on 25.12.2020. Cost of last event was 1.00 AFN. Last recharge: 45.00 AFN on 26.12.2018. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20190101074505.20190101075031:2019-01-01 12:18:07.133 FAIL RETRY: Failed for request id: 12973754 **Cause:** **userUnavailable** Info: <undef> Code: 9,USSD RequestId=12973754 OriginalId=12973754 EventCorrelationId="615-493|-1909431042||1546328251000" CreationTime="20190101121807" ResendCount=0 Timestamp=1546328887082 (Tue Jan 01 12:18:07 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100012120 UssdText=Last event was charged 0.62 MB from 3GB Monthly, Remaining data 3,717.45 MB (Exp 25.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
Now I wanted to extract all the SubscriberNumber having the following counter values in separate output files together with their duplicate occurrences for each SubscriberNumber of previous day.
Cause: equipmentFailure
Cause: timeout Info
Cause: unknown Info
Cause: userNotReachable
Cause: userUnavailable
For example:
Sample Output file = equipmentFailure.out
Dupl_Occur SubscriberNumber
3 93700000022
2 93700000030
3 93700000048
2 93700000065
2 93700000066
19 93700000165
shell text-processing scripting
add a comment |
I wanted to collect some stats in a specific format (of previous day) from my server log files using a cronjob everyday.
Below is the sample list of log files:-
-rw-rw-r-- 1 lpt tkg 2.0M **Jan 3** 15:15 FTM.FC108.**20190103**103505.20190103104500
-rw-rw-r-- 1 lpt tkg 1.9M Jan 3 15:15 FTM.FC107.20190103103504.20190103104500
-rw-rw-r-- 1 lpt tkg 2.8M Jan 3 15:15 FTM.FC108.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 2.9M Jan 3 15:15 FTM.FC107.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 1.4M Jan 3 15:15 FTM.FC108.20190103104503.20190103104504
each file is containing following log entries apart from many others:
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:51.512 **FAIL** **DROP**: Too many resend tries failed Failed for request id: 12194518 **Cause:** **equipmentFailure** Info: <undef> Code: 34,USSD RequestId=12194518 OriginalId=12194456 EventCorrelationId="0b97104124" CreationTime="20190101041411" ResendCount=1 Timestamp=1546299951308 (Tue Jan 01 04:15:51 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100306252 UssdText=Last event was charged MPM 1200 Minutes 1.00 minutes, Remaining MPM 1200 Minutes 556.00 min1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:06.002 FAIL RETRY: Failed for request id: 12194431 **Cause:** **timeout Info**: <undef> Code: <undef>,USSD RequestId=12194431 OriginalId=12194431 EventCorrelationId="1211005491|244828886||1546299669000" CreationTime="20190101041325" ResendCount=0 Timestamp=1546299805999 (Tue Jan 01 04:13:25 AFT 2019) State=STATE_SENT **SubscriberNumber**=91106347299 UssdText=Last event was charged 2.33 MB from FB Monthly, Remaining data 15.61 MB (Exp 22.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:20:02.564 FAIL RETRY: Failed for request id: 12194657 **Cause:** **unknown Info**: <undef> Code: <undef>,USSD RequestId=12194657 OriginalId=12194657 EventCorrelationId="0165905324" CreationTime="20190101042001" ResendCount=0 Timestamp=1546300201043 (Tue Jan 01 04:20:01 AFT 2019) State=STATE_SENT SubscriberNumber=91104919244 UssdText=Last event was charged 0.50 AFN, Duration 0:01:09, Remaining balance 13.28 AFN and will expire 27.12.2020.1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:04.908 FAIL RETRY: Failed for request id: 12194487 **Cause:** **userNotReachable** Info: <undef> Code: 27,USSD RequestId=12194487 OriginalId=12194487 EventCorrelationId="-6700793294474983013" CreationTime="20190101041504" ResendCount=0 Timestamp=1546299904761 (Tue Jan 01 04:15:04 AFT 2019) State=STATE_SENT **SubscriberNumber**=91102615905 UssdText=Account update on 01.01.2019: Your new balance is 33.30 AFN and expires on 25.12.2020. Cost of last event was 1.00 AFN. Last recharge: 45.00 AFN on 26.12.2018. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20190101074505.20190101075031:2019-01-01 12:18:07.133 FAIL RETRY: Failed for request id: 12973754 **Cause:** **userUnavailable** Info: <undef> Code: 9,USSD RequestId=12973754 OriginalId=12973754 EventCorrelationId="615-493|-1909431042||1546328251000" CreationTime="20190101121807" ResendCount=0 Timestamp=1546328887082 (Tue Jan 01 12:18:07 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100012120 UssdText=Last event was charged 0.62 MB from 3GB Monthly, Remaining data 3,717.45 MB (Exp 25.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
Now I wanted to extract all the SubscriberNumber having the following counter values in separate output files together with their duplicate occurrences for each SubscriberNumber of previous day.
Cause: equipmentFailure
Cause: timeout Info
Cause: unknown Info
Cause: userNotReachable
Cause: userUnavailable
For example:
Sample Output file = equipmentFailure.out
Dupl_Occur SubscriberNumber
3 93700000022
2 93700000030
3 93700000048
2 93700000065
2 93700000066
19 93700000165
shell text-processing scripting
add a comment |
I wanted to collect some stats in a specific format (of previous day) from my server log files using a cronjob everyday.
Below is the sample list of log files:-
-rw-rw-r-- 1 lpt tkg 2.0M **Jan 3** 15:15 FTM.FC108.**20190103**103505.20190103104500
-rw-rw-r-- 1 lpt tkg 1.9M Jan 3 15:15 FTM.FC107.20190103103504.20190103104500
-rw-rw-r-- 1 lpt tkg 2.8M Jan 3 15:15 FTM.FC108.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 2.9M Jan 3 15:15 FTM.FC107.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 1.4M Jan 3 15:15 FTM.FC108.20190103104503.20190103104504
each file is containing following log entries apart from many others:
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:51.512 **FAIL** **DROP**: Too many resend tries failed Failed for request id: 12194518 **Cause:** **equipmentFailure** Info: <undef> Code: 34,USSD RequestId=12194518 OriginalId=12194456 EventCorrelationId="0b97104124" CreationTime="20190101041411" ResendCount=1 Timestamp=1546299951308 (Tue Jan 01 04:15:51 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100306252 UssdText=Last event was charged MPM 1200 Minutes 1.00 minutes, Remaining MPM 1200 Minutes 556.00 min1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:06.002 FAIL RETRY: Failed for request id: 12194431 **Cause:** **timeout Info**: <undef> Code: <undef>,USSD RequestId=12194431 OriginalId=12194431 EventCorrelationId="1211005491|244828886||1546299669000" CreationTime="20190101041325" ResendCount=0 Timestamp=1546299805999 (Tue Jan 01 04:13:25 AFT 2019) State=STATE_SENT **SubscriberNumber**=91106347299 UssdText=Last event was charged 2.33 MB from FB Monthly, Remaining data 15.61 MB (Exp 22.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:20:02.564 FAIL RETRY: Failed for request id: 12194657 **Cause:** **unknown Info**: <undef> Code: <undef>,USSD RequestId=12194657 OriginalId=12194657 EventCorrelationId="0165905324" CreationTime="20190101042001" ResendCount=0 Timestamp=1546300201043 (Tue Jan 01 04:20:01 AFT 2019) State=STATE_SENT SubscriberNumber=91104919244 UssdText=Last event was charged 0.50 AFN, Duration 0:01:09, Remaining balance 13.28 AFN and will expire 27.12.2020.1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:04.908 FAIL RETRY: Failed for request id: 12194487 **Cause:** **userNotReachable** Info: <undef> Code: 27,USSD RequestId=12194487 OriginalId=12194487 EventCorrelationId="-6700793294474983013" CreationTime="20190101041504" ResendCount=0 Timestamp=1546299904761 (Tue Jan 01 04:15:04 AFT 2019) State=STATE_SENT **SubscriberNumber**=91102615905 UssdText=Account update on 01.01.2019: Your new balance is 33.30 AFN and expires on 25.12.2020. Cost of last event was 1.00 AFN. Last recharge: 45.00 AFN on 26.12.2018. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20190101074505.20190101075031:2019-01-01 12:18:07.133 FAIL RETRY: Failed for request id: 12973754 **Cause:** **userUnavailable** Info: <undef> Code: 9,USSD RequestId=12973754 OriginalId=12973754 EventCorrelationId="615-493|-1909431042||1546328251000" CreationTime="20190101121807" ResendCount=0 Timestamp=1546328887082 (Tue Jan 01 12:18:07 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100012120 UssdText=Last event was charged 0.62 MB from 3GB Monthly, Remaining data 3,717.45 MB (Exp 25.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
Now I wanted to extract all the SubscriberNumber having the following counter values in separate output files together with their duplicate occurrences for each SubscriberNumber of previous day.
Cause: equipmentFailure
Cause: timeout Info
Cause: unknown Info
Cause: userNotReachable
Cause: userUnavailable
For example:
Sample Output file = equipmentFailure.out
Dupl_Occur SubscriberNumber
3 93700000022
2 93700000030
3 93700000048
2 93700000065
2 93700000066
19 93700000165
shell text-processing scripting
I wanted to collect some stats in a specific format (of previous day) from my server log files using a cronjob everyday.
Below is the sample list of log files:-
-rw-rw-r-- 1 lpt tkg 2.0M **Jan 3** 15:15 FTM.FC108.**20190103**103505.20190103104500
-rw-rw-r-- 1 lpt tkg 1.9M Jan 3 15:15 FTM.FC107.20190103103504.20190103104500
-rw-rw-r-- 1 lpt tkg 2.8M Jan 3 15:15 FTM.FC108.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 2.9M Jan 3 15:15 FTM.FC107.20190103104501.20190103104502
-rw-rw-r-- 1 lpt tkg 1.4M Jan 3 15:15 FTM.FC108.20190103104503.20190103104504
each file is containing following log entries apart from many others:
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:51.512 **FAIL** **DROP**: Too many resend tries failed Failed for request id: 12194518 **Cause:** **equipmentFailure** Info: <undef> Code: 34,USSD RequestId=12194518 OriginalId=12194456 EventCorrelationId="0b97104124" CreationTime="20190101041411" ResendCount=1 Timestamp=1546299951308 (Tue Jan 01 04:15:51 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100306252 UssdText=Last event was charged MPM 1200 Minutes 1.00 minutes, Remaining MPM 1200 Minutes 556.00 min1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:06.002 FAIL RETRY: Failed for request id: 12194431 **Cause:** **timeout Info**: <undef> Code: <undef>,USSD RequestId=12194431 OriginalId=12194431 EventCorrelationId="1211005491|244828886||1546299669000" CreationTime="20190101041325" ResendCount=0 Timestamp=1546299805999 (Tue Jan 01 04:13:25 AFT 2019) State=STATE_SENT **SubscriberNumber**=91106347299 UssdText=Last event was charged 2.33 MB from FB Monthly, Remaining data 15.61 MB (Exp 22.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:20:02.564 FAIL RETRY: Failed for request id: 12194657 **Cause:** **unknown Info**: <undef> Code: <undef>,USSD RequestId=12194657 OriginalId=12194657 EventCorrelationId="0165905324" CreationTime="20190101042001" ResendCount=0 Timestamp=1546300201043 (Tue Jan 01 04:20:01 AFT 2019) State=STATE_SENT SubscriberNumber=91104919244 UssdText=Last event was charged 0.50 AFN, Duration 0:01:09, Remaining balance 13.28 AFN and will expire 27.12.2020.1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20181231234501.20190101000000:2019-01-01 04:15:04.908 FAIL RETRY: Failed for request id: 12194487 **Cause:** **userNotReachable** Info: <undef> Code: 27,USSD RequestId=12194487 OriginalId=12194487 EventCorrelationId="-6700793294474983013" CreationTime="20190101041504" ResendCount=0 Timestamp=1546299904761 (Tue Jan 01 04:15:04 AFT 2019) State=STATE_SENT **SubscriberNumber**=91102615905 UssdText=Account update on 01.01.2019: Your new balance is 33.30 AFN and expires on 25.12.2020. Cost of last event was 1.00 AFN. Last recharge: 45.00 AFN on 26.12.2018. NumberingPlan=1 Nadi=4 UssdFormat=2
FTM.FC103.20190101074505.20190101075031:2019-01-01 12:18:07.133 FAIL RETRY: Failed for request id: 12973754 **Cause:** **userUnavailable** Info: <undef> Code: 9,USSD RequestId=12973754 OriginalId=12973754 EventCorrelationId="615-493|-1909431042||1546328251000" CreationTime="20190101121807" ResendCount=0 Timestamp=1546328887082 (Tue Jan 01 12:18:07 AFT 2019) State=STATE_SENT **SubscriberNumber**=91100012120 UssdText=Last event was charged 0.62 MB from 3GB Monthly, Remaining data 3,717.45 MB (Exp 25.01.2019)1500 AFS = 32GB valid 30 Days, Dial *477*32*1#. NumberingPlan=1 Nadi=4 UssdFormat=2
Now I wanted to extract all the SubscriberNumber having the following counter values in separate output files together with their duplicate occurrences for each SubscriberNumber of previous day.
Cause: equipmentFailure
Cause: timeout Info
Cause: unknown Info
Cause: userNotReachable
Cause: userUnavailable
For example:
Sample Output file = equipmentFailure.out
Dupl_Occur SubscriberNumber
3 93700000022
2 93700000030
3 93700000048
2 93700000065
2 93700000066
19 93700000165
shell text-processing scripting
shell text-processing scripting
edited Jan 3 at 14:00
Rui F Ribeiro
39.4k1479131
39.4k1479131
asked Jan 3 at 11:48
Jack AndersonJack Anderson
1
1
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492219%2fshell-script-required-for-collection-compiling-of-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f492219%2fshell-script-required-for-collection-compiling-of-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown