Cfrtjryk

I have about 120 files each with >1000 lines

Each line has it's own key in it. The columns are | separated

here is an example line, the key column (column 11 always column 11) is: 201075ITE854075_RECardProtectionlogi.msg

Error: null, Data:
|862799|00318070L|EMA|EMAIL|null|20100705|2010-07-05
14:59:39.0|null|AUTO_20100705|201075ITE854075_RECardProtectionlogi.msg|Country
not known

Is there a way to find all lines that have matching key/column 11 value(the whole line won't match)?
Can i do this on the command line or in a script?
I will be using cygwin.

I have no idea how to even start, so even if you only feel willing to give me suitable commands to look up, that would be gratefully received.

Each line has it's own key so there are potentially as many keys as there are lines.

I just want the script to run on an entire directory and report duplicate keys amongst all the files, without any other user input.

What defines a key is being in column 11.

Assuming that by "key" you mean "column", you could use something like this:

cut -f 11 -d "|" $(find . -type f -iname "*.txt") | sort | uniq -d | sed 's/\/./g' | while read duplicate; do grep -rHn "|$duplicate|" * ; done

You will probably have to change the contents of the $(find -iname) to whatever extension your log files have (or just remove it if the only files in the directory are log files. This will recursively find all log files and match them.

The output for some test data looks like this:


test_data.txt:1:Error: null, Data: |862799|00318070L|EMA|EMAIL|null|20100705|2010-07-05 14:59:39.0|null|AUTO_20100705|201075ITE854075_RECardProtectionlogi.msg|Country not known
test_data.txt:5:Error: null, Data: |862799|00318070L|EMA|EMAIL|null|20100705|2010-07-05 14:59:39.0|null|AUTO_20100705|201075ITE854075_RECardProtectionlogi.msg|Country not known
test_data_2.txt:2:Error: null, Data: |862799|00318070L|EMA|EMAIL|null|20100705|2010-07-05 14:59:39.0|null|AUTO_20100705|201075ITE854075_RECardProtectionlogi.msgIDONTMATCH|Country not known
test_data.txt:3:Error: null, Data: |862799|00318070L|EMA|EMAIL|null|20100705|2010-07-05 14:59:39.0|null|AUTO_20100705|201075ITE854075_RECardProtectionlogi.msgIDONTMATCH|Country not known
test_data_2.txt:4:Error: null, Data: |862799|00318070L|EMA|EMAIL|null|20100705|2010-07-05 14:59:39.0|null|AUTO_20100705|201075ITE854075_RECardProtectionlogi.msgIlikecake|Country not known
test_data.txt:7:Error: null, Data: |862799|00318070L|EMA|EMAIL|null|20100705|2010-07-05 14:59:39.0|null|AUTO_20100705|201075ITE854075_RECardProtectionlogi.msgIlikecake|Country not known


Where those are all lines within the files with field 11 duplicated.

Explanation of what the command does.

cut -f 11 -d "|" Get 11th field (as delimited by |)

find . -type f -iname "*.txt" consider any files ending in .txt in current directory (recursively)

sort | uniq -d show all duplicated "field 11s"

sed /\/./g' This is a hack because messes up bash. We replace it with ., which grep matches as any character.

while read duplicate; do grep -rHn "|$duplicate|" *; done - iterate over list of duplicates and find all occurrences of them, outputting filename and line numbers of where duplicates occured.

It's no clear what are you trying to do, but, I'll give a try:

First, what is your line? You gave this as a line:

Error: null, Data:|862799|00318070L|EMA|EMAIL|null|20100705|2010-07-05 14:59:39.0|null|AUTO_20100705|201175ITE854075_RECardProtectionlogi.msg|Country not known

If your lines looks like that, then, your key is in field 11

201175ITE854075_RECardProtectionlogi.msg

but, what define your key? Is it just to be in field 11?

If that so, you can do something like this, in the directory with your target files:

sort --field-separator='|' --key=11 <(grep --recursive --line-number --color=always --with-filename '' *)

this will give you a colorized output of the name of the file, followed by the line number in that file, and then the line itself, all sorted by key field 11; so, in the output, all matching keys in any file, appears one on top of each other...

I think that this will give you a clue, at least

Note: the backslash in front of grep it's to prevent any grep aliases.