Hello all,
I have a txt file containing millions of lines. Below is the example:
Code :
{tx:be} head -50 file.txt
Instr1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Instr1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000000000000000000000001100001010000000010011101001111000000000100010100111111110010000000000000000000000000000000000000001
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000000000000000000000001100001010000000010010101001111000000000100010100111111111000000000000000000000000000000000000000001
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000000000000000000000001100001010000000000000101000011000000000100010101000000000010000000000000000000000000000000000000001
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000110110000000000000000100001010000000010011100101000000000000100010101000000001110000000000000000000000000000000000000001
Instr1: 000000000100000000000000000001111110000000000000000000010000000000100010101000000010110000000000000010001001111011011100000111
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000110111000111110000100100001010000000011100100101000000000000100010011110110111000000000000000000000000000000000000000001
Instr1: 000001010110000000000100000100000000101001011101100110100000000000100010011110111000000000000000000000000000000000000000000001
Instr1: 000001010110000000000011000100000000100101011101100110100000000000100010011110111001000000000000000000000000000000000000000001
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000110111000111101110100100001010000000011100100101000000000000100010011110111010100000000000000000000000000000000000000001
Instr1: 000000110111000111101110100100001010000000011100100101000000000000100010011110111010100000000000000000000000000000000000000001
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000100110000000000001011100001110000000010011100100010000000000100010100111111110110000000000000000000000000000000000000001
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000000000000000000000001100001010000000000000101000011000000000100010101000000000010000000000000000000000000000000000000001
Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Instr1: 000000110110000000000000000100001010000000010011100101000000000000100010101000000001110000000000000000000000000000000000000001
Instr1: 000000000100000000000000000001111110000000000000000000010000000000100010101000000010110000000000000010001001111011011100000111
There are empty lines which I take off using "sed 's/^$/d' file.txt"
Now the problem is, I want to find number of uniq values on the binary field. Here is what I want:
in the binary values, I was to find how many times the uniq values in field [57:50] are occuring. (MSB -> 125, LSB -> 0). There are total 126 bits in the lines.
I have sorted the files using sort:
Code :
sort -k2.50,2.57 file.txt
output:
{tx:be} tail -50 file.txt
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000011111111110000000000100010100000001101110000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000100111111110000000000100010100000100101100000000000000000000000000000000000000001
Instr1: 000000000010000000000000000010000010100111000100111111110000000000100010100000100101100000000000000000000000000000000000000001
Instr1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Instr1: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
As you can see, the files are sorted based on the fields that I am interested in. Now I am not sure how to find the Number of occurence (uniq) in those fields.
I have tried the uniq command, but surely it doesn't help:
Code :
uniq -c -f1 -s75 -w69 file.txt
Output: (truncated)
2751026 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
23 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001
23 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000001
23 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000001
24 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000011000000000000000000000000000000000000000001
24 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000
22 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000101000000000000000000000000000000000000000001
19 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000110000000000000000000000000000000000000000001
17 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000000111000000000000000000000000000000000000000000
18 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000001
18 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000001001000000000000000000000000000000000000000001
17 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000001010000000000000000000000000000000000000000001
14 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000001011000000000000000000000000000000000000000001
8 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000001100000000000000000000000000000000000000000001
11 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000001101000000000000000000000000000000000000000001
6 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000001110000000000000000000000000000000000000000001
5 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000001111000000000000000000000000000000000000000001
1 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000
2 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000001
4 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000010001000000000000000000000000000000000000000001
4 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000010010000000000000000000000000000000000000000001
4 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000010011000000000000000000000000000000000000000001
4 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000010100000000000000000000000000000000000000000001
3 Instr1: 000000000000000000000000000000000000000000000000000000000000000000000000000000010101000000000000000000000000000000000000000001
11 Instr1: 000000000100000000000000000000000000000000000000000000001000000000100001110001111111000000000000000010000111001000010010000001
12 Instr1: 0000000001000000000000000000000000000000000000000000000010
What I am looking for in output is perhaps: (i am randomly putting values here)
Code :
2000 Instr1[or any sutitable text]: '00000000'
150 Instr1: '10001100'
120 Instr1: '00100000'
and so on
I think the 'uniq' command should be ok, but I am open to anything.
Thanks in advance.