Shell Programming and Scripting

Hi Don,

Thank you very much for your answer. Please have a look at the input. I don't need to count the duplicate lines. I get the following output using your code.

Code:

80 2
45 2
28 1
6 1
76 2
58 0
30 1
12 3
50 0

---------- Post updated at 11:23 AM ---------- Previous update was at 11:13 AM ----------

Hi Don,

I want to skip all lines where 1st two fields appear in the same order as in a previous line.

The output order doesn't matter.

Thanks

OK. Try:

Code:

awk '
($1, $2) in seen {next}
{       c[$1] += ($3 <= 10)
        c[$2] += ($3 <= 10)
        seen[$1, $2]
}
END {   for(i in c) print i, c[i]}' file

Hi Don,

I appreciate your sincere help!

---------- Post updated 06-02-13 at 02:41 AM ---------- Previous update was 06-01-13 at 11:50 AM ----------

Hi Don cragon,

Is it possible to use your code for large text files(around 90000 lines)? It works perfectly for small files. But for large files, number of counts are incorrect. Is there any solution for this problem?

Thanks!

90000 lines does not sound like a very large file. Could there be some other problem? Could you specify in what way the counts are incorrect? What does your real data look like?

Hi Scrutinizer,

Thanks for your comment. My real file looks as follows(9 columns).

Code:

aaa  bbb   290-307  290  307  A  jj-i      9.51013   17
aaa  bbb   290-307  290  307  A  gh-vb   17.79166  17
aaa  bbb   290-307  290  307  A  hj-k    17.93902  17
jkg  jnm   307-309  307  309  A  kf-cf   5.36406  2
jkg  jnm   307-309  307  309  A  gf-vg   5.31676  2
ght  tyu   307-310  307  310  A  hj-k    4.34634  3
ght  kyu   283-29   283  29   A  io-io   19.89    5

I used the program like this

Code:

awk '
($4, $5) in seen {next}
{       c[$4] += ($8 <=10)
        c[$5] += ($8 <=10)
        seen[$4, $5]
}
END {   for(i in c) print i, c[i]}'  file

The output using the above program

Code:

310 1
29  0
290 0
283 0
307 1
309 0

My desired output is

Code:

290 1
307 3
309 1
310 1
283 0
29  0

I get the desired output.
What's your OS and awk version?

Like MadeInGermany, I get your desired output when I run this script with the data given here. If you aren't getting those results, it is clear that the size of the file is not the problem; it is something in this data that is different from what we saw before. I note that the text I marked in red in your new input data, doesn't line up with the other data in this column in this file. I get the impression that there may be something else in your data that was lost when you copied and pasted your data into this posting. In addition to telling us which OS and version of awk you're using, please also show us the output of the command:

Code:

od -bc filename

with filename replaced by the name of the file containing the data above.

PS. Note also that the output line shown in blue above would not be produced by this script. The print statement in this script will never put two spaces between the output fields???