Help needed with Sort and uniq data


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help needed with Sort and uniq data
# 1  
Old 08-17-2009
Help needed with Sort and uniq data

Hi All,

After Sorting directories and files i have got following output as below, now i only want the strings common in them, so the actual output should be as below in the bottom. How do i do that?

Thanks
-adsi

File to be modified:-

Code:
Common Components for ----> AA
atria.basement.sun5
atria.msgcat.JPN.sun5
atria.perl.sun5
ClearCaseAdministrationTools-ent-CINSTALLDIR
com.ccl.feedreader.feature
CMServer.CQ.sun5
com.ccl.feedreader.feature
com.ccl.welcome.bits.feature
com.cic.licensing.feature
com.cqweb-ua.war
ClearCaseDotNetClient
com.help.common.feature
com.help.common.rational.feature
com.java.jre

Common Components for ----> BB
CCRCWebServerINSTALLDIR
atria.basement.sun5
ClearCaseAdministrationTools-CINSTALLDIR
ClearCaseAdministrationTools-ent-CINSTALLDIR
ClearCaseAdministrationTools-pro-CINSTALLDIR
ClearCaseAlbdServer-CINSTALLDIR
com.ccl.feedreader.feature
ClearCaseClearQuestIntegration-CINSTALLDIR
ClearCaseClearQuestMultisite-R
ClearCaseClientComponentsINSTALLDIR
ClearCaseConverters-CINSTALLDIR
com.help.common.rational.feature
ClearCaseCoreComponents-CINSTALLDIR
ClearCaseDotNetClient


Actual OUTPUT:-
Code:
Common Components for ----> AA
atria.basement.sun5
ClearCaseAdministrationTools-ent-CINSTALLDIR
com.ccl.feedreader.feature
ClearCaseDotNetClient
com.help.common.rational.feature
com.java.jre

Common Components for ----> BB
atria.basement.sun5
ClearCaseAdministrationTools-ent-CINSTALLDIR
com.ccl.feedreader.feature
com.help.common.rational.feature
ClearCaseDotNetClient
com.java.jre

# 2  
Old 08-17-2009
Code:
comm -12 aa bb

# 3  
Old 08-17-2009
Hi Peterro,

But in comm we compare two file, but this was just the example. I this case i dont have a second file. I just want the common string in one file, i.e. that repeats more that one time.

Thanks
adsi
# 4  
Old 08-17-2009
something along these lines to start with - although I cannot quite correlate the input and the desired output...:

nawk -f adsi.awk myFile.txt myFile.txt

adsi.awk:
Code:
BEGIN {
  FS=RS=""
}
FNR==NR {
  f[FNR]=$0
  next
}
{
  print $1
  for(j=2; j<=NF;j++)
    for(i in f) {
       if (i==FNR) continue
       n=split(f[i], a, ORS)
       for(i=1;i<=n; i++)
          if( $j == a[i] && !($j in dup)){
             dup[$j]
             print $j
          }
  }
  print RS
  split("", dup)
}


Last edited by vgersh99; 08-17-2009 at 05:10 PM..
# 5  
Old 08-17-2009
Here's an awk program that will retain the output order. One thing.. the java entry is not printed because it doesn't appear in both groups.


Code:
BEGIN {
# init the first group's array index
AINDEX=1
# init the second group's index
BINDEX=1
# init the group flag
FIRSTGROUP=1
}


#function to find match in a group array
# returns index if match, else 0
function check_match()
{
        for (i = 1; i <= AINDEX; i++)
                if (a[i] == $0)
                        return(i)
        return(0)
}

# Store the first a group record
(FNR == 1) {
        a[AINDEX] = $0
# Set that it is printable
        p[AINDEX] = 0
# get next record
        next
}

# Do this clause if we are processing first group
(FIRSTGROUP == 1) {
# Check if we have complete first group
        if ($0 == "" ) {
# end of first group
                FIRSTGROUP = 0
# get the second group header
                getline
# initialize the second group by storing the header
                b[1]=$0
# go to next line
                next
        }

# add it to the first group's array
        a[++AINDEX] = $0
}

#Do this clause if we are processing second group
(FIRSTGROUP == 0) {
# Check to see if this in first group
        if(( j = check_match()) != 0) {
# yes - so add it to b array
                b[++BINDEX] = $0
# create index in p to indicate it's a match in a
                p[j] = 0
# next record
        }
}

# display results
END {
        for( i = 1; i <= AINDEX; i++)
# print first group entry only if it matched b group entry
                if( i in p)
                        print a[i]
        print " "
        for (i = 1; i <= BINDEX; i++)
                print b[i]
}

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort & Uniq -u

Hi All, Below the actual file which i like to sort and Uniq -u /opt/oracle/work/Antony/Shell_Script> cat emp.1st 2233|a.k. shukula |g.m. |sales |12/12/52 |6000 1006|chanchal singhvi |director |sales |03/09/38 |6700... (8 Replies)
Discussion started by: Antony Ankrose
8 Replies

2. UNIX for Dummies Questions & Answers

Uniq and sort -u

Hello all, Need to pick your brains, I have a 10Gb file where each row is a name, I am expecting about 50 names in total. So there are a lot of repetitions in clusters. So I want to do a sort -u file Will it be considerably faster or slower to use a uniq before piping it to sort... (3 Replies)
Discussion started by: senhia83
3 Replies

3. Shell Programming and Scripting

Uniq or sort -u or similar only between { }

Hi ! I am trying to remove doubbled entrys in a textfile only between delimiters. Like that example but i dont know how to do that with sort or similar. input: { aaa aaa } { aaa aaa } output: { aaa } { (8 Replies)
Discussion started by: fugitivus
8 Replies

4. Shell Programming and Scripting

Sort field and uniq

I have a flatfile A.txt 2012/12/04 14:06:07 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:07:22 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:13:27 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 14:07:39 |rain|Boards 1|tampa|merced|merced11 How do i sort and get... (3 Replies)
Discussion started by: sabercats
3 Replies

5. Shell Programming and Scripting

sort | uniq question

Hello, I have a large data file: 1234 8888 bbb 2745 8888 bbb 9489 8888 bbb 1234 8888 aaa 4838 8888 aaa 3977 8888 aaa I need to remove duplicate lines (where the first column is the duplicate). I have been using: sort file.txt | uniq -w4 > newfile.txt However, it seems to keep the... (11 Replies)
Discussion started by: palex
11 Replies

6. Shell Programming and Scripting

Help needed to sort data

Hello All, Today i have been asking lots of question, hope to become good in scripting soon with all the wonderful advices i get. The question is i want to sort data a get uniq string from it. The code i am using to generate the output is:- check_sun() { for i in $SUN_PLATFORM do $ECHO... (0 Replies)
Discussion started by: asirohi
0 Replies

7. Shell Programming and Scripting

Help with Uniq and sort

The key is first field i want only uniq record for the first field in file. I want the output as or output as Appreciate help on this (4 Replies)
Discussion started by: pinnacle
4 Replies

8. Shell Programming and Scripting

Sort, Uniq, Duplicates

Input File is : ------------- 25060008,0040,03, 25136437,0030,03, 25069457,0040,02, 80303438,0014,03,1st 80321837,0009,03,1st 80321977,0009,03,1st 80341345,0007,03,1st 84176527,0047,03,1st 84176527,0047,03, 20000735,0018,03,1st 25060008,0040,03, I am using the following in the script... (5 Replies)
Discussion started by: Amruta Pitkar
5 Replies

9. UNIX for Dummies Questions & Answers

Help with Last,uniq, sort and cut

Using the last, uniq, sort and cut commands, determine how many times the different users have logged in. I know how to use the last command and cut command... i came up with last | cut -f1 -d" " | uniq i dont know if this is right, can someone please help me... thanks (1 Reply)
Discussion started by: jay1228
1 Replies

10. UNIX for Dummies Questions & Answers

sort/uniq

I have a file: Fred Fred Fred Jim Fred Jim Jim If sort is executed on the listed file, shouldn't the output be?: Fred Fred Fred Fred Jim Jim Jim (3 Replies)
Discussion started by: jimmyflip
3 Replies
Login or Register to Ask a Question

Featured Tech Videos