Print Only second Duplicate entry in the file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Print Only second Duplicate entry in the file
# 1  
Old 11-04-2008
Print Only second Duplicate entry in the file

I have file where it contains 2 columns. In two columns the first column is repeated more than once. I wanted to take the unique record in first column
and the corresponding second column value .

The below is the example of the file:

8244100320012955|000b063471a4
8244100320012955|00080e5a6624
8244100330007664|0004bde2ac84
8244100330007664|00080e429228
8244100330011369|0004bd0ec236
8244100330011369|0012c9a1c44a
8244100330016046|00080e0b46c4
8244100330016046|000b06cb18fe

In the above example i need the output as
8244100320012955|00080e5a6624
8244100330007664|00080e429228
8244100330011369|0012c9a1c44a
8244100330016046|000b06cb18fe

I am using the following command
cat f1.txt | cut -d"|" -f 1,2 |sort |uniq -d|wc -l

but the above command is failed

Please help me.

Thank you very much in advance.
# 2  
Old 11-04-2008
Use nawk or /usr/xpg4/bin/awk on Solaris:

Code:
awk -F\| 'count[$1]++==1' infile

# 3  
Old 11-04-2008
MySQL

It has solved the problem.
I didn't get the meaning of the below code
count[$1]++==1
if you can explain it will be of great help.
genius
Thank you very much
# 4  
Old 11-04-2008
Yes,
the count is an associative array (or hash, in Perl dialect):

1. Its keys ($1) are unique.
2. It's post incremented.

Consider the following example:

Code:
$ cat file
8244100320012955|000b063471a4
8244100320012955|00080e5a6624
8244100330007664|0004bde2ac84
8244100330007664|00080e429228
8244100330011369|0004bd0ec236
8244100330011369|0012c9a1c44a
8244100330016046|00080e0b46c4
8244100330016046|000b06cb18fe

$ awk -F\| '{ print NR, $0, count[$1]++ }' file
1 8244100320012955|000b063471a4 0
2 8244100320012955|00080e5a6624 1
3 8244100330007664|0004bde2ac84 0
4 8244100330007664|00080e429228 1
5 8244100330011369|0004bd0ec236 0
6 8244100330011369|0012c9a1c44a 1
7 8244100330016046|00080e0b46c4 0
8 8244100330016046|000b06cb18fe 1

You want only the 1's.

Hope this helps.
# 5  
Old 11-04-2008
MySQL

now i got it.

Thank you very much
for your help
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Deleting duplicate glosses in a dictionary entry

I am working on an Urdu to Hindi dictionary and I have created the following file structure: Headword=Gloss1,Gloss2,Gloss3 i.e. glosses delimited by a comma. It so happens that in some cases (around 6000+ in a file of over 200,000+ the glosses are duplicated. Since this may be a... (3 Replies)
Discussion started by: gimley
3 Replies

2. Shell Programming and Scripting

awk Help: Filter Multiple Entry & print in one line.

AWK Gurus, data: srvhcm01 AZSCI srvhcm01 AZSDB srvhcm01 BZSDB srvhcm01 E2QDI31 srvhcm01 YPDCI srvhcm01 YPDDB srvhcm01 UV2FSCR srvhcm01 UV2FSBI srvhcm01 UV2FSXI srvhcm01 UV2FSUC srvhcm01 UV2FSEP srvhcm01 UV2FSRE srvhcm01 NASCI srvhcm01 NASDB srvhcm01 UV2FSSL srvhcm01 UV2FSDI (7 Replies)
Discussion started by: rveri
7 Replies

3. Shell Programming and Scripting

checking duplicate entry in file

Hi i have a file like 110.10 120.10 -1120 110.10 and the lines are having more than 10k. do we have anycommand to check the duplicate entries in the file. I applied the while loop by greping each line with whole file, but it is taking huge amount of time as the file size is large. ... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

4. Shell Programming and Scripting

Print duplicate lines

I have a file where some of the lines are duplicates. How do I use bash to print all the lines that have duplicates? (2 Replies)
Discussion started by: locoroco
2 Replies

5. UNIX for Dummies Questions & Answers

print duplicate number

input chr1 10 100 chr1 10 100 chr1 20 200 output chr1 10 100 2 chr1 20 200 1 (1 Reply)
Discussion started by: repinementer
1 Replies

6. Shell Programming and Scripting

Need to delete duplicate lease entry

Hi *, I need to delete duplicate lease entries in file according to MAC/IP. I'm having tempfile which contains many lease info and need to have one entry for each IP(not more than that), if it contains more than one entry for same set, need to be deleted that entry... EX: lease... (4 Replies)
Discussion started by: SMNK
4 Replies

7. UNIX for Dummies Questions & Answers

Delete duplicate lines and print to file

OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script. So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup ... (2 Replies)
Discussion started by: bfurlong
2 Replies

8. Shell Programming and Scripting

Not able to print variable value in crontab entry

Hi, I just stuckup to echo a variable that is assigned in .profile file from a crontab entry. I have a entry in .profile as, CONF_FILE_PATH=/home/test/conf export CONF_FILE_PATH I am able to echo it in command line but I am not able to print $CONF_FILE_PATH when i scheduled it in... (4 Replies)
Discussion started by: naren_0101bits
4 Replies

9. UNIX for Dummies Questions & Answers

Remove duplicate entry in one line

Can anyone help me how can i print only the unique entry in a line? MI_AP MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN MBS_AP NT_CAR so that it will on output the one unique entry per line. MI_AP MI_CM MI_MF RC_NAP MBS_AP SF_RAN NT_CAR I can't find the same situation on the knowledge... (5 Replies)
Discussion started by: kharen11
5 Replies

10. HP-UX

Hazardous Duplicate Cron Entry?

Hi All, How to prevent starting of processes that have duplicate entries in cron file, i have written a shell script to validate with "ps |grep" command before starting the process, but still when same process started at same time, it may not be able to detect the existing process. Sample... (3 Replies)
Discussion started by: nag_sundaram
3 Replies
Login or Register to Ask a Question