The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Operating Systems > HP-UX
How to get Duplicate rows in a file How to get Duplicate rows in a file
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


HP-UX HP-UX (Hewlett Packard UniX) is Hewlett-Packard's proprietary implementation of the Unix operating system, based on System V.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to extract duplicate rows bobbygsk Shell Programming and Scripting 5 11-20-2008 11:31 AM
Remove duplicate rows of a file based on a value of a column risk_sly UNIX for Dummies Questions & Answers 7 09-26-2008 07:26 AM
How to find all duplicate rows using awk purvi Shell Programming and Scripting 7 08-21-2008 02:34 PM
how to delete duplicate rows in a file vamshikrishnab Shell Programming and Scripting 5 06-18-2008 11:00 AM
duplicate rows in a file infyanurag Shell Programming and Scripting 3 05-22-2008 01:39 AM

Closed Thread
Page 1 of 2 1 2
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 04-02-2009
raghu.iv85 raghu.iv85 is offline
Registered User
  
 

Join Date: Mar 2009
Posts: 20
How to get Duplicate rows in a file
Hi all,

I have written one shell script. The output file of this script is having sql output.

In that file, I want to extract the rows which are having multiple entries(duplicate rows).
For example, the output file will be like the following way.

===============================================================
<SH12_MC30_CE_VS_NY_HIST_T>
===============================================================
397 44847
400 33653
401 46455
===============================================================
<SH12_MC30_CE_VS_NY_HIST_T_BKP>
===============================================================
397 44847
398 40107
399 39338
400 33653


In this output, I want numeric duplicate rows only. Suppose this file is having lines to separate the values, those lines also considered as duplicate rows. So I want only the out put from this file which is having more than one entry and which is related to numbers.

Can anyone please tell me the command?
Thanks in advance.

Regards,
Raghu.
  #2 (permalink)  
Old 04-02-2009
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
  
 

Join Date: Feb 2004
Location: NM
Posts: 5,770
Code:
cat file1 file2 | \
   grep -v -e '^='  -e '^<' | \
   awk '{ arr[$0]++} END{ for (i in arr) { if(arr[i]>1) { print i}  }}' > newfile
cat the files into grep to remove filenames in grep output, grep removes the header lines
  #3 (permalink)  
Old 04-02-2009
raghu.iv85 raghu.iv85 is offline
Registered User
  
 

Join Date: Mar 2009
Posts: 20
Hi Jim,


I could understand till second line of ur command.
I couldn't understand the awk part. Becoz i dont know the awk features.
But it is working. Thank you very much for that. 'awk' is so nice.
Can you give any aother way to get it instead of awk.

Thanks & Regards,
Raghunadh.
  #4 (permalink)  
Old 04-02-2009
siquadri siquadri is offline
Registered User
  
 

Join Date: Apr 2009
Posts: 44
Quote:
Originally Posted by raghu.iv85 View Post
Hi all,

I have written one shell script. The output file of this script is having sql output.

In that file, I want to extract the rows which are having multiple entries(duplicate rows).
For example, the output file will be like the following way.

===============================================================
<SH12_MC30_CE_VS_NY_HIST_T>
===============================================================
397 44847
400 33653
401 46455
===============================================================
<SH12_MC30_CE_VS_NY_HIST_T_BKP>
===============================================================
397 44847
398 40107
399 39338
400 33653


In this output, I want numeric duplicate rows only. Suppose this file is having lines to separate the values, those lines also considered as duplicate rows. So I want only the out put from this file which is having more than one entry and which is related to numbers.

Can anyone please tell me the command?
Thanks in advance.

Regards,
Raghu.
Try this

Code:
#!/bin/ksh
sort $1 > sortedfile
nawk '{ while (getline < sortedfile >0); array[n++]=$0; compare and remove non dup record here}'
  #5 (permalink)  
Old 04-02-2009
vgersh99's Avatar
vgersh99 vgersh99 is online now Forum Staff  
Moderator
  
 

Join Date: Feb 2005
Location: Boston, MA
Posts: 5,122
Code:
nawk '/^[0-9]/ {a[$0]++} END {for (i in a) if (a[i]>1) print i}' myOutputFile
  #6 (permalink)  
Old 04-02-2009
raghu.iv85 raghu.iv85 is offline
Registered User
  
 

Join Date: Mar 2009
Posts: 20
Hi vgersh99,

Thank you very much for ur reply.
'nawk' command id nice. But I dont know the 'awk' functionalities. So if I put this command in my script then I cant explain this command to anyone. So can you please provide me the command instead of 'awk' and 'nawk'.


Thanks in advance,

Regards,
Raghu.
  #7 (permalink)  
Old 04-02-2009
joeyg's Avatar
joeyg joeyg is offline Forum Staff  
modérateur
  
 

Join Date: Dec 2007
Location: Home of 17-time world champion Boston Celtics
Posts: 1,311
Wink another way to approach
I used awk at end only to handle output format. This could be done with a cut command also, although extra care is necessary for positioning.


Code:
> cat file9
===============================================================
<SH12_MC30_CE_VS_NY_HIST_T>
===============================================================
397 44847
400 33653
401 46455
===============================================================
<SH12_MC30_CE_VS_NY_HIST_T_BKP>
===============================================================
397 44847
398 40107
399 39338
400 33653

> grep "^[0-9]" file9 | sort | uniq -cd
      2 397 44847
      2 400 33653

> grep "^[0-9]" file9 | sort | uniq -cd | awk '{print $2" "$3}'
397 44847
400 33653
and, if your really don't want awk
Code:
> grep "^[0-9]" file9 | sort | uniq -cd | tr -s " " | cut -d" " -f3-4
397 44847
400 33653
Added quicker way -->
Code:
> grep "^[0-9]" file9 | sort | uniq -d 
397 44847
400 33653
Last edited by joeyg; 04-02-2009 at 01:38 PM.. Reason: added quicker way
Closed Thread
Page 1 of 2 1 2

Bookmarks

« abnormal process HPUX | User unlock problem in Tru64 4.0F Server »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Hybrid Mode Hybrid Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 03:58 AM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0