Sponsored Content
Top Forums Shell Programming and Scripting How to find Duplicate Records in a text file Post 302248161 by G.Aavudai on Friday 17th of October 2008 04:53:34 AM
Old 10-17-2008
Bug How to find Duplicate Records in a text file

Hi all
pls help me by providing soln for my problem
I'm having a text file which contains duplicate records .

Example:
abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 2123 1343 2176 7654 3252 8765 5643 3452
aer 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 2123 1343 2176 7654 3252 8765 5643 3452


I want to compare each and every column of every row with rest columns of file
Im doing by following method:
two for loops
first[$i]=`head -$i filename.txt | tail -1 | cut -d" " -f$num1-$num2`
in same way im extracting entire columns using another foer loop and comparing with "if" condition. Its showing wrong value
can any one help me.
By
G.Aavudai Nayagam.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove all instances of duplicate records from the file

Hi experts, I am new to scripting. I have a requirement as below. File1: A|123|NAME1 A|123|NAME2 B|123|NAME3 File2: C|123|NAME4 C|123|NAME5 D|123|NAME6 1) I have 2 merge both the files. 2) need to do a sort ( key fields are first and second field) 3) remove all the instances... (3 Replies)
Discussion started by: vukkusila
3 Replies

2. UNIX for Advanced & Expert Users

Duplicate records from oracle to text file.

Hi, I want to fetch duplicate records from an external table to a text file. Pls suggest me. Thanks (1 Reply)
Discussion started by: shilendrajadon
1 Replies

3. Shell Programming and Scripting

find duplicate records... again

Hi all: Let's suppose I have a file like this (but with many more records). XX ME 342 8688 2006 7 6 3c 60.029 -38.568 2901 0001 74 4 7603 8 969.8 958.4 3.6320 34.8630 985.5 973.9 3.6130 34.8600 998.7 986.9 3.6070 34.8610 1003.6 991.7 ... (4 Replies)
Discussion started by: rleal
4 Replies

4. Shell Programming and Scripting

compare fields in a file with duplicate records

Hi: I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file: 99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies

5. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

6. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

7. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

8. Shell Programming and Scripting

Remove somewhat Duplicate records from a flat file

I have a flat file that contains records similar to the following two lines; 1984/11/08 7 700000 123456789 2 1984/11/08 1941/05/19 7 700000 123456789 2 The 123456789 2 represents an account number, this is how I identify the duplicate record. The ### signs represent... (4 Replies)
Discussion started by: jolney
4 Replies

9. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

10. Shell Programming and Scripting

Modifying text file records, find data in one place in the record and print it elsewhere

Hello, I have some text data that is in the form of multi-line records. Each record ends with the string $$$$ and the next record starts on the next line. RDKit 2D 15 14 0 0 0 0 0 0 0 0999 V2000 5.4596 2.1267 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 ... (5 Replies)
Discussion started by: LMHmedchem
5 Replies
gipddecode(1)						      General Commands Manual						     gipddecode(1)

NAME
gipddecode - Decode a GIPD stream into human readable form. SYNOPSIS
gipddecode [options] <gipd-file DESCRIPTION
gipddecode decodes a Granite Image Printer Driver (GIPD) stream into human readable form. Granite Systems was acquired by Monotype Imag- ing. A GIPD stream is the printer langauge used by the Lexmark X500 and the Dell 1125 MFP printers. COMMAND LINE OPTIONS
These are the options that can appear on the command line. -d basename Basename of .pbm file for saving decompressed planes. -h Print hex file offsets. -o Print file offsets. -D level Set Debug level [0]. EXAMPLES
Decode an GIPD stream file. $ gipddecode -h x500-mono.prn 0: OFST 0 len=128 80: OFST 1 len=128 100: OFST 2 len=128 180: OFST 3 len=128 200: OFST 4 len=128 [SNIP] 27d80: OFST 1275 len=128 27e00: 33%-12345X@PJL SET DISPINFOWHILEPRINT=OFF 27e2a: @PJL SET DISPATPAPERCHANG=OFF 27e49: @PJL SET JAMRECOVERY=ON 27e62: @PJL SET OUTPUTBLANKPAPER=OFF 27e81: @PJL SET PRINTSLOWLY=OFF 27e9b: @PJL SET REVERSEPRINT=OFF 27eb6: 33%-12345X 27ebf: GDIJ len=108 unk0=0, unk1=0, unk2=0, unk3=0, unk4=0 unk5=16777221(0x1000005), unk6=0, unk7=0, unk8=0, paper=0 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 27f37: GDIP len=52 nplane = 1, w254 = 4896, h254 = 6110 132018de, 0, 0, 0, 0, 1000000, 0, 0, 0, 0, 0, 0, 0, 27f77: GDIB 0 len=60(0x3c) 27fcb: GDIB 1 len=252(0xfc) 280df: GDIB 2 len=124(0x7c) 28173: GDIB 3 len=956(0x3bc) 28547: GDIB 4 len=1692(0x69c) 28bfb: GDIB 5 len=572(0x23c) 28e4f: GDIB 6 len=17084(0x42bc) 2d123: GDIB 7 len=18108(0x46bc) 317f7: GDIB 8 len=8508(0x213c) 3394b: GDIB 9 len=1756(0x6dc) 3403f: GDIB 10 len=1596(0x63c) 34693: GDIB 11 len=892(0x37c) 34a27: GDIB 12 len=2332(0x91c) 3535b: GDIB 13 len=8380(0x20bc) 3742f: GDIB 14 len=3452(0xd7c) 381c3: GDIB 15 len=60(0x3c) 38217: GDIB 16 len=1468(0x5bc) 387eb: GDIB 17 len=2076(0x81c) 3901f: GDIB 18 len=284(0x11c) 39153: GDIB 19 len=1660(0x67c) 397e7: GDIB 20 len=2908(0xb5c) 3a35b: GDIB 21 len=156(0x9c) 3a40f: GDIB 22 len=188(0xbc) 3a4e3: GDIB 23 len=220(0xdc) 3a5d7: GDIB 24 len=60(0x3c) 3a62b: PIDG 3a637: JIDG Total Size = 75843(0x12843) FILES
/usr/bin/gipddecode AUTHOR
Rick Richardson <rick.richardson@comcast.net>
All times are GMT -4. The time now is 08:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy