Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Find and remove duplicate record and print list

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Find and remove duplicate record and print list
# 15  
Old 11-07-2012
Dear RudiC

Your code is working perfect. THanks a lot
# 16  
Old 11-07-2012
Actually, after a little rearranging, pamu's code should do as well:
Code:
awk     'X[$9] {print X[$9] > "removed.txt"}
         {X[$9]=$0}
         END {for (i in X) print X[i] > "clean.txt"}
        ' file

This User Gave Thanks to RudiC For This Post:
jiam912
# 17  
Old 11-07-2012
Dear RudiC,

Thanks alot for your help and time, your support is greatly appreciated... The code works perfect and I got the 2 output files at same time... Smilie
# 18  
Old 11-08-2012
Quote:
Originally Posted by jiam912
Dear RudiC,

Thanks alot for your help and time, your support is greatly appreciated... The code works perfect and I got the 2 output files at same time... Smilie
I don't think there is any difference between my script(post 10) and Rudic's script(same logic, post 16).
But don't know what you are trying...Smilie
How one script gives you desired output and other didn't...Smilie
This User Gave Thanks to pamu For This Post:
jiam912
# 19  
Old 11-08-2012
Dear Pamu
First of all I appreciate your help and support.
Regarding the code, really it was very strange but it was not working well, perhaps I did something wrong. Anyway.. Thanks again for your help... Now my problem is solved thanks to all.

Regards
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Check/print missing number in a consecutive range and remove duplicate numbers

Hi, In an ideal scenario, I will have a listing of db transaction log that gets copied to a DR site and if I have them all, they will be numbered consecutively like below. 1_79811_01234567.arc 1_79812_01234567.arc 1_79813_01234567.arc 1_79814_01234567.arc 1_79815_01234567.arc... (3 Replies)
Discussion started by: newbie_01
3 Replies

2. Shell Programming and Scripting

Modifying text file records, find data in one place in the record and print it elsewhere

Hello, I have some text data that is in the form of multi-line records. Each record ends with the string $$$$ and the next record starts on the next line. RDKit 2D 15 14 0 0 0 0 0 0 0 0999 V2000 5.4596 2.1267 0.0000 C 0 0 0 0 0 0 0 0 0 0 0 ... (5 Replies)
Discussion started by: LMHmedchem
5 Replies

3. Shell Programming and Scripting

Find key pattern and print selected lines for each record

Hi, I need help on a complicated file that I am working on. I wanted to extract important info from a very huge file. It is space delimited file. I have hundred thousands of records in this file. An example content of the inputfile as below:- ## ID Ser402 Old; 23... (2 Replies)
Discussion started by: redse171
2 Replies

4. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

5. Shell Programming and Scripting

Find x and print its record

Hi all, I have a file containing two fields with 154 rows/records/lines (forgive me, my UNIX terminology is not quite up to par yet). I am trying to read from this list, find a value (lets say 0), then print the record/line/row that value falls on (In this case it would be record/line/row #27)?... (5 Replies)
Discussion started by: StudentServitor
5 Replies

6. Shell Programming and Scripting

Find duplicate filenames and remove in different mount point

Hi Gurus, Do any kind souls encounter have the same script as mentioned here. Find and compare filenames in different mount point and remove duplicates. Thanks a million!!! wanna13e (7 Replies)
Discussion started by: wanna13e
7 Replies

7. UNIX for Advanced & Expert Users

Print Full record and substring in that record

I have i got a requirement like below. I have input file which contains following fixed width records. 00000000000088500232007112007111 I need the full record and concatenated with ~ and characters from 1to 5 and concatenated with ~ and charactes from 10 to 15 The out put will be like... (1 Reply)
Discussion started by: ukatru
1 Replies

8. UNIX for Advanced & Expert Users

How to remove duplicate lines of a record without changing the order

Hi all, I have to remove duplicate lines in a file without chainging the order.for eg if i have a record pqr def abc lmn pqr abc mkh hgf the output should be pqr def abc lmn mkh hgf (7 Replies)
Discussion started by: abhi.roy03
7 Replies

9. UNIX for Dummies Questions & Answers

How to extract duplicate records with associated header record

All, I have a task to search through several hundred files and extract duplicate detail records and keep them grouped with their header record. If no duplicate detail record exists, don't pull the header. For example, an input file could look like this: input.txt HA D1 D2 D2 D3 D4 D4... (17 Replies)
Discussion started by: run_eim
17 Replies
Login or Register to Ask a Question