Login or Register to Ask a Question and Join Our Community


Programming

[Solved] Removing duplicates from the file and saving as new file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Programming [Solved] Removing duplicates from the file and saving as new file
# 8  
Old 11-22-2012
Try:-
Code:
awk '!arr[$2]++'

# 9  
Old 11-22-2012
Hi

Changing the code works for the file.

I modified your loop script as like this..

Code:
find . -type f -name "c-000_1.pdb.c.c1" | while read file
do
   test=$( test $c-000_1.pdb )
   awk '!arr[$2]++' $c-000_1.pdb.c.c1 > ${test}/tmp; cp ${test}/tmp ${5HT2A-1}_new.pdb
done

However, It doesnt work and shows as syntax error.

I ran the script as " awk -f xx.awk".

Kindly advice how to modify the script.

Many Thanks
Balaji
Last edited by Scrutinizer; 11-22-2012 at 04:17 PM.. Reason: code tags
# 10  
Old 11-22-2012
Please don't run it as an awk script, run it as a shell script.
# 11  
Old 11-22-2012
Hi

I am running as shell script & getting error as

script:

Code:
find . -type f -name "c-000_1.pdb.c.c1" | while read file
do
   test=$( test $c-000_1.pdb )
   awk '!arr[$2]++' $c-000_1.pdb.c.c1 > ${test}/tmp; cp ${test}/tmp ${c-000_1.pdb.c.c1}_new
done


"
Code:
[Balaji@localhost test]$ sh run.sh
run.sh: line 4: /tmp: Is a directory
cp: omitting directory `/tmp'
[Balaji@localhost test]$"


Kindly advice.

Many Thanks
Balaji
Last edited by Scrutinizer; 11-22-2012 at 04:18 PM.. Reason: code tags
# 12  
Old 11-22-2012
Balaji, please use CODE TAGS if you are posting any code fragments or data samples. Perform the highlighted corrections and re-try:-
Code:
find . -type f -name "c-000_1.pdb.c.c1" | while read file
do
       dir=$( dirname $file )
       awk '!arr[$2]++' $file > ${dir}/tmp; cp ${dir}/tmp ${file}_new.pdb
done

This User Gave Thanks to Yoda For This Post:
vbe
# 13  
Old 11-23-2012
Hi

Thanks a lot.

It works.

Many Thanks
Balaji
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies

2. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies

3. UNIX for Dummies Questions & Answers

Grep from pattern file without removing duplicates?

I have been using grep to output whole lines using a pattern file with identifiers (fileA): fig|562.2322.peg.1 fig|562.2322.peg.3 fig|562.2322.peg.3 fig|562.2322.peg.3 fig|562.2322.peg.7 From fileB with corresponding identifiers in the second column: NODE_0 fig|562.2322.peg.1 peg ... (2 Replies)
Discussion started by: Mauve
2 Replies

4. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

5. Shell Programming and Scripting

formatting a file and removing duplicates

Hi, I have a file that I want to change the format of. It is a large file in rows but I want it to be comma separated (comma then a space). The current file looks like this: HI, Joe, Bob, Jack, Jack After I would want to remove any duplicates so it would look like this: HI, Joe,... (2 Replies)
Discussion started by: kylle345
2 Replies

6. Shell Programming and Scripting

Removing Duplicates from file

Hi Experts, Please check the following new requirement. I got data like the following in a file. FILE_HEADER 01cbbfde7898410| 3477945| home| 1 01cbc275d2c122| 3478234| WORK| 1 01cbbe4362743da| 3496386| Rich Spare| 1 01cbc275d2c122| 3478234| WORK| 1 This is pipe separated file with... (3 Replies)
Discussion started by: tinufarid
3 Replies

7. Shell Programming and Scripting

Removing duplicates from log file?

I have a log file with posts looking like this: -- Messages can be delivered by different systems at different times. The id number is used to sort out duplicate messages. What I need is to strip the arrival time from each post, sort posts by id number, and reattach arrival time to respective... (2 Replies)
Discussion started by: Ilja
2 Replies

8. UNIX for Dummies Questions & Answers

removing duplicates of a pattern from a file

hey all, I need some help. I have a text file with names in it. My target is that if a particular pattern exists in that file more than once..then i want to rename all the occurences of that pattern by alternate patterns.. for e.g if i have PATTERN occuring 5 times then i want to... (3 Replies)
Discussion started by: ashisharora
3 Replies

9. Shell Programming and Scripting

Removing duplicates in a sorted file by field.

I have data like this: It's sorted by the 2nd field (TID). envoy,90000000000000634600010001,04/11/2008,23:19:27,RB00266,0015,DETAIL,ERROR, envoy,90000000000000634600010001,04/12/2008,04:23:45,RB00266,0015,DETAIL,ERROR,... (1 Reply)
Discussion started by: kinksville
1 Replies

10. UNIX for Dummies Questions & Answers

removing duplicates from a file

i have a file with some 1000 entries it will contain entries like 1000,ram 2000,pankaj 1001,rahim 1000,ram 2532,govind 2000,pankaj 3000,venkat 2532,govind what i want is i want to extract only the distinct rows from this file so my output should contain only 1000,ram... (2 Replies)
Discussion started by: trichyselva
2 Replies
Login or Register to Ask a Question