07-14-2006
Removing duplicates [sort , uniq]
Hey Guys,
I have file which looks like this,
Contig201#numbPA
Contig1452#nmdynD6PA
dm022p15.r#CG6461PA
dm005e16.f#SpatPA
IGU001_0015_A06.f#CG17593PA
I need to remove duplicates based on the chracter matching upto '#'.
for example if we consider this..
Contig201#numbPA
Contig1452#nmdynD6PA
Contig201#numbPA
Contig1452#nmdynD6PA
dm022px15.r#CG6461PA
Contig201#Xg123
Contig1452#Xg345
dm022px15.r#PA12sy
Contig430#Pp345
I need the output to be ...
Contig201#numbPA
Contig1452#nmdynD6PA
dm022px15.r#CG6461PA
Contig430#Pp345
I know I could have used [sort infile | uniq -w9 > outfile] if the characters upto '#' were of the same length.
peace
sharat
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I have a file:
Fred
Fred
Fred
Jim
Fred
Jim
Jim
If sort is executed on the listed file, shouldn't the output be?:
Fred
Fred
Fred
Fred
Jim
Jim
Jim (3 Replies)
Discussion started by: jimmyflip
3 Replies
2. Shell Programming and Scripting
Input File is :
-------------
25060008,0040,03,
25136437,0030,03,
25069457,0040,02,
80303438,0014,03,1st
80321837,0009,03,1st
80321977,0009,03,1st
80341345,0007,03,1st
84176527,0047,03,1st
84176527,0047,03,
20000735,0018,03,1st
25060008,0040,03,
I am using the following in the script... (5 Replies)
Discussion started by: Amruta Pitkar
5 Replies
3. UNIX for Dummies Questions & Answers
Hello experts,
I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter
sort -u -k 2,2 File.csv > Output.csv
File.csv
File Name|Document Name|Document Title|Organization
Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies
4. Shell Programming and Scripting
The key is first field i want only uniq record for the first field in file.
I want the output as
or output as
Appreciate help on this (4 Replies)
Discussion started by: pinnacle
4 Replies
5. Shell Programming and Scripting
Hello,
I have a large data file:
1234 8888 bbb
2745 8888 bbb
9489 8888 bbb
1234 8888 aaa
4838 8888 aaa
3977 8888 aaa
I need to remove duplicate lines (where the first column is the duplicate). I have been using:
sort file.txt | uniq -w4 > newfile.txt
However, it seems to keep the... (11 Replies)
Discussion started by: palex
11 Replies
6. Shell Programming and Scripting
Hi All,
I have a text file with the format shown below. Some of the records are duplicated with the only exception being date (Field 15). I want to compare all duplicate records using subscriber number (field 7) and keep only those records with greater date.
... (1 Reply)
Discussion started by: nua7
1 Replies
7. Shell Programming and Scripting
I have a flatfile A.txt
2012/12/04 14:06:07 |trees|Boards 2, 3|denver|mekong|mekong12
2012/12/04 17:07:22 |trees|Boards 2, 3|denver|mekong|mekong12
2012/12/04 17:13:27 |trees|Boards 2, 3|denver|mekong|mekong12
2012/12/04 14:07:39 |rain|Boards 1|tampa|merced|merced11
How do i sort and get... (3 Replies)
Discussion started by: sabercats
3 Replies
8. Shell Programming and Scripting
Hi !
I am trying to remove doubbled entrys in a textfile only between delimiters.
Like that example but i dont know how to do that with sort or similar.
input:
{
aaa
aaa
}
{
aaa
aaa
}
output:
{
aaa
}
{ (8 Replies)
Discussion started by: fugitivus
8 Replies
9. UNIX for Dummies Questions & Answers
Hello all,
Need to pick your brains,
I have a 10Gb file where each row is a name, I am expecting about 50 names in total. So there are a lot of repetitions in clusters.
So I want to do a
sort -u file
Will it be considerably faster or slower to use a uniq before piping it to sort... (3 Replies)
Discussion started by: senhia83
3 Replies
10. Shell Programming and Scripting
Hi All,
Below the actual file which i like to sort and Uniq -u
/opt/oracle/work/Antony/Shell_Script> cat emp.1st
2233|a.k. shukula |g.m. |sales |12/12/52 |6000
1006|chanchal singhvi |director |sales |03/09/38 |6700... (8 Replies)
Discussion started by: Antony Ankrose
8 Replies
LEARN ABOUT MOJAVE
make_sockdfr
MAKE_SOCKDFR(8) System Manager's Manual MAKE_SOCKDFR(8)
NAME
make_sockdfr - Generates frozen route file for SOCKS server
SYNOPSIS
make_sockdfr [infile [outfile] ]
DESCRIPTION
make_sockdfr reads in a plain-text route file for the SOCKS server and produces a frozen route file as the output.
Both arguments are optional. The default for infile is /etc/sockd.route; the default for outfile is /etc/sockd.fr. You may specify infile
while omitting outfile, but you cannot specify outfile without also speficying infile.
The contents of the frozen route file is essentially the memory image of the parsed input file. Using the frozen route file can reduce the
start-up delay of the SOCKS server program since it no longer has to parse the file contents.
When the SOCKS server starts, it always looks for the frozen route file /etc/sockd.fr first. If that file is not found, it then tries to
use the plain-text route file /etc/sockd.route. If you use frozen route file, you must remember to run make_sockdfr every time after you
modify the plain-text file or the SOCKS server will continue to use the frozen version of a previous route file.
To find out the contents of a frozen route file, use dump_sockdfr.
FILES
/etc/sockd.fr, /etc/sockd.route
SEE ALSO
dump_sockdfr(8), sockd.fr(5), sockd.route(5)
AUTHOR
Ying-Da Lee, yingda@esd.sgi.com or yingda@best.com
May 6, 1996 MAKE_SOCKDFR(8)