05-16-2007
Sort, Uniq, Duplicates
Input File is :
-------------
25060008,0040,03,
25136437,0030,03,
25069457,0040,02,
80303438,0014,03,1st
80321837,0009,03,1st
80321977,0009,03,1st
80341345,0007,03,1st
84176527,0047,03,1st
84176527,0047,03,
20000735,0018,03,1st
25060008,0040,03,
I am using the following in the script :
------------------------------------
cat InputFile | sort -t, -k1,2 | uniq -d > "Duplicates"
This gets 25060008,0040,03, into the Duplicates file.
But I also want 84176527,0047,03, in the Duplicates file.
Basically I want the script to sort on the first 2 fields (delimited by comma) and if duplicates are found for first 2 fields I want it to be written to "Duplicates" file.
Please guide.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I have a file:
Fred
Fred
Fred
Jim
Fred
Jim
Jim
If sort is executed on the listed file, shouldn't the output be?:
Fred
Fred
Fred
Fred
Jim
Jim
Jim (3 Replies)
Discussion started by: jimmyflip
3 Replies
2. UNIX for Dummies Questions & Answers
Using the last, uniq, sort and cut commands, determine how many times the different users have logged in.
I know how to use the last command and cut command...
i came up with last | cut -f1 -d" " | uniq
i dont know if this is right, can someone please help me... thanks (1 Reply)
Discussion started by: jay1228
1 Replies
3. Shell Programming and Scripting
Does anyone have a quick and dirty way of performing a sort and uniq in perl?
How an array with data like:
this is bkupArr BOLADVICE_VN
this is bkupArr MLT6800PROD2A
this is bkupArr MLT6800PROD2A
this is bkupArr BOLADVICE_VN_7YR
this is bkupArr MLT6800PROD2A
I want to sort it... (4 Replies)
Discussion started by: reggiej
4 Replies
4. Shell Programming and Scripting
Hey Guys,
I have file which looks like this,
Contig201#numbPA
Contig1452#nmdynD6PA
dm022p15.r#CG6461PA
dm005e16.f#SpatPA
IGU001_0015_A06.f#CG17593PA
I need to remove duplicates based on the chracter matching upto '#'.
for example if we consider this..
Contig201#numbPA... (4 Replies)
Discussion started by: sharatz83
4 Replies
5. Shell Programming and Scripting
The key is first field i want only uniq record for the first field in file.
I want the output as
or output as
Appreciate help on this (4 Replies)
Discussion started by: pinnacle
4 Replies
6. Shell Programming and Scripting
Hello,
I have a large data file:
1234 8888 bbb
2745 8888 bbb
9489 8888 bbb
1234 8888 aaa
4838 8888 aaa
3977 8888 aaa
I need to remove duplicate lines (where the first column is the duplicate). I have been using:
sort file.txt | uniq -w4 > newfile.txt
However, it seems to keep the... (11 Replies)
Discussion started by: palex
11 Replies
7. Shell Programming and Scripting
Hi All,
I have a text file with the format shown below. Some of the records are duplicated with the only exception being date (Field 15). I want to compare all duplicate records using subscriber number (field 7) and keep only those records with greater date.
... (1 Reply)
Discussion started by: nua7
1 Replies
8. Shell Programming and Scripting
Hi again,
I have files with the following contents
datetime,ip1,port1,ip2,port2,number
How would I find out how many times ip1 field shows up a particular file? Then how would I find out how many time ip1 and port 2 shows up?
Please mind the file may contain 100k lines. (8 Replies)
Discussion started by: LDHB2012
8 Replies
9. Shell Programming and Scripting
Hi !
I am trying to remove doubbled entrys in a textfile only between delimiters.
Like that example but i dont know how to do that with sort or similar.
input:
{
aaa
aaa
}
{
aaa
aaa
}
output:
{
aaa
}
{ (8 Replies)
Discussion started by: fugitivus
8 Replies
10. UNIX for Dummies Questions & Answers
Hello all,
Need to pick your brains,
I have a 10Gb file where each row is a name, I am expecting about 50 names in total. So there are a lot of repetitions in clusters.
So I want to do a
sort -u file
Will it be considerably faster or slower to use a uniq before piping it to sort... (3 Replies)
Discussion started by: senhia83
3 Replies
msgb(9S) Data Structures for Drivers msgb(9S)
NAME
msgb, mblk - STREAMS message block structure
SYNOPSIS
#include <sys/stream.h>
INTERFACE LEVEL
Architecture independent level 1 (DDI/DKI)
DESCRIPTION
A STREAMS message is made up of one or more message blocks, referenced by a pointer to a msgb structure. The b_next and b_prev pointers
are used to link messages together on a QUEUE. The b_cont pointer links message blocks together when a message consists of more than one
block.
Each msgb structure also includes a pointer to a datab(9S) structure, the data block (which contains pointers to the actual data of the
message), and the type of the message.
STRUCTURE MEMBERS
struct msgb *b_next; /* next message on queue */
struct msgb *b_prev; /* previous message on queue */
struct msgb *b_cont; /* next message block */
unsigned char *b_rptr; /* 1st unread data byte of buffer */
unsigned char *b_wptr; /* 1st unwritten data byte of buffer */
struct datab *b_datap; /* pointer to data block */
unsigned char b_band; /* message priority */
unsigned short b_flag; /* used by stream head */
Valid flags are as follows:
MSGMARK Last byte of message is marked.
MSGDELIM Message is delimited.
The msgb structure is defined as type mblk_t.
SEE ALSO
datab(9S)
Writing Device Drivers
STREAMS Programming Guide
SunOS 5.10 11 Apr 1991 msgb(9S)