If you'd rather create both file in one step and you don't need the output files to be sorted, the following will read your sample input file twice and produce two output files. The output file named duplicated will contain the lines that have field 2 values that appear on more than one line and the output file named non-duplicated will contain the lines that have field 2 values that only appear once:
If you need the output files to be in sorted order, sort the input file before running the above.
If you are running this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
I have a input file with formating:
6000000901 ;36200103 ;h3a01f496 ;
2000123605 ;36218982 ;heefa1328 ;
2000273132 ;36246985 ;h08c5cb71 ;
2000041207 ;36246985 ;heef75497 ;
Each fields is seperated by semi-comma. Sometime, the second files is... (6 Replies)
I have searched the internet for duplicate row extracting.
All I have seen is extracting good rows or eliminating duplicate rows.
How do I extract duplicate rows from a flat file in unix.
I'm using Korn shell on HP Unix.
For.eg.
FlatFile.txt
========
123:456:678
123:456:678
123:456:876... (5 Replies)
I want to duplicate a row if found two or more values in a particular column for corresponding row which is delimitted by comma.
Input
abc,line one,value1
abc,line two, value1, value2
abc,line three,value1
needs to converted to
abc,line one,value1
abc,line two, value1
abc,line... (8 Replies)
if the key (A or B or ...others) has 4 in its 3rd column the 1st A row has to form 4 dupicates along with the all the values of A in 4th column (2.9, 3.8, 4.2) .
Hope I explain the question clearly.
Cheers
Ruby
input
"A" 1 4 2.9
"A" 2 5 ... (7 Replies)
Hi! I have a file as below:
line1
line2
line2
line3
line3
line3
line4
line4
line4
line4
I would like to extract duplicate lines (not unique, triplicate or quadruplicate lines). Output will be as below:
line2
line2
I would appreciate if anyone can help. Thanks. (4 Replies)
I am new to this forum and this is my first post.
I am looking at an old post with exactly the same name. Can not paste URL because I do not have 5 posts
My requirement is exactly opposite.
I want to get rid of duplicate rows and try to append the values of columns in those rows
... (10 Replies)
Hi All,
I need to extract duplicate rows from a file and write these bad records into another file. And need to have a count of these bad records.
i have a command
awk '
{s++}
END {
for(i in s) {
if(s>1) {
print i
}
}
}' ${TMP_DUPE_RECS}>>${TMP_BAD_DATA_DUPE_RECS}... (5 Replies)
Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed.
example of my file... (3 Replies)
JOIN(1) General Commands Manual JOIN(1)NAME
join - relational database operator
SYNOPSIS
join [ options ] file1 file2
DESCRIPTION
Join forms, on the standard output, a join of the two relations specified by the lines of file1 and file2. If one of the file names is the
standard input is used.
File1 and file2 must be sorted in increasing ASCII collating sequence on the fields on which they are to be joined, normally the first in
each line.
There is one line in the output for each pair of lines in file1 and file2 that have identical join fields. The output line normally con-
sists of the common field, then the rest of the line from file1, then the rest of the line from file2.
Input fields are normally separated spaces or tabs; output fields by space. In this case, multiple separators count as one, and leading
separators are discarded.
The following options are recognized, with POSIX syntax.
-a n In addition to the normal output, produce a line for each unpairable line in file n, where n is 1 or 2.
-v n Like -a, omitting output for paired lines.
-e s Replace empty output fields by string s.
-1 m
-2 m Join on the mth field of file1 or file2.
-jn m Archaic equivalent for -n m.
-ofields
Each output line comprises the designated fields. The comma-separated field designators are either 0, meaning the join field, or
have the form n.m, where n is a file number and m is a field number. Archaic usage allows separate arguments for field designators.
-tc Use character c as the only separator (tab character) on input and output. Every appearance of c in a line is significant.
EXAMPLES
sort /adm/users | join -t: -a 1 -e "" - bdays
Add birthdays to password information, leaving unknown birthdays empty. The layout of is given in users(6); bdays contains sorted
lines like
tr : ' ' </adm/users | sort -k 3 3 >temp
join -1 3 -2 3 -o 1.1,2.1 temp temp | awk '$1 < $2'
Print all pairs of users with identical userids.
SOURCE
/sys/src/cmd/join.c
SEE ALSO sort(1), comm(1), awk(1)BUGS
With default field separation, the collating sequence is that of sort -b -ky,y; with -t, the sequence is that of sort -tx -ky,y.
One of the files must be randomly accessible.
JOIN(1)