08-20-2011
Thanks for reply jgt
, i am allowed to use awk/sed command alone
. can someone give suggestion how exactly i can code it in single command line.
Quote:
Use the unique option of the sort command.
Sort the file using the unique option. Then use diff between the original and the output (of the sort) file. Then use the diff file to remove the records from the output file of the sort.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi all,
I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g.
COL1 COL2 COL3 ............... COL47
1 XX 45 ... (4 Replies)
Discussion started by: risk_sly
4 Replies
2. Shell Programming and Scripting
Hi,
Need to find a duplicate records on the first column,
ANU4501710430989 0000000W20389390
ANU4501710430989 0000000W67065483
ANU4501130050520 0000000W80838713
ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies
3. Shell Programming and Scripting
Can anyone help me to removing duplicate records from 2 separate files in UNIX?
Please find the sample records for both the files
cat Monday.dat
3FAHP0JA1AR319226MOHMED ATEK 966504453742 SAU2010DE
3LNHL2GC6AR636361HEA DEUK CHOI 821057314531 KOR2010LE
3MEHM0JG7AR652083MUTLAB NAL-NAFISAH... (4 Replies)
Discussion started by: zooby
4 Replies
4. Shell Programming and Scripting
I am a newbie to shell scripting ..
I have a .csv file. It has 1000 some rows and about 7 columns...
but before I insert this data to a table I have to parse it and clean it ..basing on the value of the first column..which a string of phone number type...
example below..
column 1 ... (2 Replies)
Discussion started by: mitr
2 Replies
5. Linux
I have log file having Traffic line
2011-05-21 15:11:50.356599 TCP (6), length: 52) 10.10.10.1.3020 > 10.10.10.254.50404:
2011-05-21 15:11:50.652739 TCP (6), length: 52) 10.10.10.254.50404 > 10.10.10.1.3020:
2011-05-21 15:11:50.652558 TCP (6), length: 89) 10.10.10.1.3020 >... (1 Reply)
Discussion started by: Rastamed
1 Replies
6. UNIX for Dummies Questions & Answers
Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column.
e.g.
a 1
a 2
a 3
b 1
c 1
gives
b 1
c 1
but requires 11 duplicates before it deletes.
Thanks for the help
Video tutorial on how to use code tags in The UNIX... (11 Replies)
Discussion started by: informaticist
11 Replies
7. Shell Programming and Scripting
I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1.
https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html
Can someone explain this command please? How are there no... (5 Replies)
Discussion started by: cokedude
5 Replies
8. Shell Programming and Scripting
Hi,
I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines
Command : sort -t'|' -nuk1 file.txt
Input :
38376KZ|09/25/15|1.057
38376KZ|09/25/15|1.057
02006YB|09/25/15|0.859
12593PS|09/25/15|2.803... (2 Replies)
Discussion started by: parithi06
2 Replies
9. Shell Programming and Scripting
I have csv file with 30, 40 columns
Pasting just three column for problem description
I want to filter record if column 1 matches CN or DN then,
check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345
and if column 2 contains 6789, 6789... (5 Replies)
Discussion started by: as7951
5 Replies
10. Shell Programming and Scripting
Hi Experts,
I have csv file with 30, 40 columns
Pasting just 2 column for problem description.
Need to print error if below combination is not present in file
check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same.
For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies
SORT(1) General Commands Manual SORT(1)
NAME
sort - sort a file of ASCII lines
SYNOPSIS
sort [-bcdfimnru] [-tc] [-o name] [+pos1] [-pos2] file ...
OPTIONS
-b Skip leading blanks when making comparisons
-c Check to see if a file is sorted
-d Dictionary order: ignore punctuation
-f Fold upper case onto lower case
-i Ignore nonASCII characters
-m Merge presorted files
-n Numeric sort order
-o Next argument is output file
-r Reverse the sort order
-t Following character is field separator
-u Unique mode (delete duplicate lines)
EXAMPLES
sort -nr file # Sort keys numerically, reversed
sort +2 -4 file # Sort using fields 2 and 3 as key
sort +2 -t: -o out # Field separator is :
sort +.3 -.6 # Characters 3 through 5 form the key
DESCRIPTION
Sort sorts one or more files. If no files are specified, stdin is sorted. Output is written on standard output, unless -o is specified.
The options +pos1 -pos2 use only fields pos1 up to but not including pos2 as the sort key, where a field is a string of characters delim-
ited by spaces and tabs, unless a different field delimiter is specified with -t. Both pos1 and pos2 have the form m.n where m tells the
number of fields and n tells the number of characters. Either m or n may be omitted.
SEE ALSO
comm(1), grep(1), uniq(1).
SORT(1)