Removing duplicate records in a file based on single column
Hi,
I want to remove duplicate records including the first line based on column1. For example
inputfile(filer.txt):
-------------
expected output:
----------------
Is it possible to achieve this using awk command ??
I tried below awk command , it is working but i dont want to give two times file name(filer.txt) in the command. I am allowed to give file name only one time.
Please suggest me how to achieve this.
Thanks in advance
Last edited by Franklin52; 08-22-2011 at 03:57 AM..
Reason: Please use code tags for code and data samples, thank you
Hi all,
I would like to extract records of a file based on a condition. The file contains 47 fields, and I would like to extract only those records that match a certain value in one of the columns, e.g.
COL1 COL2 COL3 ............... COL47
1 XX 45 ... (4 Replies)
Hi,
Need to find a duplicate records on the first column,
ANU4501710430989 0000000W20389390
ANU4501710430989 0000000W67065483
ANU4501130050520 0000000W80838713
ANU4501210170685 0000000W69246611... (3 Replies)
Can anyone help me to removing duplicate records from 2 separate files in UNIX?
Please find the sample records for both the files
cat Monday.dat
3FAHP0JA1AR319226MOHMED ATEK 966504453742 SAU2010DE
3LNHL2GC6AR636361HEA DEUK CHOI 821057314531 KOR2010LE
3MEHM0JG7AR652083MUTLAB NAL-NAFISAH... (4 Replies)
I am a newbie to shell scripting ..
I have a .csv file. It has 1000 some rows and about 7 columns...
but before I insert this data to a table I have to parse it and clean it ..basing on the value of the first column..which a string of phone number type...
example below..
column 1 ... (2 Replies)
Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column.
e.g.
a 1
a 2
a 3
b 1
c 1
gives
b 1
c 1
but requires 11 duplicates before it deletes.
Thanks for the help
Video tutorial on how to use code tags in The UNIX... (11 Replies)
I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1.
https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html
Can someone explain this command please? How are there no... (5 Replies)
Hi,
I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines
Command : sort -t'|' -nuk1 file.txt
Input :
38376KZ|09/25/15|1.057
38376KZ|09/25/15|1.057
02006YB|09/25/15|0.859
12593PS|09/25/15|2.803... (2 Replies)
I have csv file with 30, 40 columns
Pasting just three column for problem description
I want to filter record if column 1 matches CN or DN then,
check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345
and if column 2 contains 6789, 6789... (5 Replies)
Hi Experts,
I have csv file with 30, 40 columns
Pasting just 2 column for problem description.
Need to print error if below combination is not present in file
check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same.
For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies
LEARN ABOUT REDHAT
ragator
RAGATOR(1) General Commands Manual RAGATOR(1)NAME
ragator - aggregate argus(8) data file entries.
COPYRIGHT
Copyright (c) 2000-2003 QoSient. All rights reserved.
SYNOPSIS
ragator [-f ragator.conf] [raoptions]
DESCRIPTION
Ragator reads argus(8) data from an argus-file, and merges matching argus flow activity records together. In its default mode of options,
this effectively converts argus(8) files from detail to non-detail mode, and merges periodic flow report records to a single argus record,
thus compressing the argus(8) file to a reduced size.
You can modify the aggregation strategy used by ragator to merge records together, by using the -f ragator.conf option. See ragator(5) for
a complete description of the format and syntax of the flow model file.
OPTIONS
Ragator, like all ra based clients, supports a number of ra options including filtering of input argus records through a terminating filter
expression. See ra(1) for a complete description of ra options.
AUTHORS
Carter Bullard (carter@qosient.com).
SEE ALSO ragator(5)ra(1), rarc(5), argus(8)tcpdump(1),
21 July 1995 RAGATOR(1)