Sponsored Content
Top Forums Shell Programming and Scripting Delete Duplicate records from a tilde delimited file Post 302149253 by radoulov on Wednesday 5th of December 2007 11:47:32 AM
Old 12-05-2007
And another one (if sorting is acceptable):

Code:
sort -ut~ -k1,2 filename


Last edited by radoulov; 12-05-2007 at 12:51 PM.. Reason: ...
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to extract a tilde delimited file in unix

i have a file in unix in which datas are like this 07 01 abc data entry Z3 data entry ASSISTANT Z3 39 08 01 POD peadiatrist Z4 POD PeDIATRY Z4 67 01 operator specialist 00 operator UNSPECIFIED A0 00 ... (12 Replies)
Discussion started by: trichyselva
12 Replies

2. UNIX for Advanced & Expert Users

Duplicate records from oracle to text file.

Hi, I want to fetch duplicate records from an external table to a text file. Pls suggest me. Thanks (1 Reply)
Discussion started by: shilendrajadon
1 Replies

3. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies

4. Shell Programming and Scripting

How to delete duplicate records based on key

For example suppose I have a file which contains data as: $cat data 800,2 100,9 700,3 100,9 200,8 100,3 Now I want the output as 200,8 700,3 800,2 Key is first three characters, I don't want any reords which are having duplicate keys. Like sort +0.0 -0.3 data can we use... (9 Replies)
Discussion started by: sumitc
9 Replies

5. Shell Programming and Scripting

compare fields in a file with duplicate records

Hi: I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file: 99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies

6. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

7. UNIX for Dummies Questions & Answers

Extract records by column value - file non-delimited

the data in my file is has no delimiters. it looks like this: H52082320024740010PH333200612290000930 0.0020080131 D5208232002474000120070306200703060580T1502 TT 1.00 H52082320029180003PH333200702150001 30 100.0020080205 D5208232002918000120070726200707260580T1502 ... (3 Replies)
Discussion started by: jclanc8
3 Replies

8. Shell Programming and Scripting

Remove somewhat Duplicate records from a flat file

I have a flat file that contains records similar to the following two lines; 1984/11/08 7 700000 123456789 2 1984/11/08 1941/05/19 7 700000 123456789 2 The 123456789 2 represents an account number, this is how I identify the duplicate record. The ### signs represent... (4 Replies)
Discussion started by: jolney
4 Replies

9. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

10. UNIX for Advanced & Expert Users

Wanted best way to validate delimited file records

actually i post about this issue before but many folkz miss-understood with my quesion, We are checking for the delimited file records validation Delimited file will have data like this: Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg| Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg|... (3 Replies)
Discussion started by: Seshendranath
3 Replies
AUDITREDUCE(1)						    BSD General Commands Manual 					    AUDITREDUCE(1)

NAME
auditreduce -- select records from audit trail files SYNOPSIS
auditreduce [-A] [-a YYYYMMDD[HH[MM[SS]]]] [-b YYYYMMDD[HH[MM[SS]]]] [-c flags] [-d YYYYMMDD] [-e euid] [-f egid] [-g rgid] [-j id] [-m event] [-o object=value] [-r ruid] [-U] [-u auid] [-v] [file ...] DESCRIPTION
The auditreduce utility selects records from the audit trail files based on the specified criteria. Matching audit records are printed to the standard output in their raw binary form. If no file argument is specified, the standard input is used by default. Use the praudit(1) utility to print the selected audit records in human-readable form. The options are as follows: -A Select all records. -a YYYYMMDD[HH[MM[SS]]] Select records that occurred after or on the given datetime. -b YYYYMMDD[HH[MM[SS]]] Select records that occurred before the given datetime. -c flags Select records matching the given audit classes specified as a comma separated list of audit flags. See audit_control(5) for a description of audit flags. -d YYYYMMDD Select records that occurred on a given date. This option cannot be used with -a or -b. -e euid Select records with the given effective user ID or name. -f egid Select records with the given effective group ID or name. -g rgid Select records with the given real group ID or name. -j id Select records having a subject token with matching ID. -m event Select records with the given event name or number. This option can be used more then once to select records of multiple event types. See audit_event(5) for a description of audit event names and numbers. -o object=value file Select records containing path tokens, where the pathname matches one of the comma delimited extended regular expression con- tained in given specification. Regular expressions which are prefixed with a tilde ('~') are excluded from the search results. These extended regular expressions are processed from left to right, and a path will either be selected or deslected based on the first match. Since commas are used to delimit the regular expressions, a backslash ('') character should be used to escape the comma if it is a part of the search pattern. msgqid Select records containing the given message queue ID. pid Select records containing the given process ID. semid Select records containing the given semaphore ID. shmid Select records containing the given shared memory ID. -r ruid Select records with the given real user ID or name. -U Use unbuffered output. This option is automatically set when the input file is a character device (e.g. /dev/auditpipe), allowing records to be fully processed as they are generated. -u auid Select records with the given audit ID. -v Invert sense of matching, to select records that do not match. EXAMPLES
To select all records associated with effective user ID root from the audit log /var/audit/20031016184719.20031017122634: auditreduce -e root /var/audit/20031016184719.20031017122634 To select all setlogin(2) events from that log: auditreduce -m AUE_SETLOGIN /var/audit/20031016184719.20031017122634 Output from the above command lines will typically be piped to a new trail file, or via standard output to the praudit(1) command. Select all records containing a path token where the pathname contains /etc/master.passwd: auditreduce -o file="/etc/master.passwd" /var/audit/20031016184719.20031017122634 Select all records containing path tokens, where the pathname is a TTY device: auditreduce -o file="/dev/tty[a-zA-Z][0-9]+" /var/audit/20031016184719.20031017122634 Select all records containing path tokens, where the pathname is a TTY except for /dev/ttyp2: auditreduce -o file="~/dev/ttyp2,/dev/tty[a-zA-Z][0-9]+" /var/audit/20031016184719.20031017122634 SEE ALSO
praudit(1), audit_control(5), audit_event(5) HISTORY
The OpenBSM implementation was created by McAfee Research, the security division of McAfee Inc., under contract to Apple Computer Inc. in 2004. It was subsequently adopted by the TrustedBSD Project as the foundation for the OpenBSM distribution. AUTHORS
This software was created by McAfee Research, the security research division of McAfee, Inc., under contract to Apple Computer Inc. Addi- tional authors include Wayne Salamon, Robert Watson, and SPARTA Inc. The Basic Security Module (BSM) interface to audit records and audit event stream format were defined by Sun Microsystems. BSD
January 24, 2004 BSD
All times are GMT -4. The time now is 10:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy