Sponsored Content
Top Forums Shell Programming and Scripting To remove duplicates from pipe delimited file Post 302865745 by ginrkf on Saturday 19th of October 2013 02:38:09 PM
Old 10-19-2013
To remove duplicates from pipe delimited file

Hi some one please help me to remove duplicates from a pipe delimited file based on first two columns.

HTML Code:
123|asdf|sfsd|qwrer
431|yui|qwer|opws
123|asdf|pol|njio
Here My first record and last record are duplicates.As per my requirement I want all the latest records into one file.

I want the output looks like below

Code:
431|yui|qwer|opws
123|asdf|pol|njio

My file is having around 20 million records.So needs a faster output
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to generate a pipe ( | ) delimited file?

:)Hi Friends, I have certain log files extracted. I want it to be converted in pipe ( | ) delimited file. How do i do it? E.g. Account Balance : 123456789 Rs O/P (Account Balance: | 123456789 Rs) Account Balance (Last) > 987654321 Rs O/P (Account Balance (Last) | 987654321 Rs) Last... (5 Replies)
Discussion started by: anushree.a
5 Replies

2. Shell Programming and Scripting

convert a pipe delimited file to a':" delimited file

i have a file whose data is like this:: osr_pe_assign|-120|wg000d@att.com|4| osr_evt|-21|wg000d@att.com|4| pe_avail|-21|wg000d@att.com|4| osr_svt|-11|wg000d@att.com|4| pe_mop|-13|wg000d@att.com|4| instar_ready|-35|wg000d@att.com|4| nsdnet_ready|-90|wg000d@att.com|4|... (6 Replies)
Discussion started by: priyanka3006
6 Replies

3. Shell Programming and Scripting

Remove SPACES between PIPE delimited file

This is my input file with extra information in the HEADER and leading & trailing SPACES between PIPE delimiter. 02/04/2010 Dynamic List Display 1 --------------------------------------------------------------------------------------... (6 Replies)
Discussion started by: srimitta
6 Replies

4. Shell Programming and Scripting

How to convert a space delimited file into a pipe delimited file using shellscript?

Hi All, I have space delimited file similar to the one as shown below.. I need to convert it as a pipe delimited, the values inside the pipe delimited file should be as highlighted... AA ATIU2345098809 009697 005374 BB ATIU2345097809 005445 006518 CC ATIU9685098809 003215 003571 DD... (7 Replies)
Discussion started by: nithins007
7 Replies

5. Shell Programming and Scripting

Help with converting Pipe delimited file to Tab Delimited

I have a file which was pipe delimited, I need to make it tab delimited. I tried with sed but no use cat file | sed 's/|//t/g' The above command substituted "/t" not tab in the place of pipe. Sample file: abc|123|2012-01-30|2012-04-28|xyz have to convert to: abc 123... (6 Replies)
Discussion started by: karumudi7
6 Replies

6. Shell Programming and Scripting

Remove few columns from pipe delimited file

I have file as below column1|column2|column3|column4|column5| fill1|fill2|fill3|fill4|fill5| abc1|abc2|abc3|abc4|abc5| . . . . i need to remove column2,3, from that file column1|column4|column5| fill1|fill4|fill5| abc1|abc4|abc5| . . . (3 Replies)
Discussion started by: greenworld123
3 Replies

7. Shell Programming and Scripting

How to ignore Pipe in Pipe delimited file?

Hi guys, I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file for eg: file 1: xx|yy|"xyz|zzz"|zzz|12... using below awk command awk 'BEGIN {FS=OFS="|" } print $3 i would get xyz But i want as : xyz|zzz to consider as whole column... (13 Replies)
Discussion started by: rohit_shinez
13 Replies

8. Shell Programming and Scripting

Removing duplicates from delimited file based on 2 columns

Hi guys,Got a bit of a bind I'm in. I'm looking to remove duplicates from a pipe delimited file, but do so based on 2 columns. Sounds easy enough, but here's the kicker... Column #1 is a simple ID, which is used to identify the duplicate. Once dups are identified, I need to only keep the one... (2 Replies)
Discussion started by: kevinprood
2 Replies

9. UNIX for Dummies Questions & Answers

Need to convert a pipe delimited text file to tab delimited

Hi, I have a rquirement in unix as below . I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column. ex: Input Text file: 1|A|apple 2|B|bottle excel file to be generated as output as... (9 Replies)
Discussion started by: raja kakitapall
9 Replies

10. Shell Programming and Scripting

How to remove new line characters from data rows in a Pipe delimited file?

I have a file as below Emp1|FirstName|MiddleName|LastName|Address|Pincode|PhoneNumber 1234|FirstName1|MiddleName2|LastName3| Add1 || ADD2|123|000000000 2345|FirstName2|MiddleName3|LastName4| Add1 || ADD2| 234|000000000 OUTPUT : ... (1 Reply)
Discussion started by: styris
1 Replies
bsmrecord(1M)						  System Administration Commands					     bsmrecord(1M)

NAME
bsmrecord - display Solaris audit record formats SYNOPSIS
/usr/sbin/bsmrecord [-d] [ [-a] | [-e string] | [-c class] | [-i id] | [-p programname] | [-s systemcall] | [-h]] DESCRIPTION
The bsmrecord utility displays the event ID, audit class and selection mask, and record format for audit record event types defined in audit_event(4). You can use bsmrecord to generate a list of all audit record formats, or to select audit record formats based on event class, event name, generating program name, system call name, or event ID. There are two output formats. The default format is intended for display in a terminal window; the optional HTML format is intended for viewing with a web browser. OPTIONS
The following options are supported: -a List all audit records. -c class List all audit records selected by class. class is one of the two-character class codes from the file /etc/secu- rity/audit_class. -d Debug mode. Display number of audit records that are defined in audit_event, the number of classes defined in audit_class, any mismatches between the two files, and report which defined events do not have format information available to bsmrecord. -e string List all audit records for which the event ID label contains the string string. The match is case insensitive. -h Generate the output in HTML format. -i id List the audit records having the numeric event ID id. -p programname List all audit records generated by the program programname, for example, audit records generated by a user-space program. -s systemcall List all audit records generated by the system call systemcall, for example, audit records generated by a system call. The -p and -s options are different names for the same thing and are mutually exclusive. The -a option is ignored if any of -c, -e, -i, -p, or -s are given. Combinations of -c, -e, -i, and either -p or -s are ANDed together. EXAMPLES
Example 1: Displaying an Audit Record with a Specified Event ID The following example shows how to display the contents of a specified audit record. % bsmrecord -i 6152 login: terminal login program /usr/sbin/login see login(1) event ID 6152 AUE_login class lo(0x00001000) header-token subject-token text-token error message exit-token Example 2: Displaying an Audit Record with an Event ID Label that Contains a Specified String The following example shows how to display the contents of a audit record with an event ID label that contains the string login. # bsmrecord -e login terminal login program /usr/sbin/login see login(1) event ID 6152 AUE_login class lo(0x00001000) header-token subject-token text-token error message exit-token rlogin program /usr/sbin/login see login(1) - rlogin event ID 6155 AUE_rlogin class lo(0x00001000) header-token subject-token text-token error message exit-token EXIT STATUS
0 Successful operation non-zero Error FILES
/etc/security/audit_class Provides the list of valid classes and the associated audit mask. /etc/security/audit_event Provides the numeric event ID, the literal event name, and the name of the associated system call or program. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsr | +-----------------------------+-----------------------------+ |CSI |Enabled | +-----------------------------+-----------------------------+ |Interface Stability |Unstable | +-----------------------------+-----------------------------+ SEE ALSO
audit_class(4), audit_event(4), attributes(5) DIAGNOSTICS
If unable to read either of its input files or to write its output file, bsmrecord shows the name of the file on which it failed and exits with a non-zero return. If no options are provided, if an invalid option is provided, or if both -s and -p are provided, an error message is displayed and bsm- record displays a usage message then exits with a non-zero return. NOTES
If /etc/security/audit_event has been modified to add user-defined audit events, bsmrecord displays the record format as undefined. SunOS 5.10 14 Jan 2003 bsmrecord(1M)
All times are GMT -4. The time now is 01:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy