Sponsored Content
Top Forums Shell Programming and Scripting Removing Duplicates from file Post 302552198 by jimmymj on Friday 2nd of September 2011 02:14:28 AM
Old 09-02-2011
Hi Tinu,

Please try out the below command

Code:
sort -t "|" +1 -3 test_file |uniq -u

or
Code:
sed '1d' test_file|sort -t "|" +1 -3|uniq -u (removing the header line)

Smilie
~jimmy

Last edited by Franklin52; 09-02-2011 at 03:42 AM.. Reason: Please use code tags for code and data samples, thank you
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

removing duplicates from a file

i have a file with some 1000 entries it will contain entries like 1000,ram 2000,pankaj 1001,rahim 1000,ram 2532,govind 2000,pankaj 3000,venkat 2532,govind what i want is i want to extract only the distinct rows from this file so my output should contain only 1000,ram... (2 Replies)
Discussion started by: trichyselva
2 Replies

2. Shell Programming and Scripting

Removing duplicates in a sorted file by field.

I have data like this: It's sorted by the 2nd field (TID). envoy,90000000000000634600010001,04/11/2008,23:19:27,RB00266,0015,DETAIL,ERROR, envoy,90000000000000634600010001,04/12/2008,04:23:45,RB00266,0015,DETAIL,ERROR,... (1 Reply)
Discussion started by: kinksville
1 Replies

3. UNIX for Dummies Questions & Answers

removing duplicates of a pattern from a file

hey all, I need some help. I have a text file with names in it. My target is that if a particular pattern exists in that file more than once..then i want to rename all the occurences of that pattern by alternate patterns.. for e.g if i have PATTERN occuring 5 times then i want to... (3 Replies)
Discussion started by: ashisharora
3 Replies

4. Shell Programming and Scripting

Removing duplicates from log file?

I have a log file with posts looking like this: -- Messages can be delivered by different systems at different times. The id number is used to sort out duplicate messages. What I need is to strip the arrival time from each post, sort posts by id number, and reattach arrival time to respective... (2 Replies)
Discussion started by: Ilja
2 Replies

5. Shell Programming and Scripting

formatting a file and removing duplicates

Hi, I have a file that I want to change the format of. It is a large file in rows but I want it to be comma separated (comma then a space). The current file looks like this: HI, Joe, Bob, Jack, Jack After I would want to remove any duplicates so it would look like this: HI, Joe,... (2 Replies)
Discussion started by: kylle345
2 Replies

6. Shell Programming and Scripting

Removing duplicates depending on file size

Hi all, I am working with a huge amount of files in a Linux environment and I was trying to filter my data. Here's what my data looks like Name............................Size OLUSDN.gf.gif-1.JPEG.......5 kb LKJFDA01.gf.gif-1.JPEG.....3 kb LKJFDA01.gf.gif-2.JPEG.....1 kb... (7 Replies)
Discussion started by: Error404
7 Replies

7. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

8. UNIX for Dummies Questions & Answers

Grep from pattern file without removing duplicates?

I have been using grep to output whole lines using a pattern file with identifiers (fileA): fig|562.2322.peg.1 fig|562.2322.peg.3 fig|562.2322.peg.3 fig|562.2322.peg.3 fig|562.2322.peg.7 From fileB with corresponding identifiers in the second column: NODE_0 fig|562.2322.peg.1 peg ... (2 Replies)
Discussion started by: Mauve
2 Replies

9. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies

10. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies
CSREQ(1)						    BSD General Commands Manual 						  CSREQ(1)

NAME
csreq -- Expert tool for manipulating Code Signing Requirement data SYNOPSIS
csreq [-v] -r requirement-input -t csreq [-v] -r requirement-input -b outputfile DESCRIPTION
The csreq command manipulates Code Signing Requirement data. It reads one requirement from a file or command arguments, converts it into internal form, checks it, and then optionally outputs it in a different form. The options are as follows: -b path Requests that the requirement read be written in binary form to the path given. -r requirement-input Specifies the input requirement. See "specifying requirements" below. This is exactly the same format as is accepted by the -r and -R options of the codesign(1) command. -t Requests that the requirement read be written as text to standard output. -v Increases the verbosity of output. Multiple instances of -v produce increasing levels of commentary output. In the first synopsis form, csreq reads a Code Requirement and writes it to standard output as canonical source text. Note that with text input, this actually compiles the requirement into internal form and then converts it back to text, giving you the system's view of the requirement code. In the second synopsis form, csreq reads a Code Requirement and writes its binary representation to a file. This is the same form produced by the SecRequirementCopyData API, and is readily acceptable as input to Code Signing verification APIs. It can also be used as input to subse- quent invocations of csreq by passing the filename to the -r option. SPECIFYING REQUIREMENTS
The requirement argument (-r) can be given in various forms. A plain text argument is taken to be a path to a file containing the require- ment. This program will accept both binary files containing properly compiled requirements code, and source files that are automatically com- piled for use. An argument of "-" requests that the requirement(s) are read from standard input. Again, standard input can contain either binary form or text. Finally, an argument that begins with an equal sign "=" is taken as a literal requirements source text, and is compiled accordingly for use. EXAMPLES
To compile an explicit requirement program and write its binary form to file "output": csreq -r="identifier com.foo.test" -b output.csreq To display the requirement program embedded at offset 1234 of file "foo": tail -b 1234 foo | csreq -r- -t FILES
DIAGNOSTICS
The csreq program exits 0 on success or 1 on failure. Errors in arguments yield exit code 2. SEE ALSO
codesign(1) HISTORY
The csreq command first appeared in Mac OS 10.5.0 . BSD
June 1, 2006 BSD
All times are GMT -4. The time now is 07:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy