Remove duplicates Post: 302898180

Login or Register to Ask a Question and Join Our Community

Sponsored Content

Top Forums Shell Programming and Scripting Remove duplicates Post 302898180 by dtdt on Saturday 19th of April 2014 10:43:00 PM

Old

04-19-2014

Registered User

Remove duplicates

I have a file with the following format:

fields seperated by "|"

title1|something class|long...content1|keys
title2|somhing class|log...content1|kes
title1|sothing class|lon...content1|kes
title3|shing cls|log...content1|ks

I want to remove all duplicates with the same "title field"(the 1st column) regardless anything following in that line.

thanks a lot.

dtdt

View Public Profile for dtdt

Find all posts by dtdt

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra...

2. Shell Programming and Scripting

Script to remove duplicates

Hi I need a script that removes the duplicate records and write it to a new file for example I have a file named test.txt and it looks like abcd.23 abcd.24 abcd.25 qwer.25 qwer.26 qwer.98 I want to pick only $1 and compare with the next record and the output should be abcd.23...

3. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ...

4. UNIX for Dummies Questions & Answers

Remove duplicates from a file

Can u tell me how to remove duplicate records from a file?

5. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk...

6. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC...

7. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in...

8. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert...

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001...

10. Shell Programming and Scripting

How to remove duplicates using for loop?

values=(1 2 3 5 4 2 3 1 6 8 3 5 ) #i need the output like this by removing the duplicates 1 2 3 5 4 6 8 #i dont need sorting in my program #plz explain me as simple using for loop #os-ubuntu ,shell=bash

LEARN ABOUT REDHAT

console.perms

console.perms(5)					   System Administrator's Manual					  console.perms(5)

NAME

       console.perms - permissions control file for users at the system console

DESCRIPTION

       /etc/security/console.perms  determines	the permissions that will be given to priviledged users of the console at login time, and the per-
       missions to which to revert when the users log out.  It is read by the pam_console module.

       The format is:

       <class>=space-separated list of words

       login-regexp|<login-class> perm dev-glob|<dev-class> 
	       revert-mode revert-owner[.revert-group]

       The revert-mode, revert-owner, and revert-group fields are optional, and default to 0600, root, and root, respectively.

       The words in a class definition are evaluated as globs if they refer to files, but as regular expressions if they apply to a console  defi-
       nition.	Do not mix them.

       Any line can be broken and continued on the next line by using a  character as the last character on the line.

       The login-class class and the login-regexp word are evaluated as regular expressions.  The dev-class and the dev-glob word are evaluated as
       shell-style globs.  If a name given corresponds to a directory, and if it is a mount point listed in /etc/fstab, the device node associated
       with the filesystem mounted at that point will be substituted in its place.

       Classes	are  denoted  by  being contained in < angle bracket > characters; a lack of < angle brackets > indicates that the string is to be
       taken literally as a login-regexp or a dev-glob, depending on its input position.

SEE ALSO

       pam_console(8)
       pam_console_apply(8)
       console.apps(5)

AUTHOR

       Michael K. Johnson <johnsonm@redhat.com>

Red Hat Software						     1999/2/3							  console.perms(5)