Sponsored Content
Top Forums Shell Programming and Scripting Deleting Duplicates leaving the first entry Post 302497274 by Samingla on Wednesday 16th of February 2011 05:58:36 PM
Old 02-16-2011
Deleting Duplicates leaving the first entry

Hi,

I need to delete duplicate records in a file that is around 30MB. Below is what I need. Below are the entries of input file and the output file that I need. Each section of input file is separated by an empty line. Some of these sections have duplicate uid values. I want to retain only one UID per section. I do not want to make any changes to other entries though they are repeated. I tried some options but the values of d and e which are same in other sections are getting deleted. I have also attached a file with actual entries in it.

Thanks in advance
sam




Input file entries:

Code:
a: t1.com
uid: t1
b: t2
c: t3
uid: t1
d: t3
e: t4
 
a: u2.com
uid: u2
b: u2
c: t3
uid: u2
d: t3
e: t4
 
a: v2.com
b: v3
c: t3
uid: v3
d: t3
e: t4
 
Expected OutPut:
a: t1.com
uid: t1
b: t2
c: t3
d: t3
e: t4
 
a: u2.com
uid: u2
b: u2
c: t3
d: t3
e: t4
 
a: v2.com
uid: v3
b: v3
c: t3
d: t3
e: t4

Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!

Last edited by vgersh99; 02-16-2011 at 07:28 PM.. Reason: code tags, please!
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Deleting double entry in a file

Hi, I am having almost the same problem as junior member 'oupsforum' (refer to subjuct "deleting double entry in a log file"), only that I am using Sun Sorlaris Unix which the uniq command does not has the flag -w. So I am not able to ignore certain portion of the line when the uniq doing the... (3 Replies)
Discussion started by: Wing m. Cheng
3 Replies

2. Shell Programming and Scripting

Deleting file entry

Hello everyone, I want to compare the first line of a file(ABC) with that of a folder,XYZ(folder contents) and want that line to be deleted from the file(ABC) if that entry doesn't exist in the folder(XYZ) I want to put this in a loop. please can anyone help thanks (6 Replies)
Discussion started by: swasid
6 Replies

3. Red Hat

Adding or deleting an entry in /etc/inittab without using vi editrors or any editor.

Hi masters Is there any way to edit or delete an entry in inittab file without using vi or any editors? We can use commands instead or any shell script .. If any one can help deeply appreciated Thanks a lot sai (3 Replies)
Discussion started by: saidiya
3 Replies

4. Shell Programming and Scripting

Remove copies/duplicates of same entry

Hi, I have a very big (with around 1 million entries) txt file with IPv4 addresses in the standard format, i.e. a.b.c.d The file looks like 10.1.1.1 10.1.1.1 10.1.1.1 10.1.2.4 10.1.2.4 12.1.5.6 . . . . and so on.... There are duplicate/multiple entries for some IP... (1 Reply)
Discussion started by: sajal.bhatia
1 Replies

5. Shell Programming and Scripting

Searching for an entry and deleting the line

Hi Im trying to scan a file for certain entries and remove their corresponding lines completely. What I have now is this, for USER in user1 user2 user3 user4 do sed '/$USER/d' /etc/sudoers done However this doesn't remove the entries at all. Is there another way for this? Thanks... (2 Replies)
Discussion started by: bludhemn
2 Replies

6. Shell Programming and Scripting

Deleting duplicate glosses in a dictionary entry

I am working on an Urdu to Hindi dictionary and I have created the following file structure: Headword=Gloss1,Gloss2,Gloss3 i.e. glosses delimited by a comma. It so happens that in some cases (around 6000+ in a file of over 200,000+ the glosses are duplicated. Since this may be a... (3 Replies)
Discussion started by: gimley
3 Replies

7. Shell Programming and Scripting

Deleting only 2nd and third duplicates in field 2

(7 Replies)
Discussion started by: newbie2010
7 Replies

8. How to Post in the The UNIX and Linux Forums

Help me, write a bash script to delete parent entry with all their child entry in LDAP UNIX server

Hi All, Please help me and guide me to write a bash/shell script on Linux box to delete parent entry with all their child entries. example: Parent is : ---------- dn: email=yogesh.kumar@wipro.com, o=wipro, o=in child is: ---------- dn: cn: yogesh kumar, email=yogesh.kumar@wipro.com,... (1 Reply)
Discussion started by: Chand
1 Replies

9. Shell Programming and Scripting

How to replace multiple "&nbsp;" entry with in <td> tag into single entry using sed?

I have the input file like this. Input file: 12.txt 1) There are one or more than one <tr> tags in same line. 2) Some tr tags may have one <td> or more tna one <td> tags within it. 3) Few <td> tags having "<td> &nbsp; </td>". Few having more than one "&nbsp;" entry in it. <tr> some td... (4 Replies)
Discussion started by: thomasraj87
4 Replies
PTS_DELETE(1)						       AFS Command Reference						     PTS_DELETE(1)

NAME
pts_delete - Deletes a Protection Database entry SYNOPSIS
pts delete -nameorid <user or group name or id>+ [-cell <cell name>] [-noauth] [-localauth] [-force] [-help] pts d -na <user or group name or id>+ [-c <cell name>] [-no] [-l] [-f] [-h] DESCRIPTION
The pts delete command removes each entry specified by the -nameorid argument from the Protection Database. Deleting entries affects other parts of the system in various ways: o Deleted users and groups still appear on access control lists (ACLs), but are listed by AFS UID or GID rather than by name, because there is no longer an associated name to which to translate the ID. To remove these obsolete entries from ACLs, use the fs cleanacl command. o Deleting a user or machine's entry removes it from the membership list of any group to which it belonged. o Deleting a group entry removes it from the membership list of any user or machine entry that belonged to the group, and also increments the group-creation quota of the group's creator by one, even if the creator no longer owns the group. To remove a user or machine from a group without actually deleting the entry, use the pts removeuser command. OPTIONS
-nameorid <user or group name or ID>+ Specifies the name or AFS UID of each user, the name or AFS GID of each group, or the IP address (complete or wildcard-style) or AFS UID of each machine entry to delete. It is acceptable to mix users, machines, and groups on the same command line, as well as names (IP addresses for machines) and IDs. Precede the GID of each group with a hyphen to indicate that it is negative. -cell <cell name> Names the cell in which to run the command. For more details, see pts(1). -force Enables the command to continue executing as far as possible when errors or other problems occur, rather than halting execution at the first error. -help Prints the online help for this command. All other valid options are ignored. -localauth Constructs a server ticket using a key from the local /etc/openafs/server/KeyFile file. Do not combine this flag with the -cell or -noauth options. For more details, see pts(1). -noauth Assigns the unprivileged identity anonymous to the issuer. For more details, see pts(1). EXAMPLES
The following example deletes the user entries "pat" and "terry": % pts delete pat terry The following example deletes the Protection Database entry of the group with AFS GID -215. % pts delete -215 PRIVILEGE REQUIRED
The issuer must belong to the system:administrators group to delete user and machine entries. To delete group entries, the issuer must either own the group or belong to the system:administrators group. SEE ALSO
fs_cleanacl(1), pts(1), pts_removeuser(1) COPYRIGHT
IBM Corporation 2000. <http://www.ibm.com/> All Rights Reserved. This documentation is covered by the IBM Public License Version 1.0. It was converted from HTML to POD by software written by Chas Williams and Russ Allbery, based on work by Alf Wachsmann and Elizabeth Cassell. OpenAFS 2014-04-08 PTS_DELETE(1)
All times are GMT -4. The time now is 03:47 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy