Sponsored Content
Top Forums Shell Programming and Scripting fastest way to remove duplicates. Post 76043 by amit_sapre on Friday 24th of June 2005 08:19:06 AM
Old 06-24-2005
Hi Vino,

This command will keep the first entry as it is and delete the other entries,

irrespective of whether the file is sorted or not.

No prior assumptions while executing this command.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to delete/remove directory in fastest way

hello i need help to remove directory . The directory is not empty ., it contains several sub directories and files inside that.. total number of files in one directory is 12,24,446 . rm -rf doesnt work . it is prompting for every file .. i want to delete without prompting and... (6 Replies)
Discussion started by: getdpg
6 Replies

2. UNIX for Dummies Questions & Answers

How to remove duplicates without sorting

Hello, I can remove duplicate entries in a file by: sort File1 | uniq > File2 but how can I remove duplicates without sorting the file? I tried cat File1 | uniq > File2 but it doesn't work thanks (4 Replies)
Discussion started by: orahi001
4 Replies

3. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

4. Shell Programming and Scripting

Remove duplicates from a file

Hi, I need to remove duplicates from a file. The file will be like this 0003 10101 20100120 abcdefghi 0003 10101 20100121 abcdefghi 0003 10101 20100122 abcdefghi 0003 10102 20100120 abcdefghi 0003 10103 20100120 abcdefghi 0003 10103 20100121 abcdefghi Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies

5. Shell Programming and Scripting

Script to remove duplicates

Hi I need a script that removes the duplicate records and write it to a new file for example I have a file named test.txt and it looks like abcd.23 abcd.24 abcd.25 qwer.25 qwer.26 qwer.98 I want to pick only $1 and compare with the next record and the output should be abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies

6. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

7. Shell Programming and Scripting

Fastest way to delete duplicates from a large filelist.....

OK I have two filelists...... The first is formatted like this.... /path/to/the/actual/file/location/filename.jpg and has up to a million records The second list shows filename.jpg where there is more then on instance. and has maybe up to 65,000 records I want to copy files... (4 Replies)
Discussion started by: Bashingaway
4 Replies

8. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

9. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

10. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies
PTS_DELETE(1)						       AFS Command Reference						     PTS_DELETE(1)

NAME
pts_delete - Deletes a Protection Database entry SYNOPSIS
pts delete -nameorid <user or group name or id>+ [-cell <cell name>] [-noauth] [-localauth] [-force] [-help] pts d -na <user or group name or id>+ [-c <cell name>] [-no] [-l] [-f] [-h] DESCRIPTION
The pts delete command removes each entry specified by the -nameorid argument from the Protection Database. Deleting entries affects other parts of the system in various ways: o Deleted users and groups still appear on access control lists (ACLs), but are listed by AFS UID or GID rather than by name, because there is no longer an associated name to which to translate the ID. To remove these obsolete entries from ACLs, use the fs cleanacl command. o Deleting a user or machine's entry removes it from the membership list of any group to which it belonged. o Deleting a group entry removes it from the membership list of any user or machine entry that belonged to the group, and also increments the group-creation quota of the group's creator by one, even if the creator no longer owns the group. To remove a user or machine from a group without actually deleting the entry, use the pts removeuser command. OPTIONS
-nameorid <user or group name or ID>+ Specifies the name or AFS UID of each user, the name or AFS GID of each group, or the IP address (complete or wildcard-style) or AFS UID of each machine entry to delete. It is acceptable to mix users, machines, and groups on the same command line, as well as names (IP addresses for machines) and IDs. Precede the GID of each group with a hyphen to indicate that it is negative. -cell <cell name> Names the cell in which to run the command. For more details, see pts(1). -force Enables the command to continue executing as far as possible when errors or other problems occur, rather than halting execution at the first error. -help Prints the online help for this command. All other valid options are ignored. -localauth Constructs a server ticket using a key from the local /etc/openafs/server/KeyFile file. Do not combine this flag with the -cell or -noauth options. For more details, see pts(1). -noauth Assigns the unprivileged identity anonymous to the issuer. For more details, see pts(1). EXAMPLES
The following example deletes the user entries "pat" and "terry": % pts delete pat terry The following example deletes the Protection Database entry of the group with AFS GID -215. % pts delete -215 PRIVILEGE REQUIRED
The issuer must belong to the system:administrators group to delete user and machine entries. To delete group entries, the issuer must either own the group or belong to the system:administrators group. SEE ALSO
fs_cleanacl(1), pts(1), pts_removeuser(1) COPYRIGHT
IBM Corporation 2000. <http://www.ibm.com/> All Rights Reserved. This documentation is covered by the IBM Public License Version 1.0. It was converted from HTML to POD by software written by Chas Williams and Russ Allbery, based on work by Alf Wachsmann and Elizabeth Cassell. OpenAFS 2014-04-08 PTS_DELETE(1)
All times are GMT -4. The time now is 09:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy