Sponsored Content
Top Forums Shell Programming and Scripting Removal of Duplicate Entries from the file Post 302254248 by ravi_rn on Tuesday 4th of November 2008 01:54:39 AM
Old 11-04-2008
Java Removal of Duplicate Entries from the file

I have a file which consists of 1000 entries. Out of 1000 entries i have 500 Duplicate Entires. I want to remove the first Duplicate Entry (i,e entire Line) in the File.

The example of the File is shown below:
8244100010143276|MARISOL CARO||MORALES|HSD768|CARR 430 KM 1.7
8244100010143276|MARISOL CARO||MORALES|New512|CARR 430 KM 1.7
8244100010196084|CARMEN L||VELEZ|Internet128|BO
8244100010196084|CARMEN L||VELEZ|Internet128|BO

from the above example i have to remove first Duplicate Entry That is 8244100010143276 and 8244100010196084.

Please help me in resolving the above issue.

Thanks in advance
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Check host file for duplicate entries

I need a KSH script that will check a host file for duplicate IP's and/or host names and report out the errors. Anyone out there have one they would like to share? Something like: Hostname blahblah appears X times IP Address xxx.xxx.xxx.xxx appears X times TIA (4 Replies)
Discussion started by: ThreeDot
4 Replies

2. UNIX for Dummies Questions & Answers

Sort, duplicate removal - Query

Hi All, I have a problem with the sort and duplicate filter command I am using in one of my scripts. I have a '|' delimited file and want to sort and remove duplicates on the 1,2,15 fields. These fields constitute the primary key of the table I will be loading the data into. But I see that some... (4 Replies)
Discussion started by: novice1324
4 Replies

3. Shell Programming and Scripting

duplicate entries /.rhosts file

Hi All, I have one problem related to /.rhosts file. According to my understanding, /.rhosts file is used for "rsh". What will happen if I have duplicate entries in this file? e.g> my .rhosts file looks like Code: wcars42g wcars89j wcars42g wcars42b wcars42b Will duplicate entries... (1 Reply)
Discussion started by: akash_mahakode
1 Replies

4. Shell Programming and Scripting

duplicate entries /.rhosts file

Hi, I forgot how to start a new thread. :( Can somebody please guide me? I have one problem related to /.rhosts file. According to my understanding, /.rhosts file is used for "rsh". What will happen if I have duplicate entries in this file? e.g> my .rhosts file looks like wcars42g... (2 Replies)
Discussion started by: akash_mahakode
2 Replies

5. Shell Programming and Scripting

Counting duplicate entries in a file using awk

Hi, I have a very big (with around 1 million entries) txt file with IPv4 addresses in the standard format, i.e. a.b.c.d The file looks like 10.1.1.1 10.1.1.1 10.1.1.1 10.1.2.4 10.1.2.4 12.1.5.6 . . . . and so on.... There are duplicate/multiple entries for some IP... (3 Replies)
Discussion started by: sajal.bhatia
3 Replies

6. Shell Programming and Scripting

Duplicate line removal matching some columns only

I'm looking to remove duplicate rows from a CSV file with a twist. The first row is a header. There are 31 columns. I want to remove duplicates when the first 29 rows are identical ignoring row 30 and 31 BUT the duplicate that is kept should have the shortest total character length in rows 30... (6 Replies)
Discussion started by: Michael Stora
6 Replies

7. UNIX for Advanced & Expert Users

Duplicate removal

I have an input file of 5GB which contains duplicate records and have to remove duplicate records by retaing first instance of that record . Based on 5 fields the duplicates has to be removed . Kindly request to help me in writing a Unix Script. Thanks Asim (11 Replies)
Discussion started by: duplicate
11 Replies

8. Shell Programming and Scripting

String removal from file

Dear all From below mention input file I needed op file as show below. I am using below code but not worked. I/p file BSCBCH1 EXAL-1-4 WO* SMPS MAINS FAIL BSCBCH1 EXAL-1-5 WO* SMPS RECTIFIER FAIL BSCBCH1 EXAL-1-6 WO* SMPS MAJOR ALARM BSCBCH2 EXAL-1-10 WO* ... (5 Replies)
Discussion started by: jaydeep_sadaria
5 Replies

9. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

10. Shell Programming and Scripting

How to check duplicate entries in file ? (Solaris-9)

Hi, There are duplicate entries in file, but uniq will not see because first field is different. How will I catch all lines, which are having duplicate IPs ? bash-2.05# cat db.file | grep 172.30.133.11 dsrq-ctrl1-prod A 172.30.133.11 e911q-db1-nxge0 A 172.30.133.11... (4 Replies)
Discussion started by: solaris_1977
4 Replies
ipa-managed-entries(1)						 IPA Manual Pages					    ipa-managed-entries(1)

NAME
ipa-managed-entries - Enables or disables the schema Managed Entry plugins SYNOPSIS
ipa-managed-entries [options] <enable|disable|status> DESCRIPTION
Run the command with the enable option to enable the Managed Entry plugin. Run the command with the disable option to disable the Managed Entry plugin. Run the command with the status to determine the current status of the Managed Entry plugin. In all cases the user will be prompted to provide the Directory Manager's password unless option -p is used. Directory Server will need to be restarted after the Managed Entry plugin has been enabled. OPTIONS
-h, --help Show a help message and exit -d, --debug Enable debug logging when more verbose output is needed -e, --entry DN for the Managed Entry Definition -l, --list List available Managed Entries -p DM_PASSWORD, --password=DM_PASSWORD The Directory Manager password to use for authentication EXIT STATUS
0 if the command was successful 1 if an error occurred 2 if the plugin is already in the required status (enabled or disabled) IPA
Feb 06 2012 ipa-managed-entries(1)
All times are GMT -4. The time now is 06:13 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy