Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Help with remove duplicate content
Top Forums Shell Programming and Scripting Help with remove duplicate content Post 302573508 by perl_beginner on Monday 14th of November 2011 07:29:39 PM
Old 11-14-2011
Help with remove duplicate content
Input file
Code:
data_1 10 US
data_1 2 US
data_1 5 UK
data_2 20 ENGLAND
data_2 12 KOREA
data_3 4 CHINA
.
.
data_60 123 US
data_60 23 UK
data_60 45 US

Desired output file
Code:
data_1 10 US
data_1 5 UK
data_2 20 ENGLAND
data_2 12 KOREA
data_3 4 CHINA
.
.
data_60 123 US
data_60 23 UK

If data content in column 1 and column 3 are exactly the same. Then print out only the first record.
Thanks for any advice.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help... (4 Replies)
Discussion started by: kazanoova2
4 Replies

2. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies

3. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

4. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

5. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

6. Shell Programming and Scripting

Help with replace duplicate content

Input file: CCNI data564_input1 264 CORO1A data564_input2 155 ABC-B data17_input1 3466 ABC-B data17_input2 1133 ABC-B data17_input3 2162 ABC-B data17_input4 2019 HNRNPA2B1 data95_input1 101 HNRNPA2B1 data95_input2 340 IFITM1 data105_input2 291 IFITM2 data105_input1 505... (3 Replies)
Discussion started by: cpp_beginner
3 Replies

7. Shell Programming and Scripting

Help with duplicate data content problem asking

Input file: A_69510335_ASD>aw 1199470 USA A_119571157_C>awe,QWEQE 113932840 USA C_34646666_qwe>TAWTT,G,TT 112736796 UK C_69510335_QW>T 1199470 USA D_70520237_WR>QEE,G 34459863 UK D_71380003_QWR>T 145418226 IK . Desired output: A_69510335_ASD>aw 1199470 USA... (1 Reply)
Discussion started by: perl_beginner
1 Replies

8. Shell Programming and Scripting

Help with duplicate common data content

Input file: #data_131 0 >content..._* 1 >content..._at_+/97.20% #data_137 0 >content..._* 1 >content..._at_+/97.20% 2 >seq..._* 3 >content..._at_+/97.20% 4 >content..._at_+/97.20% #data_141 0 >content..._* #data_150 0 >content..._* 1 >content..._at_+/97.20% 2 >seq..._* 3... (3 Replies)
Discussion started by: perl_beginner
3 Replies

9. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies

10. Shell Programming and Scripting

Remove the duplicate content in a file

Here is the contents of test.txt Dependencies Resolved Changes in packages about to be updated: ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64, - Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code) Dependencies Resolved Changes in packages about to be updated: ... (5 Replies)
Discussion started by: ashokvpp
5 Replies

LEARN ABOUT PHP

gendergender

GENDERGENDER(3) 							 1							   GENDERGENDER(3)

The GenderGender class

INTRODUCTION

CLASS SYNOPSIS

       GenderGender

       GenderGender

       Constants

       o const integer$GenderGender::IS_FEMALE70

       o const integer$GenderGender::IS_MOSTLY_FEMALE102

       o const integer$GenderGender::IS_MALE77

       o const integer$GenderGender::IS_MOSTLY_MALE109

       o const integer$GenderGender::IS_UNISEX_NAME63

       o const integer$GenderGender::IS_A_COUPLE67

       o const integer$GenderGender::NAME_NOT_FOUND32

       o const integer$GenderGender::ERROR_IN_NAME69

       o const integer$GenderGender::ANY_COUNTRY0

       o const integer$GenderGender::BRITAIN1

       o const integer$GenderGender::IRELAND2

       o const integer$GenderGender::USA3

       o const integer$GenderGender::SPAIN4

       o const integer$GenderGender::PORTUGAL5

       o const integer$GenderGender::ITALY6

       o const integer$GenderGender::MALTA7

       o const integer$GenderGender::FRANCE8

       o const integer$GenderGender::BELGIUM9

       o const integer$GenderGender::LUXEMBOURG10

       o const integer$GenderGender::NETHERLANDS11

       o const integer$GenderGender::GERMANY12

       o const integer$GenderGender::EAST_FRISIA13

       o const integer$GenderGender::AUSTRIA14

       o const integer$GenderGender::SWISS15

       o const integer$GenderGender::ICELAND16

       o const integer$GenderGender::DENMARK17

       o const integer$GenderGender::NORWAY18

       o const integer$GenderGender::SWEDEN19

       o const integer$GenderGender::FINLAND20

       o const integer$GenderGender::ESTONIA21

       o const integer$GenderGender::LATVIA22

       o const integer$GenderGender::LITHUANIA23

       o const integer$GenderGender::POLAND24

       o const integer$GenderGender::CZECH_REP25

       o const integer$GenderGender::SLOVAKIA26

       o const integer$GenderGender::HUNGARY27

       o const integer$GenderGender::ROMANIA28

       o const integer$GenderGender::BULGARIA29

       o const integer$GenderGender::BOSNIA30

       o const integer$GenderGender::CROATIA31

       o const integer$GenderGender::KOSOVO32

       o const integer$GenderGender::MACEDONIA33

       o const integer$GenderGender::MONTENEGRO34

       o const integer$GenderGender::SERBIA35

       o const integer$GenderGender::SLOVENIA36

       o const integer$GenderGender::ALBANIA37

       o const integer$GenderGender::GREECE38

       o const integer$GenderGender::RUSSIA39

       o const integer$GenderGender::BELARUS40

       o const integer$GenderGender::MOLDOVA41

       o const integer$GenderGender::UKRAINE42

       o const integer$GenderGender::ARMENIA43

       o const integer$GenderGender::AZERBAIJAN44

       o const integer$GenderGender::GEORGIA45

       o const integer$GenderGender::KAZAKH_UZBEK46

       o const integer$GenderGender::TURKEY47

       o const integer$GenderGender::ARABIA48

       o const integer$GenderGender::ISRAEL49

       o const integer$GenderGender::CHINA50

       o const integer$GenderGender::INDIA51

       o const integer$GenderGender::JAPAN52

       o const integer$GenderGender::KOREA53

       Methods

       o public boolean GenderGender::connect (string	$dsn)

       o public GenderGender::__construct ([string  $dsn])

       o public array GenderGender::country (integer  $country)

       o public integer GenderGender::get (string  $name, [integer  $country])

       o public array GenderGender::isNick (string  $name0, string  $name1, [integer  $country])

       o public array GenderGender::similarNames (string  $name, [integer  $country])

PREDEFINED CONSTANTS

	      o GenderGender::IS_FEMALE -

	      o GenderGender::IS_MOSTLY_FEMALE -

	      o GenderGender::IS_MALE -

	      o GenderGender::IS_MOSTLY_MALE -

	      o GenderGender::IS_UNISEX_NAME -

	      o GenderGender::IS_A_COUPLE -

	      o GenderGender::NAME_NOT_FOUND -

	      o GenderGender::ERROR_IN_NAME -

	      o GenderGender::ANY_COUNTRY -

	      o GenderGender::BRITAIN -

	      o GenderGender::IRELAND -

	      o GenderGender::USA -

	      o GenderGender::SPAIN -

	      o GenderGender::PORTUGAL -

	      o GenderGender::ITALY -

	      o GenderGender::MALTA -

	      o GenderGender::FRANCE -

	      o GenderGender::BELGIUM -

	      o GenderGender::LUXEMBOURG -

	      o GenderGender::NETHERLANDS -

	      o GenderGender::GERMANY -

	      o GenderGender::EAST_FRISIA -

	      o GenderGender::AUSTRIA -

	      o GenderGender::SWISS -

	      o GenderGender::ICELAND -

	      o GenderGender::DENMARK -

	      o GenderGender::NORWAY -

	      o GenderGender::SWEDEN -

	      o GenderGender::FINLAND -

	      o GenderGender::ESTONIA -

	      o GenderGender::LATVIA -

	      o GenderGender::LITHUANIA -

	      o GenderGender::POLAND -

	      o GenderGender::CZECH_REP -

	      o GenderGender::SLOVAKIA -

	      o GenderGender::HUNGARY -

	      o GenderGender::ROMANIA -

	      o GenderGender::BULGARIA -

	      o GenderGender::BOSNIA -

	      o GenderGender::CROATIA -

	      o GenderGender::KOSOVO -

	      o GenderGender::MACEDONIA -

	      o GenderGender::MONTENEGRO -

	      o GenderGender::SERBIA -

	      o GenderGender::SLOVENIA -

	      o GenderGender::ALBANIA -

	      o GenderGender::GREECE -

	      o GenderGender::RUSSIA -

	      o GenderGender::BELARUS -

	      o GenderGender::MOLDOVA -

	      o GenderGender::UKRAINE -

	      o GenderGender::ARMENIA -

	      o GenderGender::AZERBAIJAN -

	      o GenderGender::GEORGIA -

	      o GenderGender::KAZAKH_UZBEK -

	      o GenderGender::TURKEY -

	      o GenderGender::ARABIA -

	      o GenderGender::ISRAEL -

	      o GenderGender::CHINA -

	      o GenderGender::INDIA -

	      o GenderGender::JAPAN -

	      o GenderGender::KOREA -

PHP Documentation Group 													   GENDERGENDER(3)
All times are GMT -4. The time now is 05:18 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy