Remove not only the duplicate string but also the keyword of the string in Perl


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove not only the duplicate string but also the keyword of the string in Perl
# 1  
Old 03-12-2014
Remove not only the duplicate string but also the keyword of the string in Perl

Hi Perl users,

I have another problem with text processing in Perl. I have a file below:
Code:
Linux   Unix   Linux  Windows SUN
MACOS  SUN  SUN HP-AUX

I want the result below:
Code:
Unix     Windows   SUN
MACOS   HP-AUX

so the duplicate string will be removed and also the keyword of the string on that line.

I have a code below so far but the problem hasn't solved yet.
Code:
#!/usr/bin/perl

use strict;
use warnings;

my $replacement = "";
my (@temp, @data, %result_of_hash) = ();

while (<>)
{  
  chomp;
  @temp = sort split " ", $_;

  my $prev_line = undef;
  for my $i (0 .. (scalar @temp -1))
  {
    if ($temp[$i] eq $prev_line)
    {
       push @data, "$temp[$i]";
    }
  
  $prev_line = $temp[$i];
  }

  $store{$line} = 1;  
}


foreach my $key_store ( keys %store )
{
  for my $k (@data)
  {
         my $pos = index($key_store,$k);
        if ($pos > -1)
    { 
       my $result = substr ($key_store, $pos, length($k),$replacement);
       $result_of_hash{$key_store} = 1;
    }
  }    
}

print "$_\n" for keys %result_of_hash;
close (FH);

# 2  
Old 03-12-2014
Hi,
you can use this way:
Code:
$ cat in.txt
Linux   Unix   Linux  Windows SUN
MACOS  SUN  SUN HP-AUX
$ perl -sane 'foreach $i (@F) { $X{$i}+=1 } ; foreach $i (@F) {printf "%s ",$i if $X{$i}==1 };print "\n";%X=""' in.txt
Unix Windows SUN
MACOS HP-AUX

Regards.

Last edited by disedorgue; 03-12-2014 at 05:10 PM..
This User Gave Thanks to disedorgue For This Post:
# 3  
Old 03-12-2014
Hi disedorgue,

Thanks for the script, it works for one-liners but why the script is not work when I try to modify the script as below.
Code:
#!/usr/bin/perl -w
use strict;

my $file = "in.txt";
my %X = ();

open (FH, '<', $file) or die $!;

my @F = <FH>;

foreach my $i (@F) 
{ 
  $X{$i}+=1; 
}  

foreach my $i (@F) 
{
  
  if ($X{$i}==1)
  {
    printf "%s ","$i";
  } 
}
print "\n";
%X= ();

close (FH);

---------- Post updated at 08:09 AM ---------- Previous update was at 06:36 AM ----------

Hi,
it works with small modification. Thanks

Code:
#!/usr/bin/perl -w
use strict;

my %X = ();
open (FH, '<', "in.txt") or die $!;

my @F = <FH>;
my @temp;

foreach my $line (@F) 
{ 
  chomp $line;
  @temp = split " ", $line;  
  for my $i ( 0 .. $#temp)
  {
    $X{$temp[0]}{$temp[$i]}++;    
  }
} 

foreach my $key(keys %X)
{ 
  foreach my $subkey(keys %{$X{$key}})
  {
     print "$subkey " if $X{$key}{$subkey} == 1;
  }
  print "\n";
}

close (FH);


Last edited by askari; 03-12-2014 at 09:28 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate consecutive lines with specific string

Hello, I'm trying to remove the duplicate consecutive lines with specific string "WARNING". File.txt abc; WARNING 2345 WARNING 2345 WARNING 2345 WARNING 2345 WARNING 2345 bcd; abc; 123 123 123 WARNING 1234 WARNING 2345 WARNING 2345 efgh; (6 Replies)
Discussion started by: Mannu2525
6 Replies

2. Shell Programming and Scripting

Remove string perl with first or last word is in a list

Hello, I try to delete all strings if their first or last word is one of this list of words : "the", "i", "in", "there", "this", "with", "on", "we", "that", "of" For example if i have this string in an input file "with me" this string will be removed, Example: input "the european... (2 Replies)
Discussion started by: cyrine
2 Replies

3. Shell Programming and Scripting

Remove multiple lines from a particular string to particular string

Hi, I have a file containing the DDLs of tables in a schema. From that I need to remove all the lines from a starting string till a specific string. Here is an example. File1.txt ------------- CREATE TABLE "SCHEMA1"."LKP11_TBL_USERS" ( "ID" NUMBER(8,0) NOT NULL ENABLE, "USER_ID"... (3 Replies)
Discussion started by: satyaatcgi
3 Replies

4. Shell Programming and Scripting

Remove lines between the start string and end string including start and end string Python

Hi, I am trying to remove lines once a string is found till another string is found including the start string and end string. I want to basically grab all the lines starting with color (closing bracket). PS: The line after the closing bracket for color could be anything (currently 'more').... (1 Reply)
Discussion started by: Dabheeruz
1 Replies

5. Shell Programming and Scripting

Remove duplicate chars and sort string [SED]

Hi, INPUT: DCBADD OUTPUT: ABCD The SED script should alphabetically sort the chars in the string and remove the duplicate chars. (5 Replies)
Discussion started by: jds93
5 Replies

6. Shell Programming and Scripting

remove characters from string based on occurrence of a string

Hello Folks.. I need your help .. here the example of my problem..i know its easy..i don't all the commands in unix to do this especiallly sed...here my string.. dwc2_dfg_ajja_dfhhj_vw_dec2_dfgh_dwq desired output is.. dwc2_dfg_ajja_dfhhj it's a simple task with tail... (5 Replies)
Discussion started by: victor369
5 Replies

7. Shell Programming and Scripting

filtering out duplicate substrings, regex string from a string

My input contains a single word lines. From each line data.txt prjtestBlaBlatestBlaBla prjthisBlaBlathisBlaBla prjthatBlaBladpthatBlaBla prjgoodBlaBladpgoodBlaBla prjgood1BlaBla123dpgood1BlaBla123 Desired output --> data_out.txt prjtestBlaBla prjthisBlaBla... (8 Replies)
Discussion started by: kchinnam
8 Replies

8. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

9. Shell Programming and Scripting

How to remove duplicate sentence/string in perl?

Hi, I have two strings like this in an array: For example: @a=("Brain aging is associated with a progressive imbalance between intracellular concentration of Reactive Oxygen Species","Brain aging is associated with a progressive imbalance between intracellular concentration of Reactive... (9 Replies)
Discussion started by: vanitham
9 Replies

10. Shell Programming and Scripting

Match keyword on string and substitute under vi

Hi guys, with sed when I need to make a substitution inside a line containing a specific keyword, I usually use: sed '/keyword/ s/cat/dog/g' This will substitute "cat" with "dog" on those lines containing "keyword". Now I want to use this inside vi, for several reason that I cannot... (2 Replies)
Discussion started by: lycaon
2 Replies
Login or Register to Ask a Question