Help in modifying existing Perl Script to produce report of dupes


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help in modifying existing Perl Script to produce report of dupes
# 1  
Old 04-25-2012
Help in modifying existing Perl Script to produce report of dupes

Hello,
I have a large amount of data with the following structure:
Word=Transliterated word
I have written a Perl Script (reproduced below) which goes through the full file and identifies all dupes on the right hand side. It creates successfully a new file with two headers: Singletons and Dupes.
I have tried to modify the script to produce additionally a record listing the frequency count of all dupes. Thus in the sample provided, I would like to know how many times the dupe Albert has been transliterated in different ways. I am providing pseudo-data since the original data is in a foreign script.
Quote:
Albert=albt
Albert=albut
Albert=albat
Mary=mari
Mary=meri
Mary=merry
Mary=marey
The script should give me a report in a separate output with the following structure:
Quote:
Albert,3, albt,albut,albat
Mary,4,mari,meri,merry,marey
The final output would thus have two files:
The output file listing Singletons and Dupes
The report which would have the dupes listed along with their frequency.
I am not very good at generating reports in Perl and hence the request:
Perl script follows.
Many thanks for excellent help and advice given.
Code:
#!/usr/bin/perl

$dupes = $singletons = "";		# This goes at the head of the file

do {
    $dupefound = 0;			# These go at the head of the loop
    $text = $line = $prevline = $name = $prevname = "";
    do {
	$line = <>;
	$line =~ /^(.+)\=.+$/ and $name = $1;
	$prevline =~ /^(.+)\=.+$/ and $prevname = $1;
	if ($name eq $prevname) { $dupefound += 1 }
	$text .= $line;
	$prevline = $line;
    } until ($dupefound > 0 and $text !~ /^(.+?)\=.*?\n(?:\1=.*?\n)+\z/m) or eof;
    if ($text =~ s/(^(.+?)\=.*?\n(?:\2=.*?\n)+)//m) { $dupes .= $1 }
    $singletons .= $text;
} until eof;
print "SINGLETONS\n$singletons\n\DUPES\n$dupes";


Last edited by Franklin52; 04-26-2012 at 03:48 AM.. Reason: Corrected code tags
# 2  
Old 04-26-2012
Code:
[user@cygwin ~]$ cat input.txt
Albert=albt
Albert=albut
Albert=albat
Mary=mari
Mary=meri
Mary=merry
Mary=marey
[user@cygwin ~]$
[user@cygwin ~]$ perl -F= -ane 'BEGIN {open O, "> output.txt"}
chomp $F[1]; $x{$F[0]} .= "$F[1],"; $y{$F[0]}++;
END {
    for (sort keys %x) {
        $x{$_} =~ s/,$//;
        print O "$_,$y{$_},$x{$_}\n";
    }
    close O;
}' input.txt
[user@cygwin ~]$
[user@cygwin ~]$ cat output.txt
Albert,3,albt,albut,albat
Mary,4,mari,meri,merry,marey
[user@cygwin ~]$

This User Gave Thanks to balajesuri For This Post:
# 3  
Old 04-26-2012
Hi,
Many thanks.
Unluckily I work under windows and "cat" commands do not function correctly under this OS.
I cut the snippet of the code and applied it but it would not work.
Many thanks for the help all the same
# 4  
Old 04-26-2012
Code:
#! C:\Perl\bin\perl.exe
use strict;
use warnings;

my (@F, %x, %y);

open I, "< input.txt";
for (<I>) {
    chomp;
    @F = split /=/;
    $x{$F[0]} .= "$F[1],";
    $y{$F[0]}++;
}
close I;

open O, "> output.txt";
for (sort keys %x) {
    $x{$_} =~ s/,$//;
    print O "$_,$y{$_},$x{$_}\n";
}
close O;

# 5  
Old 04-26-2012
How about this ?
Code:
#!/usr/bin/perl

while (<DATA>) {
        chomp;
        ($word,$meaning)=split(/\=/);
        push  @{$word} , $meaning ;
        $hashWord{$word}=\@{$word};
}

foreach (keys %hashWord ) {
        $KeyWord=$_;
        printf "%s,%d",$KeyWord,scalar @{$hashWord{$_}};
        foreach (@{$hashWord{$_}}) {
        printf ",%s",$_;
        }
        print "\n";
}



__DATA__
Albert=albt
Albert=albut
Albert=albat
Mary=mari
Mary=meri
Mary=merry
Mary=marey

This User Gave Thanks to pravin27 For This Post:
# 6  
Old 04-26-2012
Many thanks. The script runs like a charm and sorts and identifies dupes
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help in modifying a PERL script to sort Singletons and Duplicates

I have a large database which has the following structure a=b where a is one language and b is the other and = is the delimiter Since the data treats of language, homographs occur i.e. the same word on the left hand side can map in two different entries to two different glosses on the right... (3 Replies)
Discussion started by: gimley
3 Replies

2. Shell Programming and Scripting

Help with Perl script for identifying dupes in column1

Dear all, I have a large dictionary database which has the following structure source word=target word e.g. book=livre Since the database is very large in spite of all the care taken, it so happens that at times the source word is repeated e.g. book=livre book=tome Since I want to... (7 Replies)
Discussion started by: gimley
7 Replies

3. Shell Programming and Scripting

Removing Dupes from huge file- awk/perl/uniq

Hi, I have the following command in place nawk -F, '!a++' file > file.uniq It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error: bash-3.2$ nawk -F, '!a++'... (17 Replies)
Discussion started by: makn
17 Replies

4. Shell Programming and Scripting

Script for identifying and deleting dupes in a line

I am compiling a synonym dictionary which has the following structure Headword=Synonym1,Synonym2 and so on, with each synonym separated by a comma. As is usual in such cases manual preparation of synonyms results in repeating the synonym which results in dupes as in the example below:... (3 Replies)
Discussion started by: gimley
3 Replies

5. Shell Programming and Scripting

Script to produce report of High Utilization Processes

Hi, I have requirement to produce a report on high CPU utilization processes and the processes lying on the CPU for long time (Long running queries). The report should append into the files every 3 minutes. I use prstat to pull top 5 and found the following result. ... (3 Replies)
Discussion started by: thinakarmani
3 Replies

6. Shell Programming and Scripting

Using an awk script to identify dupes in two files

Hello, I have two files. File1 or the master file contains two columns separated by a delimiter: a=b b=d e=f g=h File 2 which is the file to be processed has only a single column a h c b What I need is an awk script to identify unique names from file 2 which are not found in the... (6 Replies)
Discussion started by: gimley
6 Replies

7. Shell Programming and Scripting

Shell script that will compare two config files and produce 2 outputs 1)actual config file 2)report

Hi I am new to shell scripting. There is a requirement to write a shell script to meet follwing needs.Prompt reply shall be highly appreciated. script that will compare two config files and produce 2 outputs - actual config file and a report indicating changes made. OS :Susi linux ver 10.3. ... (4 Replies)
Discussion started by: muraliinfy04
4 Replies

8. Infrastructure Monitoring

modifying existing file using C

Hi all, I have a snmpd.conf file as below. in "SECTION: Trap Destinations" line I want to add "trap2dest <IP>:162 <com_str>" on a new line. For this I wrote following code #include <stdio.h> #include <stdlib.h> int main(void) { FILE *fp; ssize_t read_char_count = 0; ... (2 Replies)
Discussion started by: zing_foru
2 Replies

9. Shell Programming and Scripting

modifying perl script

Hi freinds I have a small problem I want u to help me in, I have a syslog server and configured it to send me email automatically, I get a small perl script to help me in, and tested it to send alerts to root and it worked successfully without any problems Now I want to send it outside, I... (4 Replies)
Discussion started by: reaky
4 Replies

10. Shell Programming and Scripting

Perl Script to produce a file

hi i got a file called essay which contain few pages with many paragraphs. now i wanna with PERL to produce another file which called Essaylist that contain a sorted list of words that appear in the file essay. the format for Essaylist: $word found $times times on page a b c.... where $word... (3 Replies)
Discussion started by: mingming88
3 Replies
Login or Register to Ask a Question