Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Shell scripting: frequency of specific word in a string and statistics
Top Forums Shell Programming and Scripting Shell scripting: frequency of specific word in a string and statistics Post 302805795 by Skrynesaver on Saturday 11th of May 2013 05:53:33 AM
Old 05-11-2013
Hi Kraterions,

that wasn't a script, but rather a command line one off, as a script it would look like this, with advised addons.
Code:
#!/usr/bin/perl
use strict;

open(my $tweets, $ARGV[0])|| die "Couldn't open $ARGV[0] $!\n";
my %counts;
my %freq;
my $tot;
while(<$tweets>){
        chomp;
        my @rec=split(/,/, $_, 3);
        my @words=split/\b\s*/,$rec[2];
        map {   $counts{lc($_)}++ if /^\w+$/;
                $freq{$rec[0]}{lc($_)}++ if /^\w+$/;
                $tot++}@words;
}
my @wanted=qw(that really great day);
print "WORD,TOTFrequency,(1)Frequency,(0)Frequency,(-1)Frequency,1%Frequency,0%Frequency,-1%Frequency\n";
for (sort {$counts{b}<=>$counts{a}} @wanted){
        print "$_,$counts{$_},",$freq{1}{$_}||0,",",$freq{0}{$_}||0,",",$freq{-1}{$_}||0,sprintf("%0.2f",($counts{$_}/$tot)*100),"%,",sprintf("%0.2f",($freq{0}{$_}/$tot)*100),"%,",sprintf("%0.2f",($freq{-1}{$_}/$tot)*100),"%\n";
}

Last edited by Skrynesaver; 05-11-2013 at 07:10 AM..
This User Gave Thanks to Skrynesaver For This Post:
kraterions
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Determining Word Frequency of Specific Terms

Hello, I require a perl script that will read a .txt file that contains words like 224.199.207.IN-ADDR.ARPA. IN NS NS1.internet.com. 4.200.162.207.in-addr.arpa. IN PTR beeriftw.internet.com. arroyoeinternet.com. IN A 200.199.227.49 I want to focus on words: IN... (23 Replies)
Discussion started by: richsark
23 Replies

2. Shell Programming and Scripting

Finding a word at specific location in a string

Hi All , I have different strings (SQL queries infact) of different lengths such as: 1. "SELECT XYZ FROM ABC WHERE ABC.DEF='123' " 2. "DELETE FROM ABC WHERE ABC.DEF='567'" 3. "SELECT * FROM ABC" I need to find out the word coming after the... (1 Reply)
Discussion started by: swapnil.nawale
1 Replies

3. Shell Programming and Scripting

search a word and print specific string using awk

Hi, I have list of directory paths in a variable and i want to delete those dirs and if dir does not exist then search that string and get the correct path from xml file after that delete the correct directory. i tried to use grep and it prints the entire line from the search.once i get the entire... (7 Replies)
Discussion started by: dragon.1431
7 Replies

4. Shell Programming and Scripting

awk or sed command to print specific string between word and blank space

My source is on each line 98.194.245.255 - - "GET /disp0201.php?poc=4060&roc=1&ps=R&ooc=13&mjv=6&mov=5&rel=5&bod=155&oxi=2&omj=5&ozn=1&dav=20&cd=&daz=&drc=&mo=&sid=&lang=EN&loc=JPN HTTP/1.1" 302 - "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.0.3705; .NET CLR... (5 Replies)
Discussion started by: elamurugu
5 Replies

5. Shell Programming and Scripting

search-word-print-specific-string

Hi, Our input xml looks like: <doc> <str name="account_id">1111</str> <str name="prd_id">DHEP155EK</str> </doc> - <doc> <str name="account_id">6666</str> <str name="prd_id">394531662</str> </doc> - <doc> <str name="account_id">6666</str> <str... (1 Reply)
Discussion started by: Jassz
1 Replies

6. Shell Programming and Scripting

Parse a String for a Specific Word

Hello, I'm almost there with scripting, and I've looked at a few examples that could help me out here. But I'm still at a lost where to start. I'm looking to parse each line in the log file below and save the output like below. Log File AABBCGCAT022|242|3 AABBCGCAT023|243|4... (6 Replies)
Discussion started by: ravzter
6 Replies

7. Shell Programming and Scripting

break the string and print it in a new line after a specific word

Hi Gurus I am new to this forum.. I am using HP Unix OS. I have one single string in input file as shown below Abc123 | cde | fgh | ghik| lmno | Abc456 |one |two |three | four | Abc789 | five | Six | seven | eight | Abc098 | ........ I want to achive the result in a output file as shown... (3 Replies)
Discussion started by: kannansr621
3 Replies

8. UNIX for Dummies Questions & Answers

How to print line starts with specific word and contains specific word using sed?

Hi, I have gone through may posts and dint find exact solution for my requirement. I have file which consists below data and same file have lot of other data. <MAPPING DESCRIPTION ='' ISVALID ='YES' NAME='m_TASK_UPDATE' OBJECTVERSION ='1'> <MAPPING DESCRIPTION ='' ISVALID ='NO'... (11 Replies)
Discussion started by: tmalik79
11 Replies

9. Shell Programming and Scripting

Help with calculating frequency of specific word in a string

Input file: #read_1 AWEAWQQRZZZQWQQWZ #read_2 ZZAQWRQTWQQQWADSADZZZ #read_3 POGZZZZZZADWRR . . Desired output file: #read_1 3 #read_1 1 #read_2 2 #read_2 3 #read_3 6 . . (3 Replies)
Discussion started by: perl_beginner
3 Replies

10. UNIX for Beginners Questions & Answers

Get string before specific word in UNIX

Hi All, I'm writing unix shell script and I have these files. I need to get name before _DETL.tmp. ABC_AAA_DETL.tmp ABC_BBB_DETL.tmp ABC_CCC_DETL.tmp PQR_DETL.tmp DEF_DETL.tmp JKL_DETL.tmp YUI_DETL.tmp TG_NM_DDD_DETL.tmp TG_NM_EEE_DETL.tmp GHJ_DETL.tmp RTY_DETL.tmp output will... (3 Replies)
Discussion started by: ace_friends22
3 Replies

LEARN ABOUT OSF1

sh

sh(1)							      General Commands Manual							     sh(1)

NAME

       sh - Shell, the standard command language interpreter

DESCRIPTION

       [Tru64  UNIX]  Tru64  UNIX  provides two command interpreters with the name sh.	The XCU5.0 and POSIX.2 compliant command interpreter sh is
       available in the file /usr/bin/posix/sh and is described in the sh(1p) reference page.  The Bourne shell,  historically	known  as  sh,	is
       available in the file /usr/bin/sh and is described in the sh(1b) reference page.

       [Tru64 UNIX]  Your initial, or login, shell is determined by your entry in the file /etc/passwd. This file can be changed only by your sys-
       tem administrator. You must use whatever procedures are in place at your location to have this entry changed.

       [Tru64 UNIX]  If available on your system, you may use the passwd -s or the chsh commands to change your login shell.

									  Note

       This option is not available if your site manages passwords through the Network Information Service (NIS) facility.  Check with your system
       administrator.

       [Tru64 UNIX]  Subsequent shells spawned from the initial shell depend on the value in the environment variable BIN_SH.  If this variable is
       set to xpg4, the POSIX shell is started. If this variable is set to svr4, an SVR4 compliant version of the shell is started. If this  vari-
       able  is  unset,  the  Bourne shell is started. If this variable is set to any other value, an error is reported and the results are unpre-
       dictable. See the EXAMPLES section for information on setting this variable.

NOTES

       [Tru64 UNIX]  With Tru64 UNIX Version 4.0 the Korn shell, /usr/bin/ksh is the same as the POSIX shell /usr/bin/posix/sh.

RESTRICTIONS

       [Tru64 UNIX]  The file /etc/shells must include entries for both the POSIX shell /usr/bin/posix/sh and the Bourne  shell,  /usr/bin/sh.	If
       this file is incorrect, see your system administrator.

EXAMPLES

       Using  the  Bourne,  Korn,  or  POSIX  shell, to set the variable BIN_SH to use the POSIX/ XCU5.0compliant shell, enter: BIN_SH=xpg4 export
       BIN_SH Using the Bourne, Korn, or POSIX shell, to set the variable BIN_SH to use the SVR4 compliant shell, enter: BIN_SH=svr4 export BIN_SH
       Using the Bourne, Korn, or POSIX shell, to unset the variable BIN_SH, enter: unset BIN_SH Using the C/ shell, to set the variable BIN_SH to
       use the POSIX/XCU5.0 compliant shell, enter: setenv BIN_SH xpg4 Using the C/ shell, to set the variable BIN_SH to use  the  SVR4  compliant
       shell, enter: setenv BIN_SH svr4 Using the C/ shell, to unset the variable BIN_SH, enter: unsetenv BIN_SH

FILES

       User profile.  Contains user information, including the login shell name.  Contains the names of available and permitted shells.

SEE ALSO

       Commands:  csh(1), ksh(1), Bourne shell sh(1b), POSIX shell sh(1p), passwd(1)

       Files:  passwd(4), shells(4)

       Standards:  standards(5)

																	     sh(1)
All times are GMT -4. The time now is 03:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy