Sponsored Content
Top Forums Shell Programming and Scripting Determining Word Frequency of Specific Terms Post 302295029 by ShawnMilo on Friday 6th of March 2009 10:54:04 AM
Old 03-06-2009
Code:
#!/usr/bin/env python

import re

p_types = re.compile(r'^IN\s+(\w+)\s*.*$')

log_lines = open(r'c:\temp\temp.txt', 'r')

type_counts = {}

for line in log_lines:
    if p_types.search(line):
        log_type = p_types.sub(r'\1', line)
        type_counts.setdefault(log_type, 0)
        type_counts[log_type] += 1
        
for log_type, count in type_counts.items():
    
    print "Total number of %s records: %d" % (log_type, count)

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Word frequency with additional information

Hello everyone, I am using a chunk of code to display the frequency of a file name in a list of directories. The code looks like this: find . -name "*.log" | cut -d/ -f4 | cut -d. -f1 | awk '{print $1}' | sort | uniq -c | sort -nr The file paths would look something like this:... (1 Reply)
Discussion started by: ToeLint
1 Replies

2. Shell Programming and Scripting

word frequency counter - awk solution?

Dear all, i need your help on this. There is a text file, i need to count word frequency for each word with frequency >40 in each line of file and output it into another file with columns like this: word1,word2,word3, ...wordn 0,0,1 1,2,0 3,2,0 etc -- each raw represents... (13 Replies)
Discussion started by: irrevocabile
13 Replies

3. Shell Programming and Scripting

Word Frequency Sort

hello, Here is a program for creating a word-frequency # wf.gk --- program to generate word frequencies from a file { # remove punctuation: This will remove all punctuations from the file gsub(/_]/, "", $0) #Start frequency analysis for (i = 1; i <= NF; i++) freq++ } END #Print output... (11 Replies)
Discussion started by: gimley
11 Replies

4. UNIX for Dummies Questions & Answers

How to print line starts with specific word and contains specific word using sed?

Hi, I have gone through may posts and dint find exact solution for my requirement. I have file which consists below data and same file have lot of other data. <MAPPING DESCRIPTION ='' ISVALID ='YES' NAME='m_TASK_UPDATE' OBJECTVERSION ='1'> <MAPPING DESCRIPTION ='' ISVALID ='NO'... (11 Replies)
Discussion started by: tmalik79
11 Replies

5. Shell Programming and Scripting

Help with calculating frequency of specific word in a string

Input file: #read_1 AWEAWQQRZZZQWQQWZ #read_2 ZZAQWRQTWQQQWADSADZZZ #read_3 POGZZZZZZADWRR . . Desired output file: #read_1 3 #read_1 1 #read_2 2 #read_2 3 #read_3 6 . . (3 Replies)
Discussion started by: perl_beginner
3 Replies

6. Shell Programming and Scripting

Fetch entries in front of specific word till next word

Hi all I have following file which I have to edit for research purpose file:///tmp/moz-screenshot.png body, div, table, thead, tbody, tfoot, tr, th, td, p { font-family: &quot;Liberation Sans&quot;; font-size: x-small; } Drug: KRP-104 QD Drug: Placebo Drug: Metformin|Drug:... (15 Replies)
Discussion started by: Priyanka Chopra
15 Replies

7. Shell Programming and Scripting

Convert a list of word/terms into their Regexp representation

Ok this might sound pretty weird but here is the request. Running on a linux system in bash or Perl (i really don't know perl but the end user has a few pearl script already) Start File looks something like this (4000 entries) TEST PLAN T//TF T-TF TEST (T) Hacker ... I am thinking about... (3 Replies)
Discussion started by: oly_r
3 Replies

8. Shell Programming and Scripting

Shell scripting: frequency of specific word in a string and statistics

Hello friends, I need a BIG help from UNIX collective intelligence: I have a CSV file like this: VALUE,TIMESTAMP,TEXT 1,Sun May 05 16:13:05 +0000 2013,"RT @gracecheree: Praying God sends me a really great man one day. Gotta trust in his timing. 0,Sun May 05 16:13:05 +0000 2013,@sendi__... (19 Replies)
Discussion started by: kraterions
19 Replies

9. Shell Programming and Scripting

Count frequency of unique values in specific column

Hi, I have tab-deliminated data similar to the following: dot is-big 2 dot is-round 3 dot is-gray 4 cat is-big 3 hot in-summer 5 I want to count the frequency of each individual "unique" value in the 1st column. Thus, the desired output would be as follows: dot 3 cat 1 hot 1 is... (5 Replies)
Discussion started by: owwow14
5 Replies

10. Shell Programming and Scripting

Search for a specific word and print only the word from the input file

Hi, I have a sample file as shown below, I am looking for sed or any command which prints the complete word only from the input file. Ex: $ cat "sample.log" I am searching for a word which is present in this file We can do a pattern search using grep but I need to cut only the word which... (1 Reply)
Discussion started by: mohan_kumarcs
1 Replies
lfc_python(3)							 Python Reference						     lfc_python(3)

NAME
lfc - Python interface to the LFC lfcthr - Thread enabled version of Python interface to the LFC SYNOPSIS
import lfc import lfcthr lfcthr.init() DESCRIPTION
The lfc module permits you to access the LFC client interface from python programs. The lfc module is a swig wrapping of the standard C interface. For detailed descriptions of each function see the individual man page of each function. The lfcthr module is a version of the lfc module supporting multi-threaded Python clients. Its usage is similar to the usage of the lfc module except the obligatory initialisation call lfcthr.init() in the main program before threads are started. There follows a series of examples of how to use selected functions and how to retrieve the information returned by them: Examples are finding the GUID of an existing entry, listing the replicas of a given GUID and setting and retrieving the comment associated with an entry. EXAMPLE
#!/usr/bin/python import sys import lfc """ # stat an existing entry in the LFC and print the GUID """ name = "/grid/dteam/my.test" stat = lfc.lfc_filestatg() res = lfc.lfc_statg(name,"",stat) if res == 0: guid = stat.guid print "The GUID for " + name + " is " + guid else: err_num = lfc.cvar.serrno err_string = lfc.sstrerror(err_num) print "There was an error while looking for " + name + ": Error " + str(err_num) + " (" + err_string + ")" sys.exit(1) EXAMPLE
#!/usr/bin/python import lfc """ # list the replicas of a given entry, starting from the GUID """ guid = "6a3164e0-a4d7-4abe-9f76-e3b8882735d1" listp = lfc.lfc_list() flag = lfc.CNS_LIST_BEGIN print "Listing replicas for GUID " + guid num_replicas=0 while(1): res = lfc.lfc_listreplica("",guid,flag,listp) flag = lfc.CNS_LIST_CONTINUE if res == None: break else: rep_name = res.sfn print "Replica: " + rep_name num_replicas = num_replicas + 1 lfc.lfc_listreplica("",guid,lfc.CNS_LIST_END,listp) print "Found " + str(num_replicas) + " replica(s)" EXAMPLE
#!/usr/bin/python import sys import lfc import re """ # setting and retrieving a comment on a file """ file = "/grid/dteam/my.test" comment = "MyComment" res = lfc.lfc_setcomment(file,comment) if res != 0: err_num = lfc.cvar.serrno err_string = lfc.sstrerror(err_num) print "Problem while setting comment for " + file + ": Error " + str(err_num) + " (" + err_string + ")" sys.exit(1) buffer="" for i in range(0,lfc.CA_MAXCOMMENTLEN+1): buffer=buffer + " " res = lfc.lfc_getcomment(file,buffer) if res != 0: err_num = lfc.cvar.serrno err_string = lfc.sstrerror(err_num) print "Problem while reading the comment for " + file + ": Error " + str(err_num) + " (" + err_string + ")" sys.exit(1) r = re.compile("(.*?) ", re.DOTALL) comment = r.findall(buffer)[0] print "Read back comment " + comment EXAMPLE
#!/usr/bin/python """ # Using the lfc_readdirxr method """ import sys import lfc name = "/grid/dteam/my.test" dir = lfc.lfc_opendirg(name,"") if (dir == None) or (dir == 0): err_num = lfc.cvar.serrno err_string = lfc.sstrerror(err_num) print "Error while looking for " + name + ": Error " + str(err_num) + " (" + err_string + ")" sys.exit(1) while 1: read_pt = lfc.lfc_readdirxr(dir,"") if (read_pt == None) or (read_pt == 0): break entry, list = read_pt print entry.d_name try: for i in range(len(list)): print " ==> %s" % list[i].sfn except TypeError, x: print " ==> None" lfc.lfc_closedir(dir) EXAMPLE
#!/usr/bin/python import lfc """ # Using the lfc_getlinks method """ result, list = lfc.lfc_getlinks("/grid/dteam/antotests/extratests/dir2/f105", "") print result print len(list) if (result == 0): for i in list: print i.path EXAMPLE
#!/usr/bin/python import lfc """ # Using the lfc_getreplica method """ result, list = lfc.lfc_getreplica("/grid/dteam/antotests/extratests/dir2/f105", "", "") print result print len(list) if (result == 0): for i in list: print i.host print i.sfn EXAMPLE
#!/usr/bin/python import lfc """ # Using the lfc_getacl and lfc_setacl methods to add a user ACL """ nentries, acls_list = lfc.lfc_getacl("/grid/dteam/tests_sophie3", lfc.CA_MAXACLENTRIES) print nentries print len(acls_list) for i in acls_list: print i.a_type print i.a_id print i.a_perm # When adding a first ACL for a given user, you also need to add the mask # When adding the second user ACL, it is not necessary anymore acl_user = lfc.lfc_acl() acl_mask = lfc.lfc_acl() acl_user.a_type=2 # 2 corresponds to CNS_ACL_USER acl_user.a_id=18701 # user id acl_user.a_perm=5 acl_mask.a_type=5 # 5 corresponds to CNS_ACL_MASK acl_mask.a_id=0 # no user id specified acl_mask.a_perm=5 acls_list.append(acl_user) acls_list.append(acl_mask) res = lfc.lfc_setacl("/grid/dteam/tests_sophie3", acls_list) if res == 0: print "OK" else: err_num = lfc.cvar.serrno err_string = lfc.sstrerror(err_num) print "There was an error : Error " + str(err_num) + " (" + err_string + ")" sys.exit(1) EXAMPLE
#!/usr/bin/python import lfc """ # Using the lfc_getacl and lfc_setacl methods to remove a user ACL """ nentries, acls_list = lfc.lfc_getacl("/grid/dteam/tests_sophie3", lfc.CA_MAXACLENTRIES) # Note : you cannot remove the owner ACL (i.e. for CNS_ACL_USER_OBJ type) if ACLs # ====== for other users exist. Ff all the other user ACLs are deleted, the owner # ====== ACL is automatically removed. for i in acls_list: print i.a_type print i.a_id print i.a_perm del acls_list[1] # delete a given user ACL from the list of ACLs res = lfc.lfc_setacl("/grid/dteam/tests_sophie3", acls_list) if res == 0: print "OK" else: err_num = lfc.cvar.serrno err_string = lfc.sstrerror(err_num) print "There was an error : Error " + str(err_num) + " (" + err_string + ")" sys.exit(1) EXAMPLE
#!/usr/bin/env python import lfcthr import os from threading import Thread class slave(Thread): def __init__ (self): Thread.__init__(self) def run(self): .... result = lfcthr.lfc_getreplica("", guid, "") .... return if __name__ == '__main__': os.environ['LFC_HOST'] = 'my_lfc.cern.ch' # Threaded library initialisation lfcthr.init() .... # Start up of threads for i in xrange(totalNumberOfSlaves): slv = slave(i) slv.start() .... EXAMPLE
#!/usr/bin/python import lfc """ # Using the lfc_getusrmap method """ result, list = lfc.lfc_getusrmap() print result print len(list) if (result == 0): for i in list: print i.userid + " " + i.username EXAMPLE
#!/usr/bin/python import lfc """ # Using the lfc_getgrpmap method """ result, list = lfc.lfc_getgrpmap() print result print len(list) if (result == 0): for i in list: print i.gid + " " + i.groupname KNOWN BUGS
The current interface to the lfc_getcomment(3), lfc_getcwd(3), lfc_readlink(3), lfc_seterrbuf(3) requires the passing of str object which is modified to contain the result (in a similar way to the C functions, which accept a buffer). However this breaks the immutability of python str. This will be changed in the future. SEE ALSO
LFC C interface man pages LFC
$Date: 2010-02-04 13:08:39 +0100 (Thu, 04 Feb 2010) $ lfc_python(3)
All times are GMT -4. The time now is 06:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy