Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Modifying the values of dynamically named arrays

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Modifying the values of dynamically named arrays
# 8  
Old 05-04-2012
Quote:
Originally Posted by Corona688
awk is almost certainly better suited to this. There may be limits on the size and number of shell variables, especially, which could become a severe problem here, but awk's only limit is memory itself. Show your problem in more detail please.
The log file that is being parsed looks like this:

2012-03-28 00:00:33,849 [ORB.thread.pool : 0] DEBUG - getWkfltgDecisionStart.EntityToDomainConverter133290723384913329072338490
2012-03-28 00:00:33,849 [ORB.thread.pool : 0] DEBUG - docTypeCheck(WkfltgDecision decision, String docType)Start.DMServicesHelper133290723384713329072338492
2012-03-28 00:00:33,849 [ORB.thread.pool : 0] DEBUG - docTypeCheck(WkfltgDecision decision, String docType)Start.DMServicesHelper133290723384913329072338490
2012-03-28 00:00:33,852 [ORB.thread.pool : 0] DEBUG - validateTaxQueue(WorkQueueItem wrkQueueItem)Start.DMServicesHelper1332907233822133290723385230
2012-03-28 00:00:33,982 [ORB.thread.pool : 6] DEBUG - createDMStatistics-serviceStart.DocMgtDBDAOImpl133290723397913329072339823

The order and amount of methods are random. Each log file is around 500,000 lines. I'm trying to find all instances of each method and get the average time (last column). I have a working version in java but it's very slow and wanted to see if it could be done faster in unix.
# 9  
Old 05-04-2012
What are the instances and what are the methods?
# 10  
Old 05-04-2012
Quote:
Originally Posted by Scrutinizer
What are the instances and what are the methods?
Sorry, by instance I mean occurrence of a given method name. The method names are the words after "DEBUG" and before "Start". So in the snippet I provided, "docTypeCheck(WkfltgDecision decision, String docType)" would have 2 instances and the rest would have 1. But there are thousands of each method name spread out through the log file.
# 11  
Old 05-04-2012
Are the columns separated by | ? Try this:
Code:
awk -F'DEBUG - |\\|' '{C[$2]++;T[$2]+=$NF} END{for(i in C)print i,T[i]/C[i]}' infile

This User Gave Thanks to Scrutinizer For This Post:
nicksantos1
# 12  
Old 05-04-2012
Quote:
Originally Posted by Scrutinizer
Are the columns separated by | ? Try this:
Code:
awk -F'DEBUG - |\\|' '{C[$2]++;T[$2]+=$NF} END{for(i in C)print i,T[i]/C[i]}' infile

Wow, thanks a lot. This works, although I have no idea how. You converted 30 lines of script into 1. Some of the values are off by .5 or something but it's not a big deal, it's close enough.
# 13  
Old 05-04-2012
Good, the values should not be off by 0.5 though, better check what is going on and if the results match.. If you really need extra speed, I recommend using mawk instead of awk, which is the fastest awk out there...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Dynamically merging 2 files on header values

Hi All, I have 2 files which i need to merge together based on the column names provided in the file. The first line in both files are header records. The first file has fixed columns but second file can have subset of the columns from file 1 File 1: ... (6 Replies)
Discussion started by: kushagra
6 Replies

2. Programming

Retreving the dynamically allocated values from bdb using C

In one of the assignment which i am working on, i am am trying to insert keys and values into BDB by reading the input records from a input file as below. Here keys i am inserting as character buffer and for values i am dynamically allocating the memory using malloc and then inserting into bdb.... (1 Reply)
Discussion started by: AmbikaValagonda
1 Replies

3. UNIX for Advanced & Expert Users

Retreving the dynamically allocated values from bdb using C

In one of the assignment which i am working on, i am am trying to insert keys and values into BDB by reading the input records from a input file as below. Here keys i am inserting as character buffer and for values i am dynamically allocating the memory using malloc and then inserting into bdb.... (1 Reply)
Discussion started by: AmbikaValagonda
1 Replies

4. Programming

modifying 2 or more values using method in Java

i wanna make modification in two or more identifiers,using a method in JAVA , but the only way to do this is by returning a value ,then also only one variable value can be modified .. one way 2 do do this is by use of array and passing its name in method ,, please suggest a different way of doing... (2 Replies)
Discussion started by: upvan111
2 Replies

5. Shell Programming and Scripting

arrays and two values

I have requirement where I need to capture the highest values of items from a feed that runs for N hours. For example lets assume my data looks like this first feed ======== appples 10 oranges 20 pears 14 second feed ========== apples 5 oranges 30 pears 1 Last feed... (6 Replies)
Discussion started by: BeefStu
6 Replies

6. Shell Programming and Scripting

Storing values in arrays

I have the following csh script which lets the use pass the following as an argument -legend=tag1/tag2/tag3/tag4/tag5/tag6/tag7 We pass a number of tags separated by '/'. I want to save the legend tags in an array and have done as below. How can I improve on this as things are getting quite... (3 Replies)
Discussion started by: kristinu
3 Replies

7. Shell Programming and Scripting

Storing values in arrays using csh

I am reading a number of files but then I want to put the ranges xmin xmax ymin ymax as arrays for each file. Any idea how I can do this??? set j = 1 echo "Welcome $i times" while ( $j <= $i ) echo "$j" set fname = $fin-bst-misf.xy echo " "$fname ... (0 Replies)
Discussion started by: kristinu
0 Replies

8. Shell Programming and Scripting

dynamically adding values in c-shell

I am needing to create a variable(changing) and assign it a value(changing) ... I am using C-Shell.. Example: foreach account in ($Accountlist) set account_connect = "$account/$account_pass" end I want to make set account_connect to store various values ? $account_connect did not... (3 Replies)
Discussion started by: shafi2all
3 Replies

9. Shell Programming and Scripting

Dynamically create arrays

I have a bash challenge. I have several virtualization hosts for which I need to collect VM information. I need to dynamically create an array for each host. The contents of the array should be the list VMs running on the host. Here's the logic; I just need someone to help me with the details:... (0 Replies)
Discussion started by: EdgarTorres
0 Replies

10. UNIX for Advanced & Expert Users

Modifying binary file by editing Hex values ?

Hi , i'm using special binary file (lotus notes) and modifying an hexadecimal address range with windows hex editor and it works fine ! The file is an AIX one and i'm forced to transfert (ftp) it before modifying it and re-transfert ! NOW i would do this directly under AIX ! I can... (4 Replies)
Discussion started by: Nicol
4 Replies
Login or Register to Ask a Question