Sponsored Content
Top Forums Shell Programming and Scripting Speeding up processing a file Post 302216465 by dlam on Saturday 19th of July 2008 09:28:40 AM
Old 07-19-2008
Speeding up processing a file

Hi guys, I'm hoping you can help me here. I've knocked up a script that looks at a (huge) log file, and pulls from each line the hour of each transaction and how long each transaction took.

The data is stored sequentially as:

07:01 blah blah blah 12456 blah
07:03 blah blah blah 234 blah
08:02 blah blah blah 9 blah


My script works, but because it searches through the whole script for each occurence of hour x , it is searching the file 24 times and taking about 45 seconds to complete.

The pseudo code for what I have written is:

HOUR=0
while HOUR < 24; do
read in file line by line
find line matching $HOUR and pass column 5 to array
HOUR = HOUR + 1
done


Obviously my actual code is a lot more complex than that, I can add it if it helps, but I think the above simplifies my request for help

Does anyone have any idea how to speed up the way I am doing this? I had a go at doing it like:

for variable in `cat ANYOLDFILE`
do
HOUR = first 2 chars of line
TRANSACTIONTIME=column 4
done


But that seemed to take just as long and I'm sure there must be someway to speed this up.

Using ksh by the way.

Thanks in advance for any help. Smilie
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

speeding up the compilation on SUN Solaris environment

Dear friends, Please let me know how do I increase the speed of my compilation in SUN Solaris environment. actually I have many subfolders which contains .cc files. when I compile makefile at the root it will take much time to compile all the subfolders and generates object(.o) files. Can... (2 Replies)
Discussion started by: swamymns
2 Replies

2. UNIX for Dummies Questions & Answers

Speeding up a Shell Script (find, grep and a for loop)

Hi all, I'm having some trouble with a shell script that I have put together to search our web pages for links to PDFs. The first thing I did was: ls -R | grep .pdf > /tmp/dave_pdfs.outWhich generates a list of all of the PDFs on the server. For the sake of arguement, say it looks like... (8 Replies)
Discussion started by: Dave Stockdale
8 Replies

3. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

4. UNIX for Dummies Questions & Answers

Speeding/Optimizing GREP search on CSV files

Hi all, I have problem with searching hundreds of CSV files, the problem is that search is lasting too long (over 5min). Csv files are "," delimited, and have 30 fields each line, but I always grep same 4 fields - so is there a way to grep just those 4 fields to speed-up search. Example:... (11 Replies)
Discussion started by: Whit3H0rse
11 Replies

5. Shell Programming and Scripting

Speeding up search and replace in a for loop

Hello, I am using sed in a for loop to replace text in a 100MB file. I have about 55,000 entries to convert in a csv file with two entries per line. The following script works to search file.txt for the first field from conversion.csv and then replace it with the second field. While it works fine,... (15 Replies)
Discussion started by: pbluescript
15 Replies

6. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

7. Shell Programming and Scripting

Speeding up substitutions

Hi all, I have a lookup table from which I am looking up values (from col1) and replacing them by corresponding values (from col2) in another file. lookup file a,b c,d So just replace a by b, and replace c by d. mainfile a,fvvgeggsegg,dvs a,fgeggefddddddddddg... (7 Replies)
Discussion started by: senhia83
7 Replies

8. Shell Programming and Scripting

Help speeding up script

This is my first experience writing unix script. I've created the following script. It does what I want it to do, but I need it to be a lot faster. Is there any way to speed it up? cat 'Tax_Provision_Sample.dat' | sort | while read p; do fn=`echo $p|cut -d~ -f2,4,3,8,9`; echo $p >> "$fn.txt";... (20 Replies)
Discussion started by: JohnN6
20 Replies

9. Shell Programming and Scripting

Speeding up shell script with grep

HI Guys hoping some one can help I have two files on both containing uk phone numbers master is a file which has been collated over a few years ad currently contains around 4 million numbers new is a file which also contains 4 million number i need to split new nto two separate files... (4 Replies)
Discussion started by: dunryc
4 Replies

10. Shell Programming and Scripting

Help 'speeding' up this 'parsing' script - taking 24+ hours to run

Hi, I've written a ksh script that read a file and parse/filter/format each line. The script runs as expected but it runs for 24+ hours for a file that has 2million lines. And sometimes, the input file has 10million lines which means it can be running for more than 2 days and still not finish.... (9 Replies)
Discussion started by: newbie_01
9 Replies
radiusd.conf(5) 					   FreeRADIUS configuration file					   radiusd.conf(5)

NAME
radiusd.conf - configuration file for the FreeRADIUS server DESCRIPTION
The radiusd.conf file resides in the radius database directory, by default /etc/raddb. It defines the global configuration for the FreeRA- DIUS RADIUS server. CONTENTS
There are a large number of configuration parameters for the server. Most are documented in the file itself as comments. This page docu- ments only the format of the file. Please read the radiusd.conf file itself for more information. The configuration file parser is independent of the server configuration. This means that you can put almost anything into the configura- tion file. So long as it is properly formatted, the server will start. When the server parses the configuration file, it looks only for those configurations it understands. Extra configuration items are ignored. This "feature" can be (ab)used in certain interesting ways. FILE FORMAT
The file format is line-based, like many other Unix configuration files. Each entry in the file must be placed on a line by itself, although continuations are supported. The file consists of configuration items (variable = value pairs), sections, and comments. Variables Variables can be set via: name = value Single and double-quoted strings are permitted: string1 = "hello world" string2 = 'hello mom' Sections A section begins with a section name, followed on the same line by an open bracket '{'. Section may contain other sections, com- ments, or variables. Sections may be nested to any depth, limited only by available memory. A section ends with a close bracket '}', on a line by itself. section { ... } Sections can sometimes have a second name following the first one. The situations where this is legal depend on the context. See the examples and comments in the radiusd.conf file for more information. section foo { ... } Comments Any line beginning with a (#) is deemed to be a comment, and is ignored. Comments can appear after a variable or section defini- tions. # comment foo = bar # set variable 'foo' to value 'bar' section { # start of section ... } # end of section Continuations Long lines can be broken up via continuations, using '' as the last character of the line. For example, the following entry: foo = "blah blah blah" will set the value of the variable "foo" to "blah blah blah". Any CR or LF is not turned into a space, but all other whitespace is preserved in the final value. REFERENCES
The value of a variable can reference another variable. These references are evaluated when the configuration file is loaded, which means that there is no run-time cost associated with them. This feature is most useful for turning long, repeated pieces of text into short ones. Variables are referenced by ${variable_name}, as in the following examples. foo = bar # set variable 'foo' to value 'bar' who = ${foo} # sets variable 'who' to value of variable 'foo' my = "${foo} a" # sets variable 'my' to "bar a" If the variable exists in a section or subsection, it can be referenced as ${section.subsection.variable}. Forward references are not allowed. Relative references are allowed, by pre-pending the name with one or more period. blogs = ${.foo} Will set variable blogs to the value of variable foo, from the current section. blogs = ${..foo} Will set variable blogs to the value of variable foo, from the section which contains the current section. blogs = ${modules.detail.filename} Will set variable blogs to the value of variable filename, of the detail module, which is in the modules section of the configuration file. Properties of anonymous parent sections may also be referenced, currently name and instance are supported. modules { example foo { file = ${.:name} } } Will set variable file to the name of the containing section (example). modules { example foo { file = ${.:instance} } } Will set variable file to the instance name of the containing section (foo). modules { example foo { file = ${..:name} } } Will set variable file to the name of the parent of the containing section (modules). FILES
/etc/raddb/radiusd.conf SEE ALSO
radiusd(8) unlang(5) AUTHOR
Alan DeKok <aland@freeradius.org> 28 Jun 2013 radiusd.conf(5)
All times are GMT -4. The time now is 08:45 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy