save every line in log file with matched string


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting save every line in log file with matched string
# 1  
Old 03-22-2009
Question save every line in log file with matched string

i have been doing this script to match every line in a current log file (access_log) with strings that i list from a path (consist of 100 of user's name ex: meggae )..
and then make a directory of every string from the text file (/path/meggae/) --->if it matched..
then print every line from the log file if it matched the string into a new file name base on the string (meggae.txt) in the new created directory (/path/meggae/meggae.txt)..

below is the example script that i made..it has no error so far but it uses 100% of my processes as well as my memory and sometimes it uses my swap space and my lappy go lagged and hang..i hope some explanation and some solution can help me not to crash my dear lappy Smilie


Quote:
#!/usr/bin/perl -w
use File::Basename;
use CGI;
use CGI::Carp qw(fatalsToBrowser);
use File::Path;
use File::Copy;
print "Content-type: text/html\r\n";
print "\r\n";

open (example7, "/path/foo/foo/access_log") || die ("Could not open file. <br> $!");
@cisco = <example7>;

$new = qx(ls /path/foo/) or die "Could not read file. <br> $!"; #contain user's name ex: meggae and other 100 users

@passwd = split(' ',$new); #to split every string from <space> char
print $passwd; # try to see if it works and it worked well

foreach $string (sort(@passwd)) {

my $path = "/path/foo/foo/$string/log";

if (! -d $path)
{
mkpath($path,755) or die "Failed to create $path: $!\n";
}

foreach $line (sort(@cisco))
{

if ($line =~ $string)
{
foreach ($line =~ $string)
{
$file = "$string.txt";
$append = 0;
if ($append){
open (OUTFILE,">$file");
}else{
open (OUTFILE,">>$file");
}
foreach $line (@cisco)
{
foreach ($line =~ $string)
{print OUTFILE $line;}
}
close (OUTFILE);
}
foreach ($path =~ $string) {
move($file,$path) or die "Failed to copy $file: $!\n"; }

}

}
}
any solutions will tremendously help me with this space/memory crashing and what have i done wrong Smilie
# 2  
Old 03-22-2009
Too many un-necessary nested foreach. From what I can see, the outer foreach for each entry in password and the inner foreach for each line in cisco is sufficient. Why do you have to repeatedly keep doing foreach even for opening and closing files?

What do you mean by "foreach ($line =~ $string)"? Why can't you write it as "if($line =~ $string)" and do what you need to do within the if block?

There are many other issues but you may want to sort out the above problems first and see if that makes the process less demanding.
# 3  
Old 03-24-2009
Java

Here's a way you could do it in ksh.. not the most efficient way, but worth a shot to see if it eats up your system resources as badly as your method..

Code:
#!/bin/ksh

logfile=/path/to/logfile
namelist=/path/to/namelist
outputdir=/output/dir


cat $namelist | while read name
do

   grep $name $logfile > temp
   if [ $(cat temp | wc -l) -ne 0 ];then 
    
     mkdir  ${outputdir}/$name
     cat temp > ${outputdir}/$name
     cat temp > ${outputdir}/${name}/${name}.txt
       
    fi

done

# 4  
Old 04-07-2009
rikxik..
thanks for the brilliant suggestion given..i also thought the problem might be with the programming (foreach statement) as i'm not into much of the programming..
i've review the code and cut out the unnecessary loop..it works and it works great if i execute it from the terminal instead of browser..
thanks insamniac for ur ksh opinion..i would really like to consider it but i don't really have much time now to learn new things..but i currently trying the code already..it surely gives me another sight way of solving coming problems..

Quote:
#!/usr/bin/perl -w

use File::Basename;
use CGI;
use CGI::Carp qw(fatalsToBrowser);
use File::Path;
use File::Copy;
print "Content-type: text/html\r\n";
print "\r\n";

$accessfile = "/var/log/foo/access_log";
open (example7, "/var/log/foo/access_log") || die ("Could not open file. <br> $!");

@cisco = <example7>;

@passwd = split(' ',$new);

open(READER, $accessfile) or die "canot read log:$accessfile:$!";
@LOGFILE = <READER>;

while(<READER>) {
($machine, $date, $time, $tzone, $method, $url, $httpvers, $response, $size) = /([^\s]*) [^\s]* [^\s]* \[([^\:]*)Smilie[^\s]*) ([^\]]*)\] \"([^\s]*) ([^\s]*) HTTP\/([^\"]*)\" (\d*) (\d*)/;

($day, $month, $year) = split('/',$date);


}

foreach $string (sort(@passwd)) {


my $path = "/path/foo/foo/$string/log";

if (! -d $path)
{
mkpath($path,755) or die "Failed to create $path: $!\n";
}

$file = "$string.txt";

if ($path =~ $string) {
foreach $line (sort(@cisco)) {
$append = 0;
if ($append){
open (OUTFILE,">$file"); if ($line =~ $string) {print OUTFILE $line;}
}else{
open (OUTFILE,">>$file"); if ($line =~ $string) {print OUTFILE $line;}
} }
close (OUTFILE);
move($file,$path) or die "Failed to copy $file: $!\n";
}


}
close (example7);

Last edited by meggae; 04-07-2009 at 12:44 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Insert a user input string after matched string in file

i am having file like this #!/bin/bash read -p 'Username: ' uservar match='<color="red" />' text='this is only a test so please be patient <color="red" />' echo "$text" | sed "s/$match/&$uservar\g" so desireble output what i want is if user type MARIA this is only a test so please... (13 Replies)
Discussion started by: tomislav91
13 Replies

2. UNIX for Dummies Questions & Answers

Commenting a line matched with a specific string in a file

Hi, I would like to comment a line that matched a string "sreenivas" in a file without opening it. Thanks in advance. Regards, Sreenivas (3 Replies)
Discussion started by: raosr020
3 Replies

3. Shell Programming and Scripting

ksh : need to get the 4 th line above and 2 nd below the matched pattern in the log file

I have a log file as given below 012/01/21 10:29:02 (111111) Processing Job '23_369468343464564' 2012/01/21 10:29:02 (111111) Making Job '23_369468343464564.0'... 2012/01/21 10:29:04 (111111) Jobnumber '23_369468343464564' was successful 2012/01/21 10:29:04 ... (12 Replies)
Discussion started by: rpm120
12 Replies

4. Shell Programming and Scripting

Print only matched string instead of entire line

Hi, I have a file whose lines are something like Tchampionspsq^@~^@^^^A^@^@^@^A^A^Aÿð^@^@^@^@^@^@^@^@^@^@^A^@^@^@^@^?ð^@^@^@^@^@^@^@?ð^@^@^@^@^@^@pppsq^@~^@#@^@^@^@^@^@^Hw^H^@^@^@^K^@^@^@^@xp^At^@^FTtime2psq^@ ~^@^^^A^@^@^@^B^A I need to extract all words matching T*psq from the file. Thing is... (4 Replies)
Discussion started by: shekhar2010us
4 Replies

5. Shell Programming and Scripting

Help required on joining one line above & below to the pattern matched string line.

Hi Experts, Help needed on joining one line above & below to the pattern matched string line. The input file, required output is mentioned below Input file ABCD DEFG5 42.0.1-63.38.31 KKKK iokl IP Connection Available ABCD DEFG5 42.0.1-63.38.31 ... (7 Replies)
Discussion started by: krao
7 Replies

6. Shell Programming and Scripting

Match a line in File 1 with Column in File 2 and print whole line in file 2 when matched

Hi Experts, I am very new to scripting and have a prb since few days and it is urgent to solve so much appreciated if u help me. i have 2 files file1.txt 9647810043118 9647810043126 9647810043155 9647810043161 9647810043166 9647810043185 9647810043200 9647810043203 9647810043250... (22 Replies)
Discussion started by: mustafa.abdulsa
22 Replies

7. UNIX for Advanced & Expert Users

capture data from matched string/line

Hi, I have a query as follows : suppose I am matching a string in a file say "start from here" and I want to pick up 'n' number of lines () from the matched string. Is there any way to do that ? 1) going forward I want to do this for every match for the above string 2) or limit this to... (2 Replies)
Discussion started by: sumoka
2 Replies

8. Shell Programming and Scripting

Extracting particular string in a file and storing matched string in output file

Hi , I have input file and i want to extract below strings <msisdn xmlns="">0492001956</ msisdn> => numaber inside brackets <resCode>3000</resCode> => 3000 needs to be extracted <resMessage>Request time getBalances_PSM.c(37): d out</resMessage></ns2:getBalancesResponse> => the word... (14 Replies)
Discussion started by: sushmab82
14 Replies

9. Shell Programming and Scripting

search a string in a line and save it in a variable

Hi I want to read a file line by line and search for a particular string in each line(say for example string containing @ )and save that string into a variable. Can someone suggest me the way to implement it.I am using K- shell Thanks Ishita (5 Replies)
Discussion started by: Ishita
5 Replies

10. UNIX for Advanced & Expert Users

Command to display nth line before the string is matched.

All, Is there any way out to display the nth line before the string is matched ??? Eg : If i have a file which has the following contents and if i want to get the 3rd line before the string is matched a b c d e f if i give the input as f and lines before the match as 3 then it should... (5 Replies)
Discussion started by: helper
5 Replies
Login or Register to Ask a Question