|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
Reading/Compressing of log file
Post 302344980 by sgbhat on Tuesday 18th of August 2009 06:30:36 AM
08-18-2009
Reading/Compressing of log file
Hello All
Posted a similar thread in some other section too. Regrets if this section is not suitable for this post. Request all the members to be tolerant as i'm a newbie here
Coming to the topic : I've this huge log files of size 20GB-30 GB in my unix server. I want to analyse the log file for some error messages. But because of the enormity of the size of these files i'm not able to grep/search the pattern in the file [ It's taking too much time]. Also, tried to gzip the file and copy it to local machine but again it's becoming tedious.
So any expert advice on how can i go about approaching this issue. Is there any particular way or method of how exactly i can extract the needed information from the log file. Your help is highly appreciated.
Posted a similar thread in some other section too. Regrets if this section is not suitable for this post. Request all the members to be tolerant as i'm a newbie here
Coming to the topic : I've this huge log files of size 20GB-30 GB in my unix server. I want to analyse the log file for some error messages. But because of the enormity of the size of these files i'm not able to grep/search the pattern in the file [ It's taking too much time]. Also, tried to gzip the file and copy it to local machine but again it's becoming tedious.
So any expert advice on how can i go about approaching this issue. Is there any particular way or method of how exactly i can extract the needed information from the log file. Your help is highly appreciated.
|
|
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi all,
I have the following shell script code which tries to sftp and writes the log into the log file.
TestConnection ()
{
echo 'Connection to ' $DESTUSERNAME@$DESTHOSTNAME
$SETDEBUG
if ]; then rm $SCRIPT ; fi
touch $SCRIPT
echo "cd" $REMOTEDIR >> $SCRIPT
echo "quit" >>... (10 Replies)
Discussion started by: valluvan
10 Replies
2. UNIX for Dummies Questions & Answers
Hello All
My first post in the forum. :)
I've this huge log files of size 20GB-30 GB in my unix server. I want to analyse the log file for some error messages. But because of the enormity of the size of these files i'm not able to grep/search the pattern in the file . Also, tried to gzip the... (1 Reply)
Discussion started by: sgbhat
1 Replies
3. Shell Programming and Scripting
Need to develop a unix shell script for the below requirement and I need your assistance:
1) search for file.log and file.bad file in a directory and read them
2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file
4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies
4. Shell Programming and Scripting
i need to backup a directory from one partition to another and and compress that directory after backing up, so i need to predict the compressed size of the directory with out actually compressing it, to check whether the space is available in the destination partition to accommodate the zipped... (2 Replies)
Discussion started by: Kesavan
2 Replies
5. Shell Programming and Scripting
I want to create a script that will zip the previous log.
Example.
abc.log.2012.12.02
abc.log.2012.12.01.gzip
abc.log
If today is 2012.12.03 , my current log is abc.log and my previous date is 2012.12.02, i want abc.log.2012.12.02 to compress everytime I run the script.
I can... (5 Replies)
Discussion started by: kaibiganmi
5 Replies
6. Shell Programming and Scripting
Hi,
Is there any way to change the file name while compressing ? using Compress and gzip and tar ?
Say, I have a file foo.txt - so I have to compress this file and the resultant file name is foo.txt_20130113.gz or foo.txt_20130113.Z
This to be done while performing the compression... (2 Replies)
Discussion started by: karumudi7
2 Replies
7. Shell Programming and Scripting
I have below files in foler one/archive>
one. txt 6/21/2013
two txt 7/23/2013
three.txt 6/20/2013
I wanted to move all the old files (>30 days) compressing single .zip file into one/archive/ as below
two txt 7/23/2013
oldfiles.zip 6/21/2013
Please provide... (6 Replies)
Discussion started by: Ganesh L
6 Replies
8. Shell Programming and Scripting
Reading line by line from live log file using while loop and considering only those lines start from
Hi,
I want to read a live log file line by line and considering those line which start from time stamp;
Below code I am using, which read line but throws an exception when comparing line that does not contain error code
tail -F /logs/COMMON-ERROR.log | while read myline; do... (2 Replies)
Discussion started by: ketanraut
2 Replies
9. Shell Programming and Scripting
Hi,
I'm trying to write a script to go through few folders read some log files and make a list from the data.
the log files are in few steps each step having a final energy but I need to read the last final energy coming after the keyword "hurray".
I have so far accomplished a code like... (16 Replies)
Discussion started by: raymondg
16 Replies
10. Homework & Coursework Questions
Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted!
1. The problem statement, all variables and given/known data:
You will create a shell script that performs the following action:
Use a parameter to pass a file that... (5 Replies)
Discussion started by: Luvs2drnk
5 Replies
LEARN ABOUT NETBSD
zfgrep
ZGREP(1) BSD General Commands Manual ZGREP(1) NAME
zgrep, zegrep, zfgrep -- print lines matching a pattern in gzip-compressed files SYNOPSIS
zgrep [grep-flags] [--] pattern [files ...] zegrep [grep-flags] [--] pattern [file ...] zfgrep [grep-flags] [--] pattern [file ...] DESCRIPTION
zgrep runs grep(1) on files or stdin, if no files argument is given, after decompressing them with zcat(1). The grep-flags and pattern arguments are passed on to grep(1). If an -e flag is found in the grep-flags, zgrep will not look for a pattern argument. zegrep calls egrep(1), while zfgrep calls fgrep(1). EXIT STATUS
In case of missing arguments or missing pattern, 1 will be returned, otherwise 0. SEE ALSO
egrep(1), fgrep(1), grep(1), gzip(1), zcat(1) AUTHORS
Thomas Klausner <wiz@NetBSD.org> BSD
December 28, 2003 BSD