Thanks RudiC for bringing up the order of the timestamp. Yes it does matter. Please help.
Current Setup:
Logs were rotated every 4hrs and they were compressed. If the logs exceed 10gb, it will be compressed (.Z), if not script will just zipped the logs (.gz). So archive directory may sometime contains *.gz and *.Z files, and there are times that it only contains *.gz files
Goal:
Help on Loops to Grep logs per domain, in a single text file, regardless if logs are in *gz or *Z.
Not sure I understand your problem nor your code snippet.
Let's assume you have a directory $dir_arch/logs containing .gz and .Z files, all of which you want to search for 5 items in domainlist. The order of the resulting output does not seem to matter, does it? So sth. like this should do the job for you:
Code:
domainlist="4prd|5prd|6prd|7prd|8prd"
# ^--- "or" in EREs
gzgrep -E "$domainlist" $dir_arch/logs/*.gz >> $dir_arch/logs/$domain_$myhost.txt
# ^--- tell grep to use ERE ^--- run this on all gzipped files
zcat $dir_arch/logs/*.Z | grep -E "$domainlist" >> $dir_arch/logs/$domain_$myhost.txt
# ^--- run this on all compressed files
gzip -f $dir_arch/logs/$domain_$myhost.txt
Let me paraphrase your problem: You have in one single directory $dir_arch/logs a bunch of zipped/compressed logfiles, which you want to scan line by line for a set of domains, outputting each line to the respective $domain_$myhost.txt file, concatenating all logfiles' entries into single respective domain.txt file.
I still don't see how the TIMESTAMP you refer to is reflected in your code snippet.
I rate it unwise to gzgrep/zcat the files several times for the different domains for performance reasons. (g)unzip once to a tmp dir, order according to timestamp required, and run sth like an awk script over all files that distributes the lines to the respective files.
Actually there are only 6 log files which covers the whole day of logs( sometimes a combination of *gz and *Z files or all *gz files or all *Z files depending on the log size upon rotation).
Logs will be huge when (g)unzip, im afraid of having an FS issue by doing so i decided to use zcat | grep and gzgrep. Im sorry but im such a newbie and not so familiar with
"awk" command as you recommended.
Code:
Example logs on the archive:
/archive/2013-Jan-10$
364M Jan 10 00:02 log.2013-Jan-10.00-00-50.Z
75M Jan 10 04:00 log.2013-Jan-10.04-00-23.gz
98M Jan 10 08:01 log.2013-Jan-10.08-00-32.gz
174M Jan 10 12:02 log.2013-Jan-10.12-01-08.gz
176M Jan 10 16:02 log.2013-Jan-10.16-01-23.gz
354M Jan 10 20:02 log.2013-Jan-10.20-01-23.Z
/archive/2013-Jan-11$
373M Jan 11 00:02 log.2013-Jan-11.00-00-53.Z
83M Jan 11 04:01 log.2013-Jan-11.04-00-26.gz
100M Jan 11 08:02 log.2013-Jan-11.08-00-31.gz
344M Jan 11 12:02 log.2013-Jan-11.12-01-07.Z
340M Jan 11 16:02 log.2013-Jan-11.16-01-23.Z
362M Jan 11 20:02 log.2013-Jan-11.20-01-27.Z
/archive/2013-Jan-18$
371M Jan 18 00:02 log.2013-Jan-18.00-00-52.Z
91M Jan 18 04:01 log.2013-Jan-18.04-00-27.gz
119M Jan 18 08:01 log.2013-Jan-18.08-00-31.gz
154M Jan 18 12:02 log.2013-Jan-18.12-01-18.gz
87M Jan 18 16:02 log.2013-Jan-18.16-01-40.gz
105M Jan 18 20:02 log.2013-Jan-18.20-01-07.gz
The scripts will look like this:
Code:
date=`date +"%Y-%h-%d"`
domainlist="4prd 5prd 6prd 7prd 8prd"
for domain in $domainlist
do
if [ -f /archive/$date/* ] ; then
# ^--- check if logs are present in the archive dir
ls -l /archive/$date/* | grep Z
# ^--- check for *Z files
if [ $? -eq 0 ] ; then
zcat /archive/$date/*Z | grep "$domain" >> /archive/$date/$domain.txt
# ^--- grep "domain" for *Z files
gzgrep "$domain" /archive/$date/*gz >> /archive/$date/$domain.txt
# ^--- grep "domain" for *gz files
gzip -f /archive/$date/$domain.txt
# ^--- archive has *Z files, perform the ff commands above, but the TIMESTAMP on the output file may sometimes won't be in order.
else
gzgrep "$domain" /archive/$date/*gz >> /archive/$date/$domain.txt
gzip -f /archive/$date/$domain.txt
# ^--- archive has no *Z files
fi
else
echo ">>Logs not found, kindly check archive directory.."
fi
done
Kindly let me know if you have better solution for this. I would really appreciate your response. Thanks!
No matter what you do, disk space will come into game. zcat and gzgrep will need space to uncompress, probably temporary files somewhere on disk, or swap file space. So - find a disk that can accomodate your huge files and uncompress there to, if possible maintining file time stamps. Then run sth like
If * does not supply your files in the correct order, try to rename the files so they show up as needed in e.g. ls. The pipes in domainlist are mandatory for the regex to work!
Hi 2 all,
i have had AIX 7.2
:/# /usr/IBMAHS/bin/apachectl -v
Server version: Apache/2.4.12 (Unix)
Server built: May 25 2015 04:58:27
:/#:/# /usr/IBMAHS/bin/apachectl -M
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
mpm_worker_module (static)
... (3 Replies)
Hello.
System : opensuse leap 42.3
I have a bash script that build a text file.
I would like the last command doing :
print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt
where :
print_cmd ::= some printing... (1 Reply)
How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address
and column 3 contains “cc” e-mail address to include with same email.
Sample input file, email.txt
Below is an sample code where... (2 Replies)
Hello,
I am somewhat new to Linux/Unix. I am currently working on a shell script that is suppose to cat a file, grep the same file for a certain line, if that line is found save the file in a different location, else remove the file.
This is a rough example of what I want.
$Dating = False... (13 Replies)
Hi Guys,
I'm sorry but I can't find answer for this, what is the meaning of -s option in "if" statement on unix scipting. Please see sample below:
opath=/home/output
for i in N1 N2 N3 N4
do
echo $i
if
then
grep $i $opath/N5_CRAI > $opath/N5_$i.crai
chmod 777 $opath/N5_$i.crai
... (7 Replies)
Hi,
I have line in input file as below:
3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL
My expected output for line in the file must be :
"1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL"
Can someone... (7 Replies)
Hi
I am trying to figure out what the following line does, I work in ksh88:
] && LIST="$big $LIST"
Not sure what "-a" means in that case.
Thanks a lot for any advice -A (1 Reply)
Dear experts,
I'm trying to write a script to calculate the usage of Log Archive in a directory, so if it gets to a point where the directory size is 60%, then send out an FYI.. email. So if then it reaches to 80%, move the logs from that directory.
I have written the script as follow but... (10 Replies)
Hi Friends,
Can any of you explain me about the below line of code?
mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`
Im not able to understand, what exactly it is doing :confused:
Any help would be useful for me.
Lokesha (4 Replies)