This is my actual script this processing my Apache logs spiting depends on CNAME and SCODE. what happening now copying all file from Raw_dir to Processing_dir and then splitting the logs depending on scode value after that zipping and taking backup of which is processed.
Required format is: I need to take one file from Raw_dir copy to processing_dir then split and zip and moved to processed_dir.
Then take 2nd file do the above process up to available files(do not process before processed from Raw_dir)
Code:
#!/bin/sh
echo ======= LogManageri start ==========
#This directory is getting the raw data from remote server
Raw_data=/opt/ftplogs
# This directory is ready for process the data
Processing_dir=/opt/processing_dir
# This directory is prcoessed files and taking backup
Processed_dir=/opt/processed_dir
# This directory spliting the files like access and error logs
split_dir=/opt/split_dir
# Copying Raw data to Processing directory
echo starting copying files from $Raw_Data to $Processing_dir
cp -p $Raw_data/*.gz $Processing_dir/
echo done copying raw files.
# Decompress .gz files from $Process_dir
echo starting unziping files in $Process_dir
gunzip $Processing_dir/*.gz
echo done unziping files in $Process_dir
# This for loops gives year,month and day from file name
for file in $Processing_dir/*.log
do
#Name of the input file
name=$(basename "$file" .log)
echo this $name
#getting year from file name
year=${name:9:4}
echo $year
#getting month from file name
month=${name:13:2}
echo $month
#getting day from file name
day=${name:15:2}
echo $day
# This while loop reading the file each and every line and spliting the Respective $cname and $code
echo start reading files
while read -r line
do
# cname is reading every line from all files
cname=$(echo ${line} | awk '{split($11,c,"/"); print c[3]}')
echo $cname
# scode is reading every line from all files
scode=$(echo ${line} | awk -F"[ ]" '{print $13}')
echo $scode
# creating directory if not exists
[[ ! -d "$split_dir/$cname/$year/$month/$day" ]] && mkdir -p "$split_dir/$cname/$year/$month/$day"
# this loop not processing starting with # line
case $line in
(\#*) continue
esac
# If condition scode less than 399 logs goes to access log
if [ $scode -le 399 ]
then
echo ${line} >> $split_dir/$cname/$year/$month/$day/${name}_${cname}_access.log
# If condition scode greater than 400 logs goes to error log
else
# [[ ( "${scode}"-ge"400" )]]
echo ${line} >> $split_dir/$cname/$year/$month/$day/${name}_${cname}_error.log
fi
done < $file
done
echo files reading done
# after successfull splitting the the logs gzip .log file
echo starting zip logs
gzip $Processing_dir/*.log
echo compression done
# gzip file moves to $Processed directory
mv $Processing_dir/*.gz $Processed_dir
echo moved file to $Processed_dir
Please suggest to me how do i change in my code.
Last edited by Franklin52; 06-25-2014 at 06:51 AM..
Reason: Please use code tags
I have a shell script that runs all the time looking for a certain type of file and then it processes the file through a series of other scripts. The script is watching a directory that has files uploaded to it via SFTP. It already checks the size of the file to make sure that it is not still... (3 Replies)
Hi All,
I am very new to Shell scripting...
I got a requirement.
I will have few text files(data files) in a particular directory. they will be with .txt extension. With same name, but with a different extension control files also will be there. For example, Sample_20081001.txt is the data... (4 Replies)
Hello,
say suppose i am processing an file emp.dat the field of which are
deptno empno empname etc
now say suppose i want to change the file to emp.lst then how can i do it? Here i what i attempted but in vain
BEGIN{
system("sort emp.dat > emp.lst")
FILENAME="emp.lst"
}
{
print... (2 Replies)
Hello,
I am trying to find a solution which could be simple rather than coding unnecessary code. Scenario I have is :-
I have a line sequential text file and while reading the file I need to call some kind of program which does data transfer, and read next so on... The issue I have is I have... (4 Replies)
Hi I am working on the script parsing specific message "TEST" from multiple file. The log file name looks like:
N3.2009-11-26-03-05-02.console.log.tar.gz
N4.2009-11-29-00-25-03.console.log.tar.gz
N6.2009-12-01-10-05-02.console.log.tar.gz
I am using the following command:
zgrep -a --text... (1 Reply)
Hi.
I am new to scripting and could really do with some advice on the best way to put a script together. Here is the scenario I am working to;
- i will get files via ftp to a tmp directory on the server
- all files will have a unique file name but with the same extension (.USM)
- for each... (5 Replies)
Hi All,
The script below read the path and searches for the directories/subdirectories and for the files. If files are found in the sub directories then read the content of the all files and put the content in csv(comma delimted) format and the call the write to xml function to write the
std... (1 Reply)
Hello,
I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this :
This is the output of ls command : I stored the output in a file filelist
1.1M... (5 Replies)
I have one input file ABC.txt and one output DEF.txt. After the ABC is processed and created output, I want to rename ABC.txt to ABC.orig and DEF to ABC.txt. Currently when I am doing this, it does not process the input file as it cannot read and write to the same file. How can I achieve this?
... (12 Replies)
I have a fixed width file coming from source system. The total characters on each record is 786.
I am getting records in the file with less than and greater than this number and the process is failing bcs of this length of the record
Is there a way to spit out the bad records (length <>786)... (5 Replies)
Discussion started by: dsravanam
5 Replies
LEARN ABOUT REDHAT
rotatelogs
rotatelogs(8) System Manager's Manual rotatelogs(8)NAME
rotatelogs - rotate Apache logs without having to kill the server
SYNOPSIS
rotatelogs logfile rotationtime [offset]
DESCRIPTION
rotatelogs is a simple program for use in conjunction with Apache's piped logfile feature, which can be used like this:
CustomLog "|bin/rotatelogs /var/logs/logfile 86400" common
This creates the files /var/logs/logfile.nnnn where nnnn is the system time at which the log nominally starts (this time will always be a
multiple of the rotation time, so you can synchronize cron scripts with it). At the end of each rotation time (here after 24 hours) a new
log is started.
OPTIONS
logfile
The path plus basename of the logfile. If logfile includes any Otherwise, the suffix .nnnnnnnnnn is automatically added and is the
time in seconds. Both formats compute the start time from the beginning of the current period.
rotationtime
The rotation time in seconds.
offset The number of minutes offset from UTC. If omitted, zero is assumed and UTC is used. For example, to use local time in the zone UTC
-5 hours, specify a value of -300 for this argument.
PORTABILITY
The following logfile format string substitutions should be supported by all strftime(3) implementations, see the strftime(3) man page for
library-specific extensions.
%A full weekday name (localized)
%a 3-character weekday name (localized)
%B full month name (localized)
%b 3-character month name (localized)
%c date and time (localized)
%d 2-digit day of month
%H 2-digit hour (24 hour clock)
%I 2-digit hour (12 hour clock)
%j 3-digit day of year
%M 2-digit minute
%m 2-digit month
%p am/pm of 12 hour clock (localized)
%S 2-digit second
%U 2-digit week of year (Sunday first day of week)
%W 2-digit week of year (Monday first day of week)
%w 1-digit weekday (Sunday first day of week)
%X time (localized)
%x date (localized)
%Y 4-digit year
%y 2-digit year
%Z time zone name
%% literal `%'
SEE ALSO httpd(8)
March 2001 rotatelogs(8)