Sponsored Content
Top Forums Shell Programming and Scripting While loop hangs in function running in background Post 302867991 by kanagalamurali on Friday 25th of October 2013 01:32:58 PM
Old 10-25-2013
While loop hangs in function running in background

Hello Everyone,

I am writing a shell script to fetch log files from remote servers within a time range. It copies log files to local server, grep for date and then compares the time stamp of each log entry with the once specified.
Below is the code.
Code:
# Get log and Parsing function

GetAndParse() {

    echo -e "\nWorking on server $host"

    # Looping through the log paths given in the argument.
    for logpath in $LOG_PATH
    do
        echo "Fetching log files from $logpath"

        # Looping through the log file names provided in the arguement.
		for logfile in $FILE_NAMES
        do
            # Finding the log file size on remote server
			logfilesize=$(ssh -n $host du -skLc $logpath/$logfile 2> /dev/null|awk '{print $1}'|tail -1)

			# Comapring the log file size with limit LOG_FILE_SIZE	            
			if [ ! -z $logfilesize ] && [ $logfilesize -le $LOG_FILE_SIZE ] && [ $logfilesize -gt 0 ]
			then
				# If the log file size is under limit copt it to local WORK_DIR
				scp $host:$logpath/$logfile $WORK_DIR/temp
				LOCAL_TEMP_FILE="$WORK_DIR"/temp/"$host"_"$logfile"_"$RANDOM".temp
				# Grep for dates in log entries
				cat $WORK_DIR/temp/$logfile| grep -E  "$START_TIME|$END_TIME" > $LOCAL_TEMP_FILE && rm $WORK_DIR/temp/$logfile
	

				while read line
				do
					line_date1=$(echo $line|grep -o "[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]")
					line_date=$(echo $line_date1|tr -s " "|cut -d " " -f1,2)
					UNIX_DATE=$(date -d"$line_date" +%s)

					if [ $UNIX_DATE -gt $START_UNIX_DATE -a $UNIX_DATE -lt $END_UNIX_DATE ]
					then
						echo "$host $line" >> $OUTPUT_UNSORTED_LOG
					fi

				done < $LOCAL_TEMP_FILE

			else
				echo -e "\e[0;33mNo file with name $logfile below $LOG_FILE_SIZE KB found under $logpath on $host\e[0m"
			fi

		done
        
    done

}


I call the above function as shown below
Code:
for host in ${ARR_HOSTS[@]}
do
    GetAndParse &
done


When i run this, the while loop which is reading lines one by one in log file is not terminating once it reaches EOF.

I have tried running it in foreground (removing '&' while calling function) which works perfect.

I want to run it in background because there are 20 servers i need to ssh, scp and read log files which are more than 10 MB each.

Please suggest me if there are any changes needed in this code.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

subshell & background function

Hello all, Can someone explain to me the advantage between using subshell over a function call in scripts? To me these are the same. Am I wrong to think this? (4 Replies)
Discussion started by: larry
4 Replies

2. Programming

want to run a function in background

consider the given prg. main() { ..... function1(); /* to write into a file or log */ printf(" "); ..... } when the control reaches function1(), it should get executed in the background.At the same time main's printf(" ") statement should also get executed.i.e... (5 Replies)
Discussion started by: bankpro
5 Replies

3. Shell Programming and Scripting

running in background

i have a script called server.sh. how to run this script in backgroung using nohup command (5 Replies)
Discussion started by: ali560045
5 Replies

4. Shell Programming and Scripting

A question about the PID of a background function

Dear all, I'm writing a KornShell script that calls inside it a function in background mode #!/bin/ksh function myfunction { . . .} myfunction |& . . . How can I capture the PID of the function myfunction that runs in background? Thanks in advance :) (2 Replies)
Discussion started by: dariyoosh
2 Replies

5. Shell Programming and Scripting

kill PID running in background in for loop

Guys, can you help me in killing the process which is running in back ground under for loop I am not able to find the PID using ps -afx|grep <word in command I entered> (1 Reply)
Discussion started by: mohan_xunil
1 Replies

6. Shell Programming and Scripting

Find hangs with automountd daemon running

Hi, I am trying to find files with specific name using find / -name core -print.. this command is hanging and never completes as it is searching for all the automount filesystems ..i cant eliminate using "! -fstype nfs" as this is not a nfs filesystem pls let me know if anyone know how to... (2 Replies)
Discussion started by: to_bsr
2 Replies

7. Programming

C Sleep function hangs @ __kernel_vsyscall ()

This is the gdb backtrace. ^C Program received signal SIGINT, Interrupt. 0xffffe424 in __kernel_vsyscall () (gdb) bt #0 0xffffe424 in __kernel_vsyscall () #1 0xb7e56a70 in __nanosleep_nocancel () from /lib/libc.so.6 #2 0xb7e568bb in __sleep (seconds=0) at sleep.c:138 #3 0x080496d5 in... (6 Replies)
Discussion started by: dragonpoint
6 Replies

8. Linux

YUM HANGS AT RUNNING TRANSACTION TEST

Hi When i trying to update using yum, hangs at Running transaction test. # yum update yum Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * addons: ftp.oss.eznetsols.org * extras: ftp.oss.eznetsols.org * base: ftp.oss.eznetsols.org * updates:... (2 Replies)
Discussion started by: anil8103
2 Replies

9. Shell Programming and Scripting

Running in background

Hello, I was trying to make some processes to run at background and went to a problem. First I tried just to loop in one line something like this: for i in {1..10}; do echo 'hello world' &; done; but it pops a syntax error, so I tried several ways to fix it but wasn't able to understand... (2 Replies)
Discussion started by: Rash
2 Replies

10. Shell Programming and Scripting

Execute a function in background and then suspend it

Here is some back ground on the script. The script is to poll an arbitrary number of DB's. To do this I am creating a function that takes the file_path to the DB and the min poll interval as arguments. The function will be called for each DB and then ran in the background. The function I was... (6 Replies)
Discussion started by: ryandavison
6 Replies
ROTATELOGS(8)							    rotatelogs							     ROTATELOGS(8)

NAME
rotatelogs - Piped logging program to rotate Apache logs SYNOPSIS
rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -D ] [ -t ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotation- time|filesize(B|K|M|G) [ offset ] SUMMARY
rotatelogs is a simple program for use in conjunction with Apache's piped logfile feature. It supports rotation based on a time interval or maximum size of the log. OPTIONS
-l Causes the use of local time rather than GMT as the base for the interval or for strftime(3) formatting with size-based rotation. -L linkname Causes a hard link to be made from the current logfile to the specified link name. This can be used to watch the log continuously across rotations using a command like tail -F linkname. -p program If given, rotatelogs will execute the specified program every time a new log file is opened. The filename of the newly opened file is passed as the first argument to the program. If executing after a rotation, the old log file is passed as the second argument. rotatelogs does not wait for the specified program to terminate before continuing to operate, and will not log any error code returned on termination. The spawned program uses the same stdin, stdout, and stderr as rotatelogs itself, and also inherits the environment. -f Causes the logfile to be opened immediately, as soon as rotatelogs starts, instead of waiting for the first logfile entry to be read (for non-busy sites, there may be a substantial delay between when the server is started and when the first request is handled, meaning that the associated logfile does not "exist" until then, which causes problems from some automated logging tools) -D Creates the parent directories of the path that the log file will be placed in if they do not already exist. This allows strftime(3) formatting to be used in the path and not just the filename. -t Causes the logfile to be truncated instead of rotated. This is useful when a log is processed in real time by a command like tail, and there is no need for archived data. No suffix will be added to the filename, however format strings containing '%' characters will be respected. -v Produce verbose output on STDERR. The output contains the result of the configuration parsing, and all file open and close actions. -e Echo logs through to stdout. Useful when logs need to be further processed in real time by a further tool in the chain. -c Create log file for each interval, even if empty. -n number-of-files Use a circular list of filenames without timestamps. With -n 3, the series of log files opened would be "logfile", "logfile.1", "logfile.2", then overwriting "logfile". Available in 2.4.5 and later. logfile rotationtime The time between log file rotations in seconds. The rotation occurs at the beginning of this interval. For example, if the rotation time is 3600, the log file will be rotated at the beginning of every hour; if the rotation time is 86400, the log file will be rotated every night at midnight. (If no data is logged during an interval, no file will be created.) filesize(B|K|M|G) The maximum file size in followed by exactly one of the letters B (Bytes), K (KBytes), M (MBytes) or G (GBytes). .PP When time and size are specified, the size must be given after the time. Rotation will occur whenever either time or size limits are reached. offset The number of minutes offset from UTC. If omitted, zero is assumed and UTC is used. For example, to use local time in the zone UTC -5 hours, specify a value of -300 for this argument. In most cases, -l should be used instead of specifying an offset. EXAMPLES
CustomLog "|bin/rotatelogs /var/log/logfile 86400" common This creates the files /var/log/logfile.nnnn where nnnn is the system time at which the log nominally starts (this time will always be a multiple of the rotation time, so you can synchronize cron scripts with it). At the end of each rotation time (here after 24 hours) a new log is started. CustomLog "|bin/rotatelogs -l /var/log/logfile.%Y.%m.%d 86400" common This creates the files /var/log/logfile.yyyy.mm.dd where yyyy is the year, mm is the month, and dd is the day of the month. Logging will switch to a new file every day at midnight, local time. CustomLog "|bin/rotatelogs /var/log/logfile 5M" common This configuration will rotate the logfile whenever it reaches a size of 5 megabytes. ErrorLog "|bin/rotatelogs /var/log/errorlog.%Y-%m-%d-%H_%M_%S 5M" This configuration will rotate the error logfile whenever it reaches a size of 5 megabytes, and the suffix to the logfile name will be cre- ated of the form errorlog.YYYY-mm-dd-HH_MM_SS. CustomLog "|bin/rotatelogs -t /var/log/logfile 86400" common This creates the file /var/log/logfile, truncating the file at startup and then truncating the file once per day. It is expected in this scenario that a separate process (such as tail) would process the file in real time. PORTABILITY
The following logfile format string substitutions should be supported by all strftime(3) implementations, see the strftime(3) man page for library-specific extensions. o %A - full weekday name (localized) o %a - 3-character weekday name (localized) o %B - full month name (localized) o %b - 3-character month name (localized) o %c - date and time (localized) o %d - 2-digit day of month o %H - 2-digit hour (24 hour clock) o %I - 2-digit hour (12 hour clock) o %j - 3-digit day of year o %M - 2-digit minute o %m - 2-digit month o %p - am/pm of 12 hour clock (localized) o %S - 2-digit second o %U - 2-digit week of year (Sunday first day of week) o %W - 2-digit week of year (Monday first day of week) o %w - 1-digit weekday (Sunday first day of week) o %X - time (localized) o %x - date (localized) o %Y - 4-digit year o %y - 2-digit year o %Z - time zone name o %% - literal `%' Apache HTTP Server 2018-07-06 ROTATELOGS(8)
All times are GMT -4. The time now is 08:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy