Trimming files concurrently Post: 302152918

Sponsored Content

Top Forums Shell Programming and Scripting Trimming files concurrently Post 302152918 by bakunin on Saturday 22nd of December 2007 07:00:14 AM

12-22-2007

Registered User

Trimming files concurrently

I have a file which is written by an ongoing process (actually it is a logfile). I want to trim this logfile with a script and save the trimmed portion to another file or display it to <stdout> (it gets picked up there by another process).

Fortunately my logfile has a known format (XML) and i can find the end of a logical paragraph (a certain closing tag) so i know up to which line i have to trim to. The problem i am facing is that the logfile is written more or less permanently (at the rate of ~10k lines per day) and i want to reduce the portion which might eventually be lost during the trimming to the absolute minimum. I am well aware that i cannot achieve the optimum of losing no output at all without job control (which i don't have) but i want to get as close as it is possible. This is what i have come up so far ("</af>" is the closing tag i am anchoring at):

Code:

                                                 # get nr of lines in log
iNumLines=$(wc -l $fLog | sed 's/^ *//' | cut -d' ' -f1)

chActLine=""
(( iNumLines += 1 ))
while [ "$chActLine" != "</af>" ] ; do
     (( iNumLines -= 1 ))
     chActLine="$(sed -n "${iNumLines}p" $fLog)"
done

sed -n "1,${iNumLines}p" $fLog                   # output to <stdout>
sed "1,${iNumLines}d" $fLog > $chTmpDir/log      # remove printed lines
cat $chTmpDir/log > $fLog                        # overwrite with shortened
                                                 # version

While this is generally doing what i want i'd like to ask if there might be a way to further reduce the risk of lost lines as i perceive to be there between line 11 and 12 of the script snippet.

Any suggestions will be welcome.

bakunin

bakunin

View Public Profile for bakunin

Find all posts by bakunin

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Executing multiple Oracle procedures concurrently

I am using KSH with an OS of AIX Version 5.3. From my shell script, that will be scheduled thorugh a CRON, I need to execute 2 Oracle stored procedures concurrently. If we execute them one at a time, the total execution time takes 4 hours or more. Since they are not dependent on each other and...

2. Programming

Run 4-processes concurrently

How can i run a back ground process.... I.e for example by using fork() i need to run seperate 4 background processes.. How can send a process to background ???

3. Programming

how to run socket programme and file management concurrently

hi i have a server socket programme which is running in HP/UX system and then need to add a function about refreshing memory every miniute because the socket programme is blocked , i have no idea about this what should i do thanks

4. Shell Programming and Scripting

Running same script multiple times concurrently...

Hi, I was hoping someone would be able to help me out. I've got a Python script that I need to run 60 times concurrently (with the number added as an argument each time) via nightly cron. I figured that this would work: 30 1 * * * for i in $(seq 0 59); do $i \&; done However, it seems to...

5. UNIX for Advanced & Expert Users

Trimming the spaces

Hi, How can I remove the unwanted spaces in the line. 123456 789 ABC DEF. - I wanna remove the sapces in this line, I need the output 123456789ABCDEF. Pls help me......

6. Shell Programming and Scripting

Running function or command concurrently in a bash script

I currently run a script over a vpnc tunnel to back-up my data to a remote server. However for a number of reasons the tunnel often collapses. If I manually restore the tunnel then the whole thing can continue, but I want to add into my script a section whereby while the transfer is taking place,...

7. AIX

Restore and upgrade concurrently

I have serveral servers that are at AIX 6.1 tl4 sp1 and want to move them to new hardware and upgrade them at the same time. Using NIM and sysback images I want to backup the current server with sysback and restore it and upgrade it to AIX 6.1 tl4 sp6 to the new hardware using my NIM server. My...

8. UNIX for Dummies Questions & Answers

How to run script concurrently

Hi all, I have one script. Its job is to get 1 file from dirA and 1 file from dirB as parameters for another python script. However, there are a lot of files in both dir A and B, so running this scripts really takes a lot of time. So I wonder if there are any ways that I can do this faster,...

9. Shell Programming and Scripting

script for Trimming the log files

Dear Gurus, Trimming the alert log files of oracle. I need to write a script retaining last 20 lines of the log files and deleting all the other lines I tried with tail -20 , Please help me how to delete all the other lines, my log files has grown upt to 2G. Thanks in Advance Dave

10. Shell Programming and Scripting

Trimming in between the words

Hi i have a log file P12345_15728710:DEBUG:Begin P12345_15728710:DEBUG:Being P12345_15729310:DEBUG:GetAgen P12345_15726510:DEBUG:end i want to trim this file and i want like this 15728710 15728710 15729310 15726510 i tried sed ..but not working.. sed "s/.*P12345__ $.*$...

LEARN ABOUT LINUX

logsave

LOGSAVE(8)						      System Manager's Manual							LOGSAVE(8)

NAME

       logsave - save the output of a command in a logfile

SYNOPSIS

       logsave [ -asv ] logfile cmd_prog [ ... ]

DESCRIPTION

       The  logsave  program  will  execute  cmd_prog with the specified argument(s), and save a copy of its output to logfile.  If the containing
       directory for logfile does not exist, logsave will accumulate the output in memory until it can be written out.	A copy of the output  will
       also be written to standard output.

       If  cmd_prog  is a single hyphen ('-'), then instead of executing a program, logsave will take its input from standard input and save it in
       logfile

       logsave is useful for saving the output of initial boot scripts until the /var partition is mounted,  so  the  output  can  be  written	to
       /var/log.

OPTIONS

       -a     This option will cause the output to be appended to logfile, instead of replacing its current contents.

       -s     This  option  will  cause  logsave  to  skip writing to the log file text which is bracketed with a control-A (ASCII 001 or Start of
	      Header) and control-B (ASCII 002 or Start of Text).  This allows progress bar information to be visible to the user on the  console,
	      while not being written to the log file.

       -v     This option will make logsave to be more verbose in its output to the user.

AUTHOR

       Theodore Ts'o (tytso@mit.edu)

SEE ALSO

       fsck(8)

E2fsprogs version 1.41.14					   December 2010							LOGSAVE(8)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Executing multiple Oracle procedures concurrently

Discussion started by: multidogzoomom

2. Programming

Run 4-processes concurrently

Discussion started by: ugp

3. Programming

how to run socket programme and file management concurrently

Discussion started by: benbenpig

4. Shell Programming and Scripting

Running same script multiple times concurrently...

Discussion started by: ckhowe