12-25-2007
Quote:
Hi, bakunin.
I don't know how fast your machine is, but you are dealing with several processes here, so those will definitely take up some time. If I understand this, we want to minimize real time to avoid loss of data. You didn't mention how large the file was. If it's really large, this might not work.
The machine is a LPAR in a IBM p570 with 2 physical CPUs (4 LCPUs). The quantity structure is as follows:
~10k lines per day
~4MB per day
The file is the garbage collector log of a JVM (the machine is running some Websphere 6.1 application servers) and the logfile is in XML format. That means, the lines are not written in constant intervals, but always a bunch of lines (one "paragraph", so to say) at a time. The information units i want to separate. are each starting with a "<af>" tag and ending with with a "</af>" tag.
Quote:
Memory is obviously faster than disk, so I suggest creating a perl script to slurp in the file, perhaps have several subroutines (if you like modularity) to take the place of the seds, etc., and write out the results. Even if you copy the perl lists a few times internally, that's still a real-time savings over disk access.
Not at all! perl is definitely way slower than sed, at about a factor 10. I came to this conclusion when working on my last project, where i had to change database dumps (frighteningly huge files) and replaced the perl programs doing it with sed - that sped up the process greatly.
As i see it the critical part is only between lines 11 and 12 of the code snippet. All the previous operations are working from line 1 up to some predetermined line x of the file and it won't hurt of there come additional lines in during this time.
As an additional requirement i have to preserve the inode of the file, because the process which writes to it (the garbage collector of the JVM) will continue to write into it. This is why i used "cat > ..." instead of "mv ...".
bakunin
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I am using KSH with an OS of AIX Version 5.3.
From my shell script, that will be scheduled thorugh a CRON, I need to execute 2 Oracle stored procedures concurrently. If we execute them one at a time, the total execution time takes 4 hours or more. Since they are not dependent on each other and... (6 Replies)
Discussion started by: multidogzoomom
6 Replies
2. Programming
How can i run a back ground process....
I.e for example by using fork() i need to run seperate 4 background processes..
How can send a process to background
??? (9 Replies)
Discussion started by: ugp
9 Replies
3. Programming
hi
i have a server socket programme which is running in HP/UX system and
then need to add a function about refreshing memory every miniute
because the socket programme is blocked , i have no idea about this
what should i do
thanks (10 Replies)
Discussion started by: benbenpig
10 Replies
4. Shell Programming and Scripting
Hi,
I was hoping someone would be able to help me out. I've got a Python script that I need to run 60 times concurrently (with the number added as an argument each time) via nightly cron. I figured that this would work:
30 1 * * * for i in $(seq 0 59); do $i \&; done
However, it seems to... (4 Replies)
Discussion started by: ckhowe
4 Replies
5. UNIX for Advanced & Expert Users
Hi,
How can I remove the unwanted spaces in the line.
123456 789 ABC DEF. - I wanna remove the sapces in this line, I need the output 123456789ABCDEF.
Pls help me...... (3 Replies)
Discussion started by: sharif
3 Replies
6. Shell Programming and Scripting
I currently run a script over a vpnc tunnel to back-up my data to a remote server. However for a number of reasons the tunnel often collapses. If I manually restore the tunnel then the whole thing can continue, but I want to add into my script a section whereby while the transfer is taking place,... (8 Replies)
Discussion started by: dj_bridges
8 Replies
7. AIX
I have serveral servers that are at AIX 6.1 tl4 sp1 and want to move them to new hardware and upgrade them at the same time. Using NIM and sysback images I want to backup the current server with sysback and restore it and upgrade it to AIX 6.1 tl4 sp6 to the new hardware using my NIM server. My... (4 Replies)
Discussion started by: daveisme
4 Replies
8. UNIX for Dummies Questions & Answers
Hi all,
I have one script. Its job is to get 1 file from dirA and 1 file from dirB as parameters for another python script.
However, there are a lot of files in both dir A and B, so running this scripts really takes a lot of time. So I wonder if there are any ways that I can do this faster,... (6 Replies)
Discussion started by: yoyomano
6 Replies
9. Shell Programming and Scripting
Dear Gurus,
Trimming the alert log files of oracle. I need to write a script retaining last 20 lines of the log files and deleting all the other lines
I tried with tail -20 , Please help me how to delete all the other lines, my log files has grown upt to 2G.
Thanks in Advance
Dave (5 Replies)
Discussion started by: dave234
5 Replies
10. Shell Programming and Scripting
Hi i have a log file
P12345_15728710:DEBUG:Begin
P12345_15728710:DEBUG:Being
P12345_15729310:DEBUG:GetAgen
P12345_15726510:DEBUG:end
i want to trim this file and i want like this
15728710
15728710
15729310
15726510
i tried sed ..but not working..
sed "s/.*P12345__ \(.*\)... (4 Replies)
Discussion started by: navsan420
4 Replies
LEARN ABOUT FREEBSD
logsave
LOGSAVE(8) System Manager's Manual LOGSAVE(8)
NAME
logsave - save the output of a command in a logfile
SYNOPSIS
logsave [ -asv ] logfile cmd_prog [ ... ]
DESCRIPTION
The logsave program will execute cmd_prog with the specified argument(s), and save a copy of its output to logfile. If the containing
directory for logfile does not exist, logsave will accumulate the output in memory until it can be written out. A copy of the output will
also be written to standard output.
If cmd_prog is a single hyphen ('-'), then instead of executing a program, logsave will take its input from standard input and save it in
logfile
logsave is useful for saving the output of initial boot scripts until the /var partition is mounted, so the output can be written to
/var/log.
OPTIONS
-a This option will cause the output to be appended to logfile, instead of replacing its current contents.
-s This option will cause logsave to skip writing to the log file text which is bracketed with a control-A (ASCII 001 or Start of
Header) and control-B (ASCII 002 or Start of Text). This allows progress bar information to be visible to the user on the console,
while not being written to the log file.
-v This option will make logsave to be more verbose in its output to the user.
AUTHOR
Theodore Ts'o (tytso@mit.edu)
SEE ALSO
fsck(8)
E2fsprogs version 1.44.1 March 2018 LOGSAVE(8)