Archiving or removing few data from log file in real time


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Archiving or removing few data from log file in real time
# 1  
Archiving or removing few data from log file in real time

Hi,

I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge size. Please advise as this is bit urgent related to space in the server.
# 2  
May be hard to execute any kind of cleanup with new data added every second.

One theory for cleanup...
determine the line count, assuming each update is on its own line
divide that number in half
use a tail command to copy the 2nd half of the list to a new file
then copy it back to the original filename
# 3  
yeah something like
Code:
var=`expr $(cat filename| wc -l) / 2`
tail -$var filename> newfile
mv newfile filename

This User Gave Thanks to Makarand Dodmis For This Post:
# 4  
It seems cat will take long time to cat the file as it is huge in size now.
var=`expr $(cat filename| wc -l) / 2` is taking long time to execute. I am waiting though

Last edited by Franklin52; 06-27-2014 at 12:26 PM.. Reason: Please use code tags
# 5  
try
Code:
var=`expr $(wc -l < filename) / 2`

This User Gave Thanks to Makarand Dodmis For This Post:
# 6  
Deleting all the data would be easy, but half? Hmm.

What is making this logfile? Many daemons allow you to send a signal to them when you want to change the logfile, which would at least let you deal with the file without it stomping it several times a second.
# 7  
Surely cat filename|wc -l adds a process and therefore considerable extra time. Would wc -l filename not be quicker?

Anyhow, is the file being appended to as in-use all the time or is it separate operations. Consider these two (probably not exactly true, but just for an example)
Code:
for i in 1 2 3 4 5
do
   echo "Hello $i" >> filename
   sleep 5
done

versus
Code:
for i in 1 2 3 4 5
do
   echo "Hello $i"
   sleep 5
done >>filename

In the first, you have five discreet "open-append and close" operations. In the second you have one, so in the gaps between the echo statements, the file remains open. If you delete the data and write the file back, where does the subsequent output go? If you rename the file, then the output follows the old file.

Like Corona688 says, we need to know what is generating the messages. It may be that you have to stop that process whilst you manipulate the file, then restart it if there is no signal you can send to get it to switch logs.




Robin
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #523
Difficulty: Medium
Some programming languages implement strings by defining arrays of characters.
True or False?

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Log all the commands input by user at real time in /var/log/messages

Below is my script to log all the command input by any user to /var/log/messages. But I cant achieve the desired output that i want. PLease see below. function log2syslog { declare COMMAND COMMAND=$(fc -ln -0) logger -p local1.notice -t bash -i -- "$USER:$COMMAND" } trap... (12 Replies)
Discussion started by: invinzin21
12 Replies

2. UNIX for Advanced & Expert Users

How to read a fast written log file at Real time speed?

Hello All, I am building a real time parser for a log file in my application. The log file is continuously written at a very fast pace and gets rolled over every 10 minutes. I have measured the speed and observed that around 1000 lines are written to it every second, each line about 30-40... (7 Replies)
Discussion started by: cool.aquarian
7 Replies

3. Shell Programming and Scripting

[solved] How to see log in real time?

Hi people I have a bash script with a line like this: python example.py >> log & But i can't see anything in the log file while python program is running only if the program ends seems to write the log file. "$ cat log" for example don't show anything until the program ends. Is there... (4 Replies)
Discussion started by: Tieso
4 Replies

4. HP-UX

HP-UX real time audit log writing

Hey all, I have a problem I was hoping to get some help on. So I have my two auditfiles, audfile1 and audfile2 that can be written to, I want to have the text version of them write to an NFS mount that I have set up. So i already know that i can do .secure/etc/audsp audfile1 > //nfsmount/folder/... (5 Replies)
Discussion started by: CleverRiver6
5 Replies

5. Shell Programming and Scripting

Real time log file redirect

Hi all, i would like to write the shell script program, it can monitor the access_log "real time" when the access_log writing the line contain "abcdef" the program will be "COPY" this line into a file named "abcdef.txt", do the same thing if the contain "123456" "COPY" it into a file named... (3 Replies)
Discussion started by: eric_wong_ch
3 Replies

6. Shell Programming and Scripting

Piped open not real-time - How would one handle live data?

When I run "/etc/myApp" I am presented with continuous output, just about once per second. However when I try to get the information in Perl via a piped open, it waits till the end to give me anything... my code: open (OUTPUT,"/etc/myApp |"); while (<OUTPUT>){ print $_; }... (2 Replies)
Discussion started by: jjinno
2 Replies

7. UNIX for Dummies Questions & Answers

log users real time

hi.... how i can configurator a log file on real time....on unix solaris.... thanks a lot.... Best Regards... (3 Replies)
Discussion started by: chanfle
3 Replies

Featured Tech Videos