Sponsored Content
Top Forums UNIX for Advanced & Expert Users Incremental extract from growing log file. Post 302873935 by one2connect on Wednesday 13th of November 2013 01:04:34 PM
Old 11-13-2013
Incremental extract from growing log file.

We have a log file which has 16 million row. We want to read all the lines appended from the last time we read using sed command

Code:
sed -n '<START_LINE>,<LAST_LINE>p' abc.csv

I can store this last line line so I can give replace that with START_LINE in my next read. The problem is wc -l which would give me the total lines in the file which I can store it as LAST_LINE takes time to read this ever growing huge file Is there a faster way to implement this?

Last edited by Scott; 11-13-2013 at 02:11 PM.. Reason: Please use code tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to check a file in UNIX is closed or growing?

We have a third party tool in UNIX to kick off a 'file copy' job based on a file existance. If a specific file exists in an UNIX directory, another process should start copy the file into another system for further processing. The issue is, the copy job is starting as soon as the file exists in... (6 Replies)
Discussion started by: kslakshm
6 Replies

2. UNIX for Dummies Questions & Answers

.osm file growing

my /etc/.osm file is growing rapidly and logging large amounts of activity. Can anyone tell me what this file is for and what types of information is logged in this file. Thanks in advance for your help!! (1 Reply)
Discussion started by: golfs4us
1 Replies

3. UNIX for Dummies Questions & Answers

Find out the maximum growing file in a mount

I need to find the file that is growing in the mount. Say yesterday the utilised space was 95% but today that is 96%. How do i find the file that is growing in size. Have checked the same with du/df options but was not able to find much. Please suggest the best possible option. (3 Replies)
Discussion started by: raman1605
3 Replies

4. HP-UX

how to redirect the growing contents of log file to another file in unix

how to redirect the growing contents of log file to another file in unix (2 Replies)
Discussion started by: megh
2 Replies

5. Shell Programming and Scripting

Incremental log parser

I have log file that is incremented every second and is rather big. I need monitor error in this file. I do not want to parse it all every time. I want parse it first time then remember last position (as byte for example) All consequent parsing I want to start from last saved position to the... (3 Replies)
Discussion started by: Narcom
3 Replies

6. Shell Programming and Scripting

Script for incremental file

Hi , I have incremental file FILE1 which locate in directory /asd/ This directory contains log files whose name entries goes in FILE1 how to make shell scripts for following condition shell script executes on day1 FILE1 entry ----> 1.log ... (3 Replies)
Discussion started by: kaushik02018
3 Replies

7. Solaris

Growing a file system-SVM

Hi gurus Im a newbie in solaris..I need to extend file system space in solaris 10 which is using SVM..I have a file system /pin02 which is 93% full n needs to be extended..only 3.6 gb avail space left..the file system is not mirrored...normal ufs file system only..can u please tel me t... (6 Replies)
Discussion started by: madanmeer
6 Replies

8. UNIX for Dummies Questions & Answers

Find the file which is growing at high speed

Hi, A log file which is growing at high speed, don't know the name of it. How to find the respective file? Many thanks. (2 Replies)
Discussion started by: venkatesht
2 Replies

9. UNIX for Dummies Questions & Answers

How to check if my log file is growing properly?

Hi All, I want to check if one my log file is updating properly, how can I achieve it. The approach I am trying is to get the file size at two different interval and than comparing it eg : $ ls -ltr | tail -1 | awk '{print $5}' 20480 (7 Replies)
Discussion started by: mukulverma2408
7 Replies

10. Shell Programming and Scripting

How to extract start/end times from log file to CSV file?

Hi, I have a log file (log.txt) that which contains lines of date/time. I need to create a script to extract a CSV file (out.csv) that gets all the sequential times (with only 1 minute difference) together by stating the start time and end time of this period. Sample log file (log.txt) ... (7 Replies)
Discussion started by: Mr.Zizo
7 Replies
grablogs.conf(4)						   File Formats 						  grablogs.conf(4)

NAME
grablogs.conf - grablogs configuration for libgrablogs.so of the plugins of gnome-system-log file SYNOPSIS
/usr/lib/gnome-system-log/plugins/grablogs.conf DESCRIPTION
The libgrablogs.so is a plugin for gnome-system-log(1), it colloct the log files from the system as many as possible. grablogs.conf is a configuration file that contains a set of lines mixed with sh(1) syntax codes and individual log files. libgrablogs.so will read the file try to get a log files list for gnome-system-log(1). Users can copy the file into $HOME/.gnome2/gnome-system-log/plugins/`uname -p` to overwrite the system default one. The grablogs.conf file contains the following configuration categories: [configs] Each line under this category is interpreted as a config file of System. The plugin will open the config file and try to find all system paths of the logs. [commands] Each line under this category is interpreted as a shell command and will be execute through a pipe. And each line of the out- put of the command will be interpreted as a log path. [logs] Each line under this category is interpreted as a log path. FILES
/usr/lib/gnome-system-log/plugins/grablogs.conf The system default configuration file for the plugin libgrablogs.so $HOME/.gnome2/gnome-system-log/plugins/`uname -p`/grablogs.conf The user specific configuration file for the plugin libgrablogs.so EMAMPLE
[configs] /etc/syslog.conf [commands] for i in `svcs -aH -o FMRI | grep -v lrc `; do svcprop -p restarter/logfile $i 2>/dev/null || svcprop -q -p restarter/alt_logfile $i 2>/dev/null ; done [logs] /var/log/messages /var/log/secure /var/log/maillog /var/log/cron /var/log/Xorg.0.log /var/log/XFree86.0.log /var/log/auth.log /var/log/cups/error_log SEE ALSO
gnome-system-log(1), pipelog.conf(1) gnome-utils 2.16.0 13 Oct 2006 grablogs.conf(4)
All times are GMT -4. The time now is 11:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy