Sponsored Content
Full Discussion: Trimming files concurrently
Top Forums Shell Programming and Scripting Trimming files concurrently Post 302153621 by rikxik on Tuesday 25th of December 2007 11:34:35 PM
Old 12-26-2007
Interesting problem. I think it will be quite difficult to handle additional lines in such a fast updating file using sed - after all, sed effectively stores the lines in your file in a buffer and operates on this using pattern & hold space. How can it keep track of new stuff coming in?

You will have to work with something which can seek to the point till which you want to archive & remove that part - DIRECTLY on the ever-changing logfile.

What you can do is reduce the window between these two:

Code:
sed -n "1,${iNumLines}p" $fLog                   # output to <stdout>
sed "1,${iNumLines}d" $fLog > $chTmpDir/log      # remove printed lines

And do it in one shot:

Code:
$ cat data
     1  file:10:no:1011
     2  file:10:file:1011
     3  data:10:say:1011
     4  data:10:data:1011
     5  file:10:file:1011
     6  file:10:file:1011
     7  file:10:file:1011
     8  file:10:file:1011
     9  file:10:file:1011
    10  file:10:file:1011
    11  file:10:file:1011
    12  file:10:file:1011
    13  file:10:file:1011
    14  file:10:file:1011
    15  file:10:file:1011
    16  file:10:file:1011
    17  file:10:file:1011
    18  file:10:file:1011
    19  file:10:file:1011
    20  file:10:file:1011
    21  data:10:say:1011
$ cat sedscr
#!/usr/bin/ksh

iNumLines=10
sed -n "
        # Get the lines to be archived and put on stdout
        1,$iNumLines p
        # Write the rest of (trimmed) data to temporary file. This file can be used to overwrite data.
        $((iNumLines+1)),\$ w data.trimmed
" data
$ sedscr
     1  file:10:no:1011
     2  file:10:file:1011
     3  data:10:say:1011
     4  data:10:data:1011
     5  file:10:file:1011
     6  file:10:file:1011
     7  file:10:file:1011
     8  file:10:file:1011
     9  file:10:file:1011
    10  file:10:file:1011
$ cat data.trimmed
    11  file:10:file:1011
    12  file:10:file:1011
    13  file:10:file:1011
    14  file:10:file:1011
    15  file:10:file:1011
    16  file:10:file:1011
    17  file:10:file:1011
    18  file:10:file:1011
    19  file:10:file:1011
    20  file:10:file:1011
    21  data:10:say:1011

I toyed with the idea of writing directly to data instead of data.trimmed but that obviously doesn't help since the additional lines which came in after sed loaded the file in its buffer will be lost. Basically you can't use sed, ed etc. which operate on a "copy" of the file.

HTH
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Executing multiple Oracle procedures concurrently

I am using KSH with an OS of AIX Version 5.3. From my shell script, that will be scheduled thorugh a CRON, I need to execute 2 Oracle stored procedures concurrently. If we execute them one at a time, the total execution time takes 4 hours or more. Since they are not dependent on each other and... (6 Replies)
Discussion started by: multidogzoomom
6 Replies

2. Programming

Run 4-processes concurrently

How can i run a back ground process.... I.e for example by using fork() i need to run seperate 4 background processes.. How can send a process to background ??? (9 Replies)
Discussion started by: ugp
9 Replies

3. Programming

how to run socket programme and file management concurrently

hi i have a server socket programme which is running in HP/UX system and then need to add a function about refreshing memory every miniute because the socket programme is blocked , i have no idea about this what should i do thanks (10 Replies)
Discussion started by: benbenpig
10 Replies

4. Shell Programming and Scripting

Running same script multiple times concurrently...

Hi, I was hoping someone would be able to help me out. I've got a Python script that I need to run 60 times concurrently (with the number added as an argument each time) via nightly cron. I figured that this would work: 30 1 * * * for i in $(seq 0 59); do $i \&; done However, it seems to... (4 Replies)
Discussion started by: ckhowe
4 Replies

5. UNIX for Advanced & Expert Users

Trimming the spaces

Hi, How can I remove the unwanted spaces in the line. 123456 789 ABC DEF. - I wanna remove the sapces in this line, I need the output 123456789ABCDEF. Pls help me...... (3 Replies)
Discussion started by: sharif
3 Replies

6. Shell Programming and Scripting

Running function or command concurrently in a bash script

I currently run a script over a vpnc tunnel to back-up my data to a remote server. However for a number of reasons the tunnel often collapses. If I manually restore the tunnel then the whole thing can continue, but I want to add into my script a section whereby while the transfer is taking place,... (8 Replies)
Discussion started by: dj_bridges
8 Replies

7. AIX

Restore and upgrade concurrently

I have serveral servers that are at AIX 6.1 tl4 sp1 and want to move them to new hardware and upgrade them at the same time. Using NIM and sysback images I want to backup the current server with sysback and restore it and upgrade it to AIX 6.1 tl4 sp6 to the new hardware using my NIM server. My... (4 Replies)
Discussion started by: daveisme
4 Replies

8. UNIX for Dummies Questions & Answers

How to run script concurrently

Hi all, I have one script. Its job is to get 1 file from dirA and 1 file from dirB as parameters for another python script. However, there are a lot of files in both dir A and B, so running this scripts really takes a lot of time. So I wonder if there are any ways that I can do this faster,... (6 Replies)
Discussion started by: yoyomano
6 Replies

9. Shell Programming and Scripting

script for Trimming the log files

Dear Gurus, Trimming the alert log files of oracle. I need to write a script retaining last 20 lines of the log files and deleting all the other lines I tried with tail -20 , Please help me how to delete all the other lines, my log files has grown upt to 2G. Thanks in Advance Dave (5 Replies)
Discussion started by: dave234
5 Replies

10. Shell Programming and Scripting

Trimming in between the words

Hi i have a log file P12345_15728710:DEBUG:Begin P12345_15728710:DEBUG:Being P12345_15729310:DEBUG:GetAgen P12345_15726510:DEBUG:end i want to trim this file and i want like this 15728710 15728710 15729310 15726510 i tried sed ..but not working.. sed "s/.*P12345__ \(.*\)... (4 Replies)
Discussion started by: navsan420
4 Replies
SED(1)								   User Commands							    SED(1)

NAME
sed - stream editor for filtering and transforming text SYNOPSIS
sed [OPTION]... {script-only-if-no-other-script} [input-file]... DESCRIPTION
Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipe- line). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed's ability to filter text in a pipeline which particularly distinguishes it from other types of editors. -n, --quiet, --silent suppress automatic printing of pattern space -e script, --expression=script add the script to the commands to be executed -f script-file, --file=script-file add the contents of script-file to the commands to be executed --follow-symlinks follow symlinks when processing in place -i[SUFFIX], --in-place[=SUFFIX] edit files in place (makes backup if extension supplied) -l N, --line-length=N specify the desired line-wrap length for the `l' command --posix disable all GNU extensions. -r, --regexp-extended use extended regular expressions in the script. -s, --separate consider files as separate rather than as a single continuous long stream. -u, --unbuffered load minimal amounts of data from the input files and flush the output buffers more often --help display this help and exit --version output version information and exit If no -e, --expression, -f, or --file option is given, then the first non-option argument is taken as the sed script to interpret. All remaining arguments are names of input files; if no input files are specified, then the standard input is read. E-mail bug reports to: bonzini@gnu.org . Be sure to include the word ``sed'' somewhere in the ``Subject:'' field. COMMAND SYNOPSIS
This is just a brief synopsis of sed commands to serve as a reminder to those who already know sed; other documentation (such as the tex- info document) must be consulted for fuller descriptions. Zero-address ``commands'' : label Label for b and t commands. #comment The comment extends until the next newline (or the end of a -e script fragment). } The closing bracket of a { } block. Zero- or One- address commands = Print the current line number. a text Append text, which has each embedded newline preceded by a backslash. i text Insert text, which has each embedded newline preceded by a backslash. q Immediately quit the sed script without processing any more input, except that if auto-print is not disabled the current pattern space will be printed. Q Immediately quit the sed script without processing any more input. r filename Append text read from filename. R filename Append a line read from filename. Commands which accept address ranges { Begin a block of commands (end with a }). b label Branch to label; if label is omitted, branch to end of script. t label If a s/// has done a successful substitution since the last input line was read and since the last t or T command, then branch to label; if label is omitted, branch to end of script. T label If no s/// has done a successful substitution since the last input line was read and since the last t or T command, then branch to label; if label is omitted, branch to end of script. c text Replace the selected lines with text, which has each embedded newline preceded by a backslash. d Delete pattern space. Start next cycle. D Delete up to the first embedded newline in the pattern space. Start next cycle, but skip reading from the input if there is still data in the pattern space. h H Copy/append pattern space to hold space. g G Copy/append hold space to pattern space. x Exchange the contents of the hold and pattern spaces. l List out the current line in a ``visually unambiguous'' form. n N Read/append the next line of input into the pattern space. p Print the current pattern space. P Print up to the first embedded newline of the current pattern space. s/regexp/replacement/ Attempt to match regexp against the pattern space. If successful, replace that portion matched with replacement. The replacement may contain the special character & to refer to that portion of the pattern space which matched, and the special escapes 1 through 9 to refer to the corresponding matching sub-expressions in the regexp. w filename Write the current pattern space to filename. W filename Write the first line of the current pattern space to filename. y/source/dest/ Transliterate the characters in the pattern space which appear in source to the corresponding character in dest. Addresses Sed commands can be given with no addresses, in which case the command will be executed for all input lines; with one address, in which case the command will only be executed for input lines which match that address; or with two addresses, in which case the command will be executed for all input lines which match the inclusive range of lines starting from the first address and continuing to the second address. Three things to note about address ranges: the syntax is addr1,addr2 (i.e., the addresses are separated by a comma); the line which addr1 matched will always be accepted, even if addr2 selects an earlier line; and if addr2 is a regexp, it will not be tested against the line that addr1 matched. After the address (or address-range), and before the command, a ! may be inserted, which specifies that the command shall only be executed if the address (or address-range) does not match. The following address types are supported: number Match only the specified line number. first~step Match every step'th line starting with line first. For example, ``sed -n 1~2p'' will print all the odd-numbered lines in the input stream, and the address 2~5 will match every fifth line, starting with the second. (This is an extension.) $ Match the last line. /regexp/ Match lines matching the regular expression regexp. cregexpc Match lines matching the regular expression regexp. The c may be any character. GNU sed also supports some special 2-address forms: 0,addr2 Start out in "matched first address" state, until addr2 is found. This is similar to 1,addr2, except that if addr2 matches the very first line of input the 0,addr2 form will be at the end of its range, whereas the 1,addr2 form will still be at the beginning of its range. addr1,+N Will match addr1 and the N lines following addr1. addr1,~N Will match addr1 and the lines following addr1 until the next line whose input line number is a multiple of N. REGULAR EXPRESSIONS
POSIX.2 BREs should be supported, but they aren't completely because of performance problems. The sequence in a regular expression matches the newline character, and similarly for a, , and other sequences. BUGS
E-mail bug reports to bonzini@gnu.org. Be sure to include the word ``sed'' somewhere in the ``Subject:'' field. Also, please include the output of ``sed --version'' in the body of your report if at all possible. COPYRIGHT
Copyright (C) 2003 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU- LAR PURPOSE, to the extent permitted by law. SEE ALSO
awk(1), ed(1), grep(1), tr(1), perlre(1), sed.info, any of various books on sed, the sed FAQ (http://sed.sf.net/grabbag/tutorials/sed- faq.txt), http://sed.sf.net/grabbag/. The full documentation for sed is maintained as a Texinfo manual. If the info and sed programs are properly installed at your site, the command info sed should give you access to the complete manual. sed version 4.1.5 July 2010 SED(1)
All times are GMT -4. The time now is 05:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy