Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Want to remove all lines but not latest 50 lines from a file

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Want to remove all lines but not latest 50 lines from a file
# 29  
Old 10-29-2013
Hi.

As alister said.

This behavior has been discussed in the forum, for example, I mentioned sponge / absorb for pipelines and re-writing-in-place in post #8 of https://www.unix.com/shell-programming-and-scripting/221749-run-command-without-over-writing.html -- that approach either keeps data in memory (say a perl array), or writes it to a scratch file first, both of which can preserve the inode if that is important. The sponge approach looks like the tee solution, but the internal logic differs by sponge collecting all the data until the end, then writing to the file argument, whereas tee streams the data to the file(s) as it comes in.

We cannot use the scratch file approach here because there is no additional disk space, so suggestions like the shell and dd above seemed appropriate. Both were validated on a file of 14M lines (~ 1 GB) with 50,000 lines being kept.

Best wishes ... cheers, drl
Last edited by drl; 10-30-2013 at 11:53 AM.. Reason: Minor typo.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

Remove lines that are subsets of other lines in File

Hello everyone, Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done. My problem is the following: I have a file where each line is a sequence of IP addresses, example : 10.0.0.1 10.0.0.2 10.0.0.5 10.0.0.1 10.0.0.2... (5 Replies)
Discussion started by: MisterJellyBean
5 Replies

3. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

4. Shell Programming and Scripting

Remove lines from file

Hey Gang- I have a list of servers. I want to exclude servers that begin with and end with certain characters. Is there an easy command to do this? Example wvm1234dev wvm1234pro uvm1122dev uvm1122bku uvm1344dev I want to exclude any lines that start with "wvm" OR "uvm" AND end... (7 Replies)
Discussion started by: idiotboy
7 Replies

5. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

6. Shell Programming and Scripting

remove : lines from file

A small question I have a test.txt file I have contents as: a:google b:yahoo : c:facebook : d:hotmail How do I remove the line with : my output should be a:google b:yahoo c:facebook d:hotmail (5 Replies)
Discussion started by: aronmelon
5 Replies

7. Shell Programming and Scripting

remove lines from file

Hi gurus, i'm trying to remove a number of lines from a large file using the following command: sed '1,5000d' oldfile > newfile Somehow the lines in the old file are not deleted... Am I doing this wrongly? Any suggestions? :confused: Thanks! :) wee (10 Replies)
Discussion started by: lweegp
10 Replies

8. UNIX for Dummies Questions & Answers

vi to remove lines in file

All, I have a text file with several entries like below: personname personname.domain.com I know there is a way to use vi to remove only the personname.domain.com line. Can someone help? I believe that it involves /s/g/ something...I just can't remember the exact syntax. Thanks (2 Replies)
Discussion started by: kjbaumann
2 Replies

9. Shell Programming and Scripting

To remove the lines in my file

Hi, There seems to some hack attempts in my site. I have attached the index page of my site and I need to remove the below lines from the index page. The below lines are at the center of the file. --> </style> <script>E V A L( unescape(... (5 Replies)
Discussion started by: gsiva
5 Replies

10. Shell Programming and Scripting

remove lines from file

file: 1 xxxxxxx 2 xxx xxx 5 xxx xxx ... 180 xxxxxx 200 xxx how to remove any lines with the first number range 1-180 (9 Replies)
Discussion started by: bluemoon1
9 Replies
Login or Register to Ask a Question

LEARN ABOUT REDHAT

prunehistory

PRUNEHISTORY(8) 					      System Manager's Manual						   PRUNEHISTORY(8)

NAME

       prunehistory - remove file names from Usenet history file

SYNOPSIS

       prunehistory [ -f filename ] [ -p ]

DESCRIPTION

       Prunehistory  modifies  the  history(5)	text file to ``remove'' a set of filenames from it.  The filenames are removed by overwriting them
       with spaces, so that the size and position of any following entries does not change.

       Prunehistory reads the standard input.  The input is taken as a set of lines.  Blank lines and lines starting with a  number  sign  (``#'')
       are ignored.  All other lines are should consist of a Message-ID followed by zero or more filenames.

       The  Message-ID	is  used as the dbz(3) key to get an offset into the text file.  If no filenames are mentioned on the input line, then all
       filenames in the text are ``removed.''  If any filenames are mentioned, they are converted into the history file notation.  If they  appear
       in the line for the specified Message-ID then they are removed.

       Since innd(8) only appends to the text file, prunehistory does not need to have any interaction with it.

OPTIONS

       -p     Prunehistory will normally complain about lines that do not follow the correct format.  If the ``-p'' flag is used, then the program
	      will silently print any invalid lines on its standard output.  (Blank lines and comment lines are also passed through.)  This can be
	      useful when prunehistory is used as a filter for other programs such as reap.

       -f     The default name of the history file is <pathdb in inn.conf>/history; to specify a different name, use the ``-f'' flag.

HISTORY

       Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews.  This is revision 1.7, dated 1998/04/09.

SEE ALSO

       dbz(3), history(5), inn.conf(5), innd(8).

																   PRUNEHISTORY(8)