Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Want to remove all lines but not latest 50 lines from a file
Top Forums UNIX for Dummies Questions & Answers Want to remove all lines but not latest 50 lines from a file Post 302868277 by drl on Sunday 27th of October 2013 12:30:28 AM
Old 10-27-2013
Hi.

For the version of tail that I used, as noted above:
Code:
       If the first character of N (the number of bytes or lines) is a `+',
       print beginning with the Nth item from the start of each file,
       otherwise, print the last N items in the file.  N may have a multiplier
       suffix: b 512, kB 1000, K 1024, MB 1000*1000, M 1024*1024, GB
       1000*1000*1000, G 1024*1024*1024, and so on for T, P, E, Z, Y.

so it looks like the GNU folks have made provisions for some large quantities of data. I have not looked at the source, however.

For the tests that I ran, I numbered all the lines, counted the final tail (50,000 of 14,000,000), and printed the first and last line of the output file. All seemed correct and intact. The system tail and the version in perl at http://cpansearch.perl.org/src/CWEST/ppt-0.14/bin/tail both return essentially immediately when asked for the last 5 lines of the 14M line file. I noticed some seeks in the perl version, so I'd guess that both those versions are using such performance techniques.

I don't disagree that some utilities will produce wrong results without warning, but that's not the right thing to do, and those systems should be avoided if possible, and especially if one will normally try to solve problems such as we are addressing here. Another possibility is to use the perl version, although I didn't rigorously test it (and the calling sequence is slightly different, but easily changed).

I might worry more about the shell capacity, although I didn't have any trouble, and I recall a test I did on an early version of SunOS, and was pleasantly surprised at its capacity for variable storage. Of course, the MMV for specific cases.

Best wishes ... cheers, drl
Last edited by drl; 10-27-2013 at 09:48 AM.. Reason: Minor typo.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove lines from file

file: 1 xxxxxxx 2 xxx xxx 5 xxx xxx ... 180 xxxxxx 200 xxx how to remove any lines with the first number range 1-180 (9 Replies)
Discussion started by: bluemoon1
9 Replies

2. Shell Programming and Scripting

To remove the lines in my file

Hi, There seems to some hack attempts in my site. I have attached the index page of my site and I need to remove the below lines from the index page. The below lines are at the center of the file. --> </style> <script>E V A L( unescape(... (5 Replies)
Discussion started by: gsiva
5 Replies

3. UNIX for Dummies Questions & Answers

vi to remove lines in file

All, I have a text file with several entries like below: personname personname.domain.com I know there is a way to use vi to remove only the personname.domain.com line. Can someone help? I believe that it involves /s/g/ something...I just can't remember the exact syntax. Thanks (2 Replies)
Discussion started by: kjbaumann
2 Replies

4. Shell Programming and Scripting

remove lines from file

Hi gurus, i'm trying to remove a number of lines from a large file using the following command: sed '1,5000d' oldfile > newfile Somehow the lines in the old file are not deleted... Am I doing this wrongly? Any suggestions? :confused: Thanks! :) wee (10 Replies)
Discussion started by: lweegp
10 Replies

5. Shell Programming and Scripting

remove : lines from file

A small question I have a test.txt file I have contents as: a:google b:yahoo : c:facebook : d:hotmail How do I remove the line with : my output should be a:google b:yahoo c:facebook d:hotmail (5 Replies)
Discussion started by: aronmelon
5 Replies

6. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

7. Shell Programming and Scripting

Remove lines from file

Hey Gang- I have a list of servers. I want to exclude servers that begin with and end with certain characters. Is there an easy command to do this? Example wvm1234dev wvm1234pro uvm1122dev uvm1122bku uvm1344dev I want to exclude any lines that start with "wvm" OR "uvm" AND end... (7 Replies)
Discussion started by: idiotboy
7 Replies

8. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

9. Shell Programming and Scripting

Remove lines that are subsets of other lines in File

Hello everyone, Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done. My problem is the following: I have a file where each line is a sequence of IP addresses, example : 10.0.0.1 10.0.0.2 10.0.0.5 10.0.0.1 10.0.0.2... (5 Replies)
Discussion started by: MisterJellyBean
5 Replies

10. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

LEARN ABOUT SUSE

split

SPLIT(1)							   User Commands							  SPLIT(1)

NAME

       split - split a file into pieces

SYNOPSIS

       split [OPTION]... [INPUT [PREFIX]]

DESCRIPTION

       Output  fixed-size  pieces  of  INPUT to PREFIXaa, PREFIXab, ...; default size is 1000 lines, and default PREFIX is `x'.  With no INPUT, or
       when INPUT is -, read standard input.

       Mandatory arguments to long options are mandatory for short options too.

       -a, --suffix-length=N
	      use suffixes of length N (default 2)

       -b, --bytes=SIZE
	      put SIZE bytes per output file

       -C, --line-bytes=SIZE
	      put at most SIZE bytes of lines per output file

       -d, --numeric-suffixes
	      use numeric suffixes instead of alphabetic

       -l, --lines=NUMBER
	      put NUMBER lines per output file

       --verbose
	      print a diagnostic just before each output file is opened

       --help display this help and exit

       --version
	      output version information and exit

       SIZE may have a multiplier suffix: b 512, kB 1000, K 1024, MB 1000*1000, M 1024*1024, GB 1000*1000*1000, G 1024*1024*1024, and so on for T,
       P, E, Z, Y.

AUTHOR

       Written by Torbjorn Granlund and Richard M. Stallman.

REPORTING BUGS

       Report split bugs to bug-coreutils@gnu.org
       GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
       General help using GNU software: <http://www.gnu.org/gethelp/>

COPYRIGHT

       Copyright (C) 2009 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO

       The  full  documentation  for split is maintained as a Texinfo manual.  If the info and split programs are properly installed at your site,
       the command

	      info coreutils 'split invocation'

       should give you access to the complete manual.

GNU coreutils 7.1						     July 2010								  SPLIT(1)
All times are GMT -4. The time now is 05:01 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy