Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Want to remove all lines but not latest 50 lines from a file
Top Forums UNIX for Dummies Questions & Answers Want to remove all lines but not latest 50 lines from a file Post 302868247 by drl on Saturday 26th of October 2013 05:34:08 PM
Old 10-26-2013
Hi.

Both of the solutions, shell variable and "dd move" are about the same as far as system resources go. I used alister's amended solution; his first failed (the one that went out with an email notification).

For a file of 14,754,910 lines, about 1 GB, the times for the shell solution keeping the last 50,000 lines were:
Code:
real	0m36.738s
user	0m1.624s
sys	0m8.321s

and for the clever "dd move" solution:
Code:
real	0m44.048s
user	0m1.408s
sys	0m8.725s

There were 2 wc executions for verification in both runs. Leaving those out:
Code:
real	0m29.443s
user	0m0.560s
sys	0m7.012s

and
Code:
real	0m30.536s
user	0m0.220s
sys	0m6.460s

The results actually surprised me -- I thought the shell would be slower. I saw paging being used a few times, slightly more often with the dd. The shell was run first, so cache advantage, if any, went to dd ... cheers, drl

This system:
Code:
 OS: Debian 5.0.8 lenny
 Kernel: x86_64 Linux 2.6.26-2-amd64
 Uptime: 59d 21h 58m
 CPU: AMD Athlon 64 3000+ @ 1.802GHz
 GPU: NVidia GeForce FX 5200
 RAM: 1170MB / 3024MB

Last edited by drl; 10-26-2013 at 06:39 PM.. Reason: Typo.
This User Gave Thanks to drl For This Post:
alister
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove lines from file

file: 1 xxxxxxx 2 xxx xxx 5 xxx xxx ... 180 xxxxxx 200 xxx how to remove any lines with the first number range 1-180 (9 Replies)
Discussion started by: bluemoon1
9 Replies

2. Shell Programming and Scripting

To remove the lines in my file

Hi, There seems to some hack attempts in my site. I have attached the index page of my site and I need to remove the below lines from the index page. The below lines are at the center of the file. --> </style> <script>E V A L( unescape(... (5 Replies)
Discussion started by: gsiva
5 Replies

3. UNIX for Dummies Questions & Answers

vi to remove lines in file

All, I have a text file with several entries like below: personname personname.domain.com I know there is a way to use vi to remove only the personname.domain.com line. Can someone help? I believe that it involves /s/g/ something...I just can't remember the exact syntax. Thanks (2 Replies)
Discussion started by: kjbaumann
2 Replies

4. Shell Programming and Scripting

remove lines from file

Hi gurus, i'm trying to remove a number of lines from a large file using the following command: sed '1,5000d' oldfile > newfile Somehow the lines in the old file are not deleted... Am I doing this wrongly? Any suggestions? :confused: Thanks! :) wee (10 Replies)
Discussion started by: lweegp
10 Replies

5. Shell Programming and Scripting

remove : lines from file

A small question I have a test.txt file I have contents as: a:google b:yahoo : c:facebook : d:hotmail How do I remove the line with : my output should be a:google b:yahoo c:facebook d:hotmail (5 Replies)
Discussion started by: aronmelon
5 Replies

6. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

7. Shell Programming and Scripting

Remove lines from file

Hey Gang- I have a list of servers. I want to exclude servers that begin with and end with certain characters. Is there an easy command to do this? Example wvm1234dev wvm1234pro uvm1122dev uvm1122bku uvm1344dev I want to exclude any lines that start with "wvm" OR "uvm" AND end... (7 Replies)
Discussion started by: idiotboy
7 Replies

8. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

9. Shell Programming and Scripting

Remove lines that are subsets of other lines in File

Hello everyone, Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done. My problem is the following: I have a file where each line is a sequence of IP addresses, example : 10.0.0.1 10.0.0.2 10.0.0.5 10.0.0.1 10.0.0.2... (5 Replies)
Discussion started by: MisterJellyBean
5 Replies

10. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

LEARN ABOUT SUSE

php-config

php-config(1)                                                   Scripting Language                                                   php-config(1)

NAME

       php-config - get information about PHP configuration and compile options

SYNOPSIS

       php-config [options]

DESCRIPTION

       php-config is a simple shell script for obtaining information about installed PHP configuration.

OPTIONS

       --prefix       Directory prefix where PHP is installed, e.g. /usr/local
       --includes     List of -I options with all include files
       --ldflags      LD Flags which PHP was compiled with
       --libs         Extra libraries which PHP was compiled with
       --man-dir      The directory prefix where the manpages is installed
       --extension-dir
                      Directory where extensions are searched by default
       --include-dir  Directory prefix where header files are installed by default
       --php-binary   Full path to php CLI or CGI binary
       --php-sapis    Show all SAPI modules installed on the Debian system
       --configure-options
                      Configure options to recreate configuration of current PHP installation
       --version      PHP version
       --vernum       PHP version as integer

SEE ALSO

       php(1)

VERSION INFORMATION

       This manpage describes php, version 7.0.33-6+ubuntu18.04.1+deb.sury.org+3.

COPYRIGHT

       Copyright (C) 1997-2017 The PHP Group

       This  source  file  is  subject to version 3.01 of the PHP license, that is bundled with this package in the file LICENSE, and is available
       through the world-wide-web at the following url:
       http://www.php.net/license/3_01.txt

       If you did not receive a copy of the PHP license and  are  unable  to  obtain  it  through  the  world-wide-web,  please  send  a  note  to
       license@php.net so we can mail you a copy immediately.

The PHP Group                                                          2017                                                          php-config(1)
All times are GMT -4. The time now is 09:53 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy