Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

retain last 1000 line in a file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting retain last 1000 line in a file
# 1  
Old 05-11-2010
Error retain last 1000 line in a file
I have large file with around 100k+ lines. I wanted to retain only the last 100 lines in that file. One way i thought was using
Code:
 
tail -1000 filename > filename1
mv filename1 filename

But there should be a better solution.. Is there a way I can use sed or any such command to change the same file so that I have just the last 1000 lines are retained? So that we dont need to copy the output to another file and rename it back?



Many Thanks
Last edited by nss280; 05-11-2010 at 04:41 PM.. Reason: code written incorrectly
# 2  
Old 05-11-2010
No, you can't use the command to change the same file.You will have to put the desired output,after applying proper filters, in a seprate file then rename it to the original one...SmilieSmilie
# 3  
Old 05-11-2010
Thank you Reebot !

But any faster operation then tail that can help out here ???
# 4  
Old 05-11-2010
The approach will depend on the Operating System and the related tools, whether the file is currently open by a process (like syslogd), and whether it is a normal unix text file suitable for processing with shell tools.

Worth looking at the actual size of 1000 records because most versions of "tail" are limited in how much data they will buffer (you could for example ask for 1000 lines and only get 300).

If we assume that the file is a static normal unix text file and not open by another process, my first inclination would be to use "wc" to count the records then "sed" to output the required number of records to a temporary file.
This User Gave Thanks to methyl For This Post:
nss280
# 5  
Old 05-11-2010
Quote:
Originally Posted by nss280
Thank you Reebot !

But any faster operation then tail that can help out here ???

Try Following :

Code:
sed -e :a -e '$q;N;1001,$D;ba'  file_name

Hope this works for you....SmilieSmilie
Last edited by Reboot; 05-11-2010 at 05:38 PM.. Reason: typo
# 6  
Old 05-11-2010
I'm thinking your issue is more I/O related than an issue with tail; maybe the onus is on the write speed as opposed to your accessing the last 1000 lines of the file. You could even read it into memory and then spit it back out to overwrite your file, but be careful as it may bite you if it's too big.

Otherwise, tail is pretty fast since alternatives would need to parse logic surrounding your line population. Some are better than others, but tail is very simple in its approach: $((EOF - integer)):

Code:
-> wc -l /opt/worklibs/tmp/i*114054.dat
  434641 /opt/worklibs/tmp/i_PPGLBL_20081231114054.dat

-> time sed -n "400000,1000p" /opt/worklibs/tmp/i_PPGLBL_20081231114054.dat >/dev/null

real    0m7.91s
user    0m3.96s
sys     0m3.94s

-> time tail -1000 /opt/worklibs/tmp/i_*114054.dat >/dev/null                                

real    0m0.03s
user    0m0.02s
sys     0m0.01s

-> time awk 'NR>=400000 && NR<=401001 {print $0;}' /opt/worklibs/tmp/i*114054.dat >/dev/null 

real    0m23.77s
user    0m23.06s
sys     0m0.71s

-> time nawk 'NR>=400000 && NR<=401001 {print $0;}' /opt/worklibs/tmp/i*114054.dat >/dev/null

real    0m4.57s
user    0m4.06s
sys     0m0.50s

Last edited by curleb; 05-11-2010 at 05:41 PM.. Reason: consistency...
This User Gave Thanks to curleb For This Post:
nss280
# 7  
Old 05-11-2010
Quote:
Originally Posted by Reboot
Try Following :

Code:
sed -e :a -e '$q;N;1001,$D;ba'  file_name

Hope this works for you....SmilieSmilie
OR Else you can use two commands in a single stratch as :

Code:
sed -e :a -e '$q;N;1001,$D;ba'  orig_file >copy_file ;mv copy_file  orig_file

This User Gave Thanks to Reboot For This Post:
nss280
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Creating a sequence of numbers in a line for 1000 files

Hi, I try to explain my problem , I have a file like this: aasdsaffsc23 scdsfsddvf46567 mionome0001.pdb asdsdvcxvds dsfdvcvc2324w What I need to do is to create 1000 files in which myname line listing a sequence of numbers from 0001 to 1000. So I want to have : nomefile0001.txt that must... (10 Replies)
Discussion started by: danyz84
10 Replies

2. Shell Programming and Scripting

Adding a line to 1000's of files right after x amt of characters.

I am trying to add a single line of text to every file in a particular folder. There are thousands of files in the folder. Each file contains this same start of the first line: {d "%w- %d %m-, %Y - %T"} <some message here> with the rest of the text following the second curly bracket... (10 Replies)
Discussion started by: dlundwall
10 Replies

3. UNIX for Dummies Questions & Answers

cat to a file but retain header

Hi, Is there a way to write to a txt file each day but retain the header on the file? I'm cat'ing 5 files into one .txt file each day but I want the new data to be written after the first 2 lines which are: Progname Size Date Owner ---------------------------- Basically I want my new... (4 Replies)
Discussion started by: Grueben
4 Replies

4. Shell Programming and Scripting

How to retain backslash in a line while reading a data file?

Hello Firends I have a file that contains data within single quotes, which has meaning of its own. When I am trying to parse through the file for a different functionality I noticed that I was loosing the backslash when occurrences in the file look like ('\0'). I would want to retain the... (3 Replies)
Discussion started by: easwam
3 Replies

5. Shell Programming and Scripting

Retain File Timestamp

There are directories of files that I have to run the dos2ux command on to get ride of the carriage return characters. Easy enough, but I have to retain the original timestamps on the files. I am thinking that I am going to have to strip off the timestamp for each file and convert it to unix time... (3 Replies)
Discussion started by: scotbuff
3 Replies

6. Shell Programming and Scripting

Retain file permissions when saving .sh file from internet [OS X]

Hello. I have written a bash script that I am sharing with an OS X community I am a member of. The purpose of the script is to execute a series of commands for members without them having to get involved with Terminal, as it can be daunting for those with no experience of it at all. I have renamed... (4 Replies)
Discussion started by: baza210
4 Replies

7. UNIX for Dummies Questions & Answers

Displays line number 1000 to 2000

Hi, I have 1 million records and want to extract lines betwen 10000 -20000 and put it in another file. Could you please suggest a command for this. Thanks in advance (3 Replies)
Discussion started by: unxusr123
3 Replies

8. UNIX for Dummies Questions & Answers

How to retain the header information of a file

Hi, I am using Bash shell to create some data and these data would be piped out to a file, let say output.txt. This output.txt I would like to add some extra header information such as comments, descriptions and general information on the text. I would like to know how could I maintain... (0 Replies)
Discussion started by: ahjiefreak
0 Replies

9. Shell Programming and Scripting

Command to add 1000 spaces to end of line

hi, could anyone tell me the command to append spaces at the end of the line. for example, i need 1000 spaces after the word "helloworld" echo "helloworld " i need to achieve this in someother way hardcoding 1000 spaces is not practical. as i am totally new... (3 Replies)
Discussion started by: kavithacs
3 Replies

10. Shell Programming and Scripting

retain Line numbers.. in Vi .. OR .. A SHELL SCRIPT

Hello everybody ! GOT SOMETHING INTERESTING... I am trying to retain line number for a text document.. usually we get line numbers in VI using :set nu , but I want to permanently store them. It's a 4000 lines of text and I want grep/search it for a list of words/fields stored in a different... (2 Replies)
Discussion started by: sdlayeeq
2 Replies
Login or Register to Ask a Question