Tail 86000 lines from 1.2 million line file?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Tail 86000 lines from 1.2 million line file?
# 1  
Old 11-10-2009
Question Tail 86000 lines from 1.2 million line file?

I have a log file that is about 1.2 million lines long and about 300MB.
we need a way to clean up this file and only keep the last few thousand lines.

if i use tail command we run our of memory as the file is too big.

I do have a key word to match on.
example, we want to keep every line after the line containing 1104100311229
and this magic number will only be in the log once.


I feel there may be a way to use awk?



I offer 10,000 Bits to the best answer...
# 2  
Old 11-10-2009
How about:
Code:
sed '1,/1104100311229/d' logfile > logfile.keepers

# 3  
Old 11-10-2009
so u need 86000 last lines or magic number (many different numbers?) or both?
P.S. 300mb is weak
tip78
# 4  
Old 11-10-2009
the total lines we need to keep will be about 86000 lines.

the exact point we need to keep from is the magic number.
# 5  
Old 11-10-2009
but u said:
Quote:
and this magic number will only be in the log once.
so there 86000 different magic numbers?
tip78
# 6  
Old 11-11-2009
Turning Scrutinizers sed round so we keep the line containing the search key, we can seek to the start line then start outputting.

Code:
sed -n "/1104100311229/,\$ p" filename >filename.new

# 7  
Old 11-11-2009
Quote:
Originally Posted by methyl
Turning Scrutinizers sed round so we keep the line containing the search key, we can seek to the start line then start outputting.

Code:
sed -n "/1104100311229/,\$ p" filename >filename.new

that works for me.

thanks.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

How to display certain line of file only using head or tail in 1 command?

First month learning about the Linux terminal and it has been a challenge yet fun so far. We're learning by using a gameshell. I'm trying to display a certain line ( only allowed 1 command ) from a file only using the head or tail. I'm pretty about this answer: head -23 history.txt | tail -1... (1 Reply)
Discussion started by: forzatekk
1 Replies

2. Shell Programming and Scripting

Find all lines in file such that each word on that line appears in at least n lines of the file

I have a file where every line includes four expressions with a caret in the middle (plus some other "words" or fields, always separated by spaces). I would like to extract from this file, all those lines such that each of the four expressions containing a caret appears in at least four different... (9 Replies)
Discussion started by: uncleMonty
9 Replies

3. Shell Programming and Scripting

Need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line...

Hello, I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be: SRVXPAPI001 ERRO JUN24 07:28:34 1775 REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F ... (8 Replies)
Discussion started by: Ferocci
8 Replies

4. Shell Programming and Scripting

How to process only new line of tail -f command from live log file?

Hi, I want to read a live log file line by line and considering those line which are newly added to file Below code I am using, which read line but as soon as it read new line from log file its starts processing from very first line of file. tail -F /logs/COMMON-ERROR.log | while read... (11 Replies)
Discussion started by: ketanraut
11 Replies

5. UNIX for Dummies Questions & Answers

Help with changing header of tsv with 30 million lines

Hi My 30 million line file has a header chr start end strand ref_context repeat_masked s1_smpl_context s1_c_count s1_ct_count s1_non_ct_count s1_m% s1_score s1_snp s1_indels s2_smpl_context s2_c_count s2_ct_count s2_non_ct_count s2_m% s2_score s2_snp s2_indels ... (2 Replies)
Discussion started by: plumb_r
2 Replies

6. UNIX for Advanced & Expert Users

SQL script with 86000 lines: new files with only 10000 lines (per file)

Hi this is my SQL script $ wc -l insert_into_customers.sql 85601 insert_into_customers.sqlI wish to cut this file into 9 files each 10000 lines (the last one less) $ wc -l insert_into_customers_00*.sql 10000 insert_into_customers_001.sql 10000 insert_into_customers_002.sql ... (1 Reply)
Discussion started by: slashdotweenie
1 Replies

7. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

8. Shell Programming and Scripting

Matching 10 Million file records with 10 Million in other file

Dear All, I have two files both containing 10 Million records each separated by comma(csv fmt). One file is input.txt other is status.txt. Input.txt-> contains fields with one unique id field (primary key we can say) Status.txt -> contains two fields only:1. unique id and 2. status ... (8 Replies)
Discussion started by: vguleria
8 Replies

9. UNIX for Dummies Questions & Answers

Display lines 30 to 40 of a text file using head and/or tail commands

Assume that the text file contains over 40 lines. How do you do this?!?!? (5 Replies)
Discussion started by: phunkypants
5 Replies

10. UNIX for Dummies Questions & Answers

how to tail last 500 lines and vi them?

I have a very large log file and it speed up scrolling. so I want to tail last 500 lies and see using vi editor. tail -n 500 large_file | small_file | vi {}; this won't work. I'm very novice on Unix. TIA. (2 Replies)
Discussion started by: kang
2 Replies
Login or Register to Ask a Question