Sponsored Content
Top Forums Shell Programming and Scripting Tail 86000 lines from 1.2 million line file? Post 302370190 by robsonde on Tuesday 10th of November 2009 05:21:13 PM
Old 11-10-2009
Question Tail 86000 lines from 1.2 million line file?

I have a log file that is about 1.2 million lines long and about 300MB.
we need a way to clean up this file and only keep the last few thousand lines.

if i use tail command we run our of memory as the file is too big.

I do have a key word to match on.
example, we want to keep every line after the line containing 1104100311229
and this magic number will only be in the log once.


I feel there may be a way to use awk?



I offer 10,000 Bits to the best answer...
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

how to tail last 500 lines and vi them?

I have a very large log file and it speed up scrolling. so I want to tail last 500 lies and see using vi editor. tail -n 500 large_file | small_file | vi {}; this won't work. I'm very novice on Unix. TIA. (2 Replies)
Discussion started by: kang
2 Replies

2. UNIX for Dummies Questions & Answers

Display lines 30 to 40 of a text file using head and/or tail commands

Assume that the text file contains over 40 lines. How do you do this?!?!? (5 Replies)
Discussion started by: phunkypants
5 Replies

3. Shell Programming and Scripting

Matching 10 Million file records with 10 Million in other file

Dear All, I have two files both containing 10 Million records each separated by comma(csv fmt). One file is input.txt other is status.txt. Input.txt-> contains fields with one unique id field (primary key we can say) Status.txt -> contains two fields only:1. unique id and 2. status ... (8 Replies)
Discussion started by: vguleria
8 Replies

4. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

5. UNIX for Advanced & Expert Users

SQL script with 86000 lines: new files with only 10000 lines (per file)

Hi this is my SQL script $ wc -l insert_into_customers.sql 85601 insert_into_customers.sqlI wish to cut this file into 9 files each 10000 lines (the last one less) $ wc -l insert_into_customers_00*.sql 10000 insert_into_customers_001.sql 10000 insert_into_customers_002.sql ... (1 Reply)
Discussion started by: slashdotweenie
1 Replies

6. UNIX for Dummies Questions & Answers

Help with changing header of tsv with 30 million lines

Hi My 30 million line file has a header chr start end strand ref_context repeat_masked s1_smpl_context s1_c_count s1_ct_count s1_non_ct_count s1_m% s1_score s1_snp s1_indels s2_smpl_context s2_c_count s2_ct_count s2_non_ct_count s2_m% s2_score s2_snp s2_indels ... (2 Replies)
Discussion started by: plumb_r
2 Replies

7. Shell Programming and Scripting

How to process only new line of tail -f command from live log file?

Hi, I want to read a live log file line by line and considering those line which are newly added to file Below code I am using, which read line but as soon as it read new line from log file its starts processing from very first line of file. tail -F /logs/COMMON-ERROR.log | while read... (11 Replies)
Discussion started by: ketanraut
11 Replies

8. Shell Programming and Scripting

Need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line...

Hello, I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be: SRVXPAPI001 ERRO JUN24 07:28:34 1775 REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F ... (8 Replies)
Discussion started by: Ferocci
8 Replies

9. Shell Programming and Scripting

Find all lines in file such that each word on that line appears in at least n lines of the file

I have a file where every line includes four expressions with a caret in the middle (plus some other "words" or fields, always separated by spaces). I would like to extract from this file, all those lines such that each of the four expressions containing a caret appears in at least four different... (9 Replies)
Discussion started by: uncleMonty
9 Replies

10. Homework & Coursework Questions

How to display certain line of file only using head or tail in 1 command?

First month learning about the Linux terminal and it has been a challenge yet fun so far. We're learning by using a gameshell. I'm trying to display a certain line ( only allowed 1 command ) from a file only using the head or tail. I'm pretty about this answer: head -23 history.txt | tail -1... (1 Reply)
Discussion started by: forzatekk
1 Replies
TAILF(1)							   User Commands							  TAILF(1)

NAME
tailf - follow the growth of a log file SYNOPSIS
tailf [OPTION] file DESCRIPTION
tailf will print out the last 10 lines of a file and then wait for the file to grow. It is similar to tail -f but does not access the file when it is not growing. This has the side effect of not updating the access time for the file, so a filesystem flush does not occur peri- odically when no log activity is happening. tailf is extremely useful for monitoring log files on a laptop when logging is infrequent and the user desires that the hard disk spin down to conserve battery life. Mandatory arguments to long options are mandatory for short options too. -n, --lines=N, -N output the last N lines, instead of the last 10. -V, --version Output version information and exit. -h, --help Display help and exit. AUTHOR
This program was originally written by Rik Faith (faith@acm.org) and may be freely distributed under the terms of the X11/MIT License. There is ABSOLUTELY NO WARRANTY for this program. The latest inotify based implementation was written by Karel Zak (kzak@redhat.com). SEE ALSO
tail(1), less(1) AVAILABILITY
The tailf command is part of the util-linux package and is available from ftp://ftp.kernel.org/pub/linux/utils/util-linux/. util-linux February 2003 TAILF(1)
All times are GMT -4. The time now is 10:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy