Sponsored Content
Top Forums Shell Programming and Scripting Tail 86000 lines from 1.2 million line file? Post 302370190 by robsonde on Tuesday 10th of November 2009 05:21:13 PM
Old 11-10-2009
Question Tail 86000 lines from 1.2 million line file?

I have a log file that is about 1.2 million lines long and about 300MB.
we need a way to clean up this file and only keep the last few thousand lines.

if i use tail command we run our of memory as the file is too big.

I do have a key word to match on.
example, we want to keep every line after the line containing 1104100311229
and this magic number will only be in the log once.


I feel there may be a way to use awk?



I offer 10,000 Bits to the best answer...
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

how to tail last 500 lines and vi them?

I have a very large log file and it speed up scrolling. so I want to tail last 500 lies and see using vi editor. tail -n 500 large_file | small_file | vi {}; this won't work. I'm very novice on Unix. TIA. (2 Replies)
Discussion started by: kang
2 Replies

2. UNIX for Dummies Questions & Answers

Display lines 30 to 40 of a text file using head and/or tail commands

Assume that the text file contains over 40 lines. How do you do this?!?!? (5 Replies)
Discussion started by: phunkypants
5 Replies

3. Shell Programming and Scripting

Matching 10 Million file records with 10 Million in other file

Dear All, I have two files both containing 10 Million records each separated by comma(csv fmt). One file is input.txt other is status.txt. Input.txt-> contains fields with one unique id field (primary key we can say) Status.txt -> contains two fields only:1. unique id and 2. status ... (8 Replies)
Discussion started by: vguleria
8 Replies

4. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

5. UNIX for Advanced & Expert Users

SQL script with 86000 lines: new files with only 10000 lines (per file)

Hi this is my SQL script $ wc -l insert_into_customers.sql 85601 insert_into_customers.sqlI wish to cut this file into 9 files each 10000 lines (the last one less) $ wc -l insert_into_customers_00*.sql 10000 insert_into_customers_001.sql 10000 insert_into_customers_002.sql ... (1 Reply)
Discussion started by: slashdotweenie
1 Replies

6. UNIX for Dummies Questions & Answers

Help with changing header of tsv with 30 million lines

Hi My 30 million line file has a header chr start end strand ref_context repeat_masked s1_smpl_context s1_c_count s1_ct_count s1_non_ct_count s1_m% s1_score s1_snp s1_indels s2_smpl_context s2_c_count s2_ct_count s2_non_ct_count s2_m% s2_score s2_snp s2_indels ... (2 Replies)
Discussion started by: plumb_r
2 Replies

7. Shell Programming and Scripting

How to process only new line of tail -f command from live log file?

Hi, I want to read a live log file line by line and considering those line which are newly added to file Below code I am using, which read line but as soon as it read new line from log file its starts processing from very first line of file. tail -F /logs/COMMON-ERROR.log | while read... (11 Replies)
Discussion started by: ketanraut
11 Replies

8. Shell Programming and Scripting

Need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line...

Hello, I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be: SRVXPAPI001 ERRO JUN24 07:28:34 1775 REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F ... (8 Replies)
Discussion started by: Ferocci
8 Replies

9. Shell Programming and Scripting

Find all lines in file such that each word on that line appears in at least n lines of the file

I have a file where every line includes four expressions with a caret in the middle (plus some other "words" or fields, always separated by spaces). I would like to extract from this file, all those lines such that each of the four expressions containing a caret appears in at least four different... (9 Replies)
Discussion started by: uncleMonty
9 Replies

10. Homework & Coursework Questions

How to display certain line of file only using head or tail in 1 command?

First month learning about the Linux terminal and it has been a challenge yet fun so far. We're learning by using a gameshell. I'm trying to display a certain line ( only allowed 1 command ) from a file only using the head or tail. I'm pretty about this answer: head -23 history.txt | tail -1... (1 Reply)
Discussion started by: forzatekk
1 Replies
rancid_par(1)						      General Commands Manual						     rancid_par(1)

NAME
rancid_par - parallel command processing SYNOPSIS
rancid_par [-dfiqx] [-c command] [-l logfile] [-n #] file [file...] DESCRIPTION
rancid_par takes a list of files to run a command on. The first line of each file begins with a colon (:) or a pound-sign (#). If a colon, the remainder of the line is a command to run for each of the subsequent lines. If a pound-sign, then each subsequent line is a (self-contained) command, unless the -c option was specified, in which case it operates as if the argument to -c had followed a colon on the first line. In each of the cases where the lines of the file following the first are not commands (i.e.: colon or -c), instances of open-close braces ({}) in the command will be replaced by these values. For example, a inputfile whose contents is: : echo {} a b c run with rancid_par like so: %rancid_par -q inputfile will produce the following output (order will vary): b a c The command-line options are as follows: -c Command to be run on each of the arguments following the command-line options, where the first line of the input file(s) begins with a pound-sign (#). -d Print debugging information on standard error (stderr). -f No file or STDIN, just run a quantity of the command specified with -c. -i Run commands interactively through (multiple) xterm(1) processes. -l Prefix of logfile name, as in prefix.N where N is the rancid_par process number ([0..]). Default: par.log.<time>.[0..] -n Number of simultaneous processes. Default: 3 -q Quiet mode. Do not log anything. -q is mutually exclusive with the -x and -l options and the option appearing last will take precedence. -x View rancid_par logs in real-time via an xterm(1). FILES
par.log.T.N Log file; where T is the current time in seconds since the epoch and N is the rancid_par process number ([0..]). 18 December 2007 rancid_par(1)
All times are GMT -4. The time now is 12:33 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy