Sponsored Content
Top Forums Shell Programming and Scripting Reading ALL BUT the first and last line of a huge file Post 302969927 by kumarjt on Wednesday 30th of March 2016 09:08:52 AM
Old 03-30-2016
Reading ALL BUT the first and last line of a huge file

Hi.

Pardon me if I'm posting a duplicate thread but..
I have a text file with over 150 Million records, file size is in the range if MB(close to GB).
The requirement is to read ALL the lines excepting the FIRST LINE which is the file header and the LAST LINE which is it's trailer record.

What is the most OPTIMUM way to do it?
I'm aware that the SED solution will take a significantly long time to process such a huge file, hence I'm not opting it.

Please advise.

Thanks.

Warm Regards,
Kumarjit.
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Insert a line as the first line into a very huge file

Hello, I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it? Any help would be... (3 Replies)
Discussion started by: shriek
3 Replies

2. UNIX for Dummies Questions & Answers

How to remove FIRST Line of huge text file on Solaris

i need help..!!!! i have one big text file estimate data file size 50 - 100GB with 70 Mega Rows. on OS SUN Solaris version 8 How i can remove first line of the text file. Please suggest me for solutions. Thank you very much in advance:) (5 Replies)
Discussion started by: madoatz
5 Replies

3. Shell Programming and Scripting

[Solved] Problem in reading a file line by line till it reaches a white line

So, I want to read line-by-line a text file with unknown number of files.... So: a=1 b=1 while ; do b=`sed -n '$ap' test` a=`expr $a + 1` $here do something with b etc done the problem is that sed does not seem to recognise the $a, even when trying sed -n ' $a p' So, I cannot read... (3 Replies)
Discussion started by: hakermania
3 Replies

4. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

I have file which contains around 5000 lines. The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters. I want to delete the lines a> if it starts with 1 and if 576th postion is a digit i,e 0-9 or b> if it starts with 0 or 9(i,e header and footer) ... (4 Replies)
Discussion started by: millan
4 Replies

5. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

6. Shell Programming and Scripting

splitting a huge line of file into multiple lines with fixed number of columns

Hi, I have a huge file with a single line. But I want to break that line into lines of with each line having five columns. My file is like this: code: "hi","there","how","are","you?","It","was","great","working","with","you.","hope","to","work","you." I want it like this: code:... (1 Reply)
Discussion started by: rajsharma
1 Replies

7. Shell Programming and Scripting

Edit a Huge one line file

We have a huge file which has just one really large line; about 500 MB. I want to 1. Count all the occurrences of a phrase 2. Replace the phrase with another. Trying to open it using vi has not helped as it complains that it is too large. Can any script help? Please advise. Thank you, (12 Replies)
Discussion started by: kaushikadya
12 Replies

8. Shell Programming and Scripting

Reading line by line from live log file using while loop and considering only those lines start from

Hi, I want to read a live log file line by line and considering those line which start from time stamp; Below code I am using, which read line but throws an exception when comparing line that does not contain error code tail -F /logs/COMMON-ERROR.log | while read myline; do... (2 Replies)
Discussion started by: ketanraut
2 Replies

9. UNIX for Dummies Questions & Answers

Need to replace new line characters in a huge file

Hi , I would like to replace new line characters(\n) in a huge file of about 2 million records . I tried this one (:%s/\n//g) but it's hanging there and no result. Does this command do not work if the file is big. Please let me know if you have any other options Regards Raj (1 Reply)
Discussion started by: rajeevm
1 Replies
ReadBackwards(3pm)					User Contributed Perl Documentation					ReadBackwards(3pm)

NAME
File::ReadBackwards.pm -- Read a file backwards by lines. SYNOPSIS
use File::ReadBackwards ; # Object interface $bw = File::ReadBackwards->new( 'log_file' ) or die "can't read 'log_file' $!" ; while( defined( $log_line = $bw->readline ) ) { print $log_line ; } # ... or the alternative way of reading until ( $bw->eof ) { print $bw->readline ; } # Tied Handle Interface tie *BW, 'File::ReadBackwards', 'log_file' or die "can't read 'log_file' $!" ; while( <BW> ) { print ; } DESCRIPTION
This module reads a file backwards line by line. It is simple to use, memory efficient and fast. It supports both an object and a tied handle interface. It is intended for processing log and other similar text files which typically have their newest entries appended to them. By default files are assumed to be plain text and have a line ending appropriate to the OS. But you can set the input record separator string on a per file basis. OBJECT INTERFACE
These are the methods in "File::ReadBackwards"' object interface: new( $file, [$rec_sep], [$sep_is_regex] ) "new" takes as arguments a filename, an optional record separator and an optional flag that marks the record separator as a regular expression. It either returns the object on a successful open or undef upon failure. $! is set to the error code if any. readline "readline" takes no arguments and it returns the previous line in the file or undef when there are no more lines in the file. If the file is a non-seekable file (e.g. a pipe), then undef is returned. getline "getline" is an alias for the readline method. It is here for compatibility with the IO::* classes which has a getline method. eof "eof" takes no arguments and it returns true when readline() has iterated through the whole file. close "close" takes no arguments and it closes the handle tell "tell" takes no arguments and it returns the current filehandle position. This value may be used to seek() back to this position using a normal file handle. get_handle "get_handle" takes no arguments and it returns the internal Perl filehandle used by the File::ReadBackwards object. This handle may be used to read the file forward. Its seek position will be set to the position that is returned by the tell() method. Note that interleaving forward and reverse reads may produce unpredictable results. The only use supported at present is to read a file backward to a certain point, then use 'handle' to extract the handle, and read forward from that point. TIED HANDLE INTERFACE
tie( *HANDLE, 'File::ReadBackwards', $file, [$rec_sep], [$sep_is_regex] ) The TIEHANDLE, READLINE, EOF, CLOSE and TELL methods are aliased to the new, readline, eof, close and tell methods respectively so refer to them for their arguments and API. Once you have tied a handle to File::ReadBackwards the only I/O operation permissible is <> which will read the previous line. You can call eof() and close() on the tied handle as well. All other tied handle operations will generate an unknown method error. Do not seek, write or perform any other unsupported operations on the tied handle. LINE AND RECORD ENDINGS
Since this module needs to use low level I/O for efficiency, it can't portably seek and do block I/O without managing line ending conversions. This module supports the default record separators of normal line ending strings used by the OS. You can also set the separator on a per file basis. The record separator is a regular expression by default, which differs from the behavior of $/. Only if the record separator is not specified and it defaults to CR/LF (e.g, VMS, redmondware) will it will be converted to a single newline. Unix and MacOS files systems use only a single character for line endings and the lines are left unchanged. This means that for native text files, you should be able to process their lines backwards without any problems with line endings. If you specify a record separator, no conversions will be done and you will get the records as if you read them in binary mode. DESIGN
It works by reading a large(8kb) block of data from the end of the file. It then splits them on the record separator and stores a list of records in the object. Each call to readline returns the top record of the list and if the list is empty it refills it by reading the previous block from the file and splitting it. When the beginning of the file is reached and there are no more lines, undef is returned. All boundary conditions are handled correctly i.e. if there is a trailing partial line (no newline) it will be the first line returned and lines larger than the read buffer size are handled properly. NOTES
There is no support for list context in either the object or tied interfaces. If you want to slurp all of the lines into an array in backwards order (and you don't care about memory usage) just do: @back_lines = reverse <FH>. This module is only intended to read one line at a time from the end of a file to the beginning. AUTHOR
Uri Guttman, uri@stemsystems.com COPYRIGHT
Copyright (C) 2003 by Uri Guttman. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.12.4 2011-07-15 ReadBackwards(3pm)
All times are GMT -4. The time now is 11:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy