Sponsored Content
Top Forums Shell Programming and Scripting Reading ALL BUT the first and last line of a huge file Post 302969946 by Scrutinizer on Wednesday 30th of March 2016 01:52:00 PM
Old 03-30-2016
Tested with a 2 GB file (excluding writing to a file which should be similar for all approaches):
Code:
$ time sed '1d;$d' greptestin1 > /dev/null

real	0m29.835s
user	0m29.186s
sys	0m0.591s
$ time awk 'NR>2{print p}{p=$0}' greptestin1 > /dev/null    # BSD awk

real	1m44.183s
user	1m43.627s
sys	0m0.481s
$ time mawk 'NR>2{print p}{p=$0}' greptestin1 > /dev/null

real	0m14.982s
user	0m14.463s
sys	0m0.498s
$ time gawk 'NR>2{print p}{p=$0}' greptestin1 > /dev/null

real	0m24.682s
user	0m24.210s
sys	0m0.414s
$ time gawk4 'NR>2{print p}{p=$0}' greptestin1 > /dev/null

real	0m27.621s
user	0m27.173s
sys	0m0.419s

This User Gave Thanks to Scrutinizer For This Post:
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Insert a line as the first line into a very huge file

Hello, I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it? Any help would be... (3 Replies)
Discussion started by: shriek
3 Replies

2. UNIX for Dummies Questions & Answers

How to remove FIRST Line of huge text file on Solaris

i need help..!!!! i have one big text file estimate data file size 50 - 100GB with 70 Mega Rows. on OS SUN Solaris version 8 How i can remove first line of the text file. Please suggest me for solutions. Thank you very much in advance:) (5 Replies)
Discussion started by: madoatz
5 Replies

3. Shell Programming and Scripting

[Solved] Problem in reading a file line by line till it reaches a white line

So, I want to read line-by-line a text file with unknown number of files.... So: a=1 b=1 while ; do b=`sed -n '$ap' test` a=`expr $a + 1` $here do something with b etc done the problem is that sed does not seem to recognise the $a, even when trying sed -n ' $a p' So, I cannot read... (3 Replies)
Discussion started by: hakermania
3 Replies

4. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

I have file which contains around 5000 lines. The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters. I want to delete the lines a> if it starts with 1 and if 576th postion is a digit i,e 0-9 or b> if it starts with 0 or 9(i,e header and footer) ... (4 Replies)
Discussion started by: millan
4 Replies

5. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

6. Shell Programming and Scripting

splitting a huge line of file into multiple lines with fixed number of columns

Hi, I have a huge file with a single line. But I want to break that line into lines of with each line having five columns. My file is like this: code: "hi","there","how","are","you?","It","was","great","working","with","you.","hope","to","work","you." I want it like this: code:... (1 Reply)
Discussion started by: rajsharma
1 Replies

7. Shell Programming and Scripting

Edit a Huge one line file

We have a huge file which has just one really large line; about 500 MB. I want to 1. Count all the occurrences of a phrase 2. Replace the phrase with another. Trying to open it using vi has not helped as it complains that it is too large. Can any script help? Please advise. Thank you, (12 Replies)
Discussion started by: kaushikadya
12 Replies

8. Shell Programming and Scripting

Reading line by line from live log file using while loop and considering only those lines start from

Hi, I want to read a live log file line by line and considering those line which start from time stamp; Below code I am using, which read line but throws an exception when comparing line that does not contain error code tail -F /logs/COMMON-ERROR.log | while read myline; do... (2 Replies)
Discussion started by: ketanraut
2 Replies

9. UNIX for Dummies Questions & Answers

Need to replace new line characters in a huge file

Hi , I would like to replace new line characters(\n) in a huge file of about 2 million records . I tried this one (:%s/\n//g) but it's hanging there and no result. Does this command do not work if the file is big. Please let me know if you have any other options Regards Raj (1 Reply)
Discussion started by: rajeevm
1 Replies
File::Path::Tiny(3pm)					User Contributed Perl Documentation				     File::Path::Tiny(3pm)

NAME
File::Path::Tiny - recursive versions of mkdir() and rmdir() without as much overhead as File::Path VERSION
This document describes File::Path::Tiny version 0.3 SYNOPSIS
use File::Path::Tiny; if(!File::Path::Tiny::mk($path)) { die "Could not make path '$path': $!"; } if(!File::Path::Tiny::rm($path)) { die "Could not remove path '$path': $!"; } DESCRIPTION
The goal here is simply to provide recursive versions of mkdir() and rmdir() with as little code and overhead as possible. This module is in no way meant to derogate File::Path and is in no way an endorsement to go out and replace all use of File::Path with File::Path::Tiny. File::Path is very good at what it does but there's simply a lot happening that we can do without much of the time. Here are some things File::Path has/does that this module attempts to do without: o multiple interfaces Backwards compatibility brings in a lot of code and logic that we don't need from here on out. o chdir's It does a ton of chdir's which could leave you somewhere you're not planning on being and requires much more overhead to do. o can croak not allowing you to detect and handle failure Just let me handle errors how I want. Don't make my entire app die or have to wrap it in an eval o A well intentioned output system Just let me do the output how I want. (Nothing, As HTML, print to a filehandle, etc...) o A well intentioned and experimental (IE subject to change) error handling system. Just keep it simple and detect failure via a boolean check and do what I want with the error. See "How can I make/remove multiple paths?" o According to its POD, removing a tree is apparently not safe unless you tell it to be with the 'safe' or 'keep_root' attributes. Seems like that should just happen, I don't want to worry about accidentally removing / when I pass it /tmp INTERFACE
Nothing in exported or exportable, that helps keep it '::Tiny'. File::Path::Tiny::mk() Takes a single path (like mkdir()) to recursively create and, optionally, a mask (like mkdir()) for all subsequent mkdir() calls. Mask defaults to 0700 (also like mkdir()) Returns false if it could not be made, true otherwise (returns '2' if it is -d already) It is recursive in the sense that given foo/bar/baz as the path to create all 3 will be created if necessary. File::Path::Tiny::rm() Takes a single path (like rmdir()) to recursively empty and remove. Returns false if it could not be emptied or removed, true otherwise. (returns '2' if it is !-d already) It is recursive in the sense that given /foo/bar/baz as the path to remove it will recursively empty 'baz' and then remove it from /foo/bar. Its parents, /, /foo, and /foo/bar are *not* touched. File::Path::Tiny::empty_dir() Takes a single path to recursively empty but not remove. Returns false when there is a problem. DIAGNOSTICS
Throws no warnings or errors of its own If the functions ever return false, $! will have been set either explicitly or by the mkdir(), rmdir(), unlink(), or opendir() that ultimately returned false. MISC
How can I make/remove multiple paths? For simplicity sake && ::Tiny status this module must be very very very simple. That said it is also very simple to do muliple paths: for my $path (@paths) { File::Path::Tiny::::mk($path) or _my_handle_failed_mk($path, $!); } for my $path (@paths) { File::Path::Tiny::::rm($path) or _my_handle_failed_rm($path, $!); } That also lets you keep going or short circuit it or handle errors however you like: PATH: for my $path qw(foo/bar bar/rat) { if (!File::Path::Tiny::mk($path)) { print "problem unlinking '$path': $! "; last PATH; } else { print "No error encountered with '$path' " } } About the '::Tiny' status See Acme::Tiny for information on the ::Tiny suffix. #3 is almost there (< 1/5th +/-), a bit more trimming and I think we'll have it! #8 is N/A since part of the "sub set" is to do single paths like their non-recursive core counterparts and there are so many ways to invoke it with different consequences [ -- RSS Memory -- ] Baseline perl 1168 File::Path 1808 (+640) File::Path::Tiny 1288 (+120) Even though "time" isn't really a ::Tiny factor, it does improve loading a bit: [ -- time -- ] Baseline perl real 0m0.007s user 0m0.002s sys 0m0.004s File::Path real 0m0.017s user 0m0.011s sys 0m0.005s File::Path::Tiny real 0m0.007s user 0m0.003s sys 0m0.004s As time allows and more tests are added I'll try to include more comprehensive benchmark results. How do I make sure the path is safe to create or remove? Of course the answer depends on what you mean by "safe". This module makes no assumptions on interpreting the "safeness" of a path, just like mkdir() and rmdir(). Also like mkdir() and rmdir() typically you'll find that filesystem permissions are a pretty reliable tool (of course if the code will be run as root you would want to setuid first...) You might use Cwd::abs_path() to sanitize a path before sending it to be made or removed. Even after that it might not be "safe" so you'll need to discern what your particular definition of "safe" is and take appropriate action. DEPENDENCIES
File::Spec of course but its only loaded if needed INCOMPATIBILITIES
None reported. BUGS AND LIMITATIONS
No bugs have been reported. Please report any bugs or feature requests to "bug-file-path-tiny@rt.cpan.org", or through the web interface at <http://rt.cpan.org>. AUTHOR
Daniel Muey "<http://drmuey.com/cpan_contact.pl>" LICENCE AND COPYRIGHT
Copyright (c) 2008, Daniel Muey "<http://drmuey.com/cpan_contact.pl>". All rights reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic. DISCLAIMER OF WARRANTY
BECAUSE THIS SOFTWARE IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE SOFTWARE, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE SOFTWARE "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE IS WITH YOU. SHOULD THE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR, OR CORRECTION. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE SOFTWARE AS PERMITTED BY THE ABOVE LICENCE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE SOFTWARE (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE SOFTWARE TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. perl v5.14.2 2012-04-04 File::Path::Tiny(3pm)
All times are GMT -4. The time now is 09:32 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy