Performance issue with 'grep' command for huge file size
I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is: Above code is working fine and I am getting expected result. But, this code is taking too much time (I don't have exact time, more than 6 hrs, later on cancelled the script) while the file size is huge. As an example, I have details.txt of around 2.5GB and record count is around 7.5lacs and the emp.txt has 55K employee name. Can you please suggest any other option/ command which will be better to handle such huge file. Thanks.
Last edited by vbe; 11-18-2011 at 09:33 AM..
Reason: attempt to use code tags ( fighting with fonts...)
111111111100000000001111111111
123232323200000010001114545454
232435424200000000001232131212
342354234301000000002323423443
232435424200000000001232131212
2390898994200000000001238908092
This is the record format.
From 11th position to 20th position in a record there are 0's occuring,and... (6 Replies)
Hi,
I have to find out the run time for 40-45 different componets. These components writes in to a genreric log file in a single directory.
eg.
directory is LOG and the log file name format is generic_log_<process_id>_<date YY_MM_DD_HH_MM_SS>.log
i am taking the run time using the time... (3 Replies)
I have file which contains around 5000 lines.
The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters.
I want to delete the lines
a> if it starts with 1 and if 576th postion is a digit i,e 0-9
or
b> if it starts with 0 or 9(i,e header and footer)
... (4 Replies)
Background
-------------
The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files.
File-1
------
Contains 50,000 rows with 2 fields in each row, separated by pipe.
Row structure is like Object_Id|Object_Name, as following:
111|XXX
222|YYY
333|ZZZ
... (6 Replies)
Dear All,
Good Evening!!
I have a requirement to ftp a 220GB backup file to a remote backup server.
I wrote a script for this purpose.
But it takes more than 8 hours to transfer this file.
Is there any other method to do it in less time???
Thanks in Advance!!!
---------- Post updated... (5 Replies)
Hi Experts,
I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Hi,
I have created a shell script for Server Log Automation Process. I have used
find xargs grep command to search the string.
for Example,
find -name | xargs grep "816995225" > test.txt .
Here my problem is,
We have lot of records and we want to grep the string... (4 Replies)
I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file.
I'm trying and already tried out a variety of options posted in similar threads/forums. But no luck so far..
Any suggestions please ?
Thanks !! (9 Replies)
Hi All,
I am new to this forum and this is my first post.
My requirement is like to optimize the time taken to grep the file with 40000 lines.
There are two files FILEA(40000 lines) FILEB(40000 lines).
The requirement is like this, both the file will be in the format below... (11 Replies)
Hi All,
This query is regarding performance improvement of a command.
I have a list of IDs in a file (say file1 with single ID column) and file2 has the data rows.
I need to get the IDs from file1 and search in file2, matching rows from file2 should be written to a file3.
For this... (4 Replies)
Discussion started by: Tanu
4 Replies
LEARN ABOUT MOJAVE
moose::cookbook::roles::applicationtoinstance5.18
Moose::Cookbook::Roles::ApplicationToInstance(3) User Contributed Perl Documentation Moose::Cookbook::Roles::ApplicationToInstance(3)NAME
Moose::Cookbook::Roles::ApplicationToInstance - Applying a role to an object instance
VERSION
version 2.1202
SYNOPSIS
package MyApp::Role::Job::Manager;
use List::Util qw( first );
use Moose::Role;
has 'employees' => (
is => 'rw',
isa => 'ArrayRef[Employee]',
);
sub assign_work {
my $self = shift;
my $work = shift;
my $employee = first { !$_->has_work } @{ $self->employees };
die 'All my employees have work to do!' unless $employee;
$employee->work($work);
}
package main;
my $lisa = Employee->new( name => 'Lisa' );
MyApp::Role::Job::Manager->meta->apply($lisa);
my $homer = Employee->new( name => 'Homer' );
my $bart = Employee->new( name => 'Bart' );
my $marge = Employee->new( name => 'Marge' );
$lisa->employees( [ $homer, $bart, $marge ] );
$lisa->assign_work('mow the lawn');
DESCRIPTION
In this recipe, we show how a role can be applied to an object. In this specific case, we are giving an employee managerial
responsibilities.
Applying a role to an object is simple. The Moose::Meta::Role object provides an "apply" method. This method will do the right thing when
given an object instance.
MyApp::Role::Job::Manager->meta->apply($lisa);
We could also use the "apply_all_roles" function from Moose::Util.
apply_all_roles( $person, MyApp::Role::Job::Manager->meta );
The main advantage of using "apply_all_roles" is that it can be used to apply more than one role at a time.
We could also pass parameters to the role we're applying:
MyApp::Role::Job::Manager->meta->apply(
$lisa,
-alias => { assign_work => 'get_off_your_lazy_behind' },
);
We saw examples of how method exclusion and alias working in Moose::Cookbook::Roles::Restartable_AdvancedComposition.
CONCLUSION
Applying a role to an object instance is a useful tool for adding behavior to existing objects. In our example, it is effective used to
model a promotion.
It can also be useful as a sort of controlled monkey-patching for existing code, particularly non-Moose code. For example, you could create
a debugging role and apply it to an object at runtime.
AUTHORS
o Stevan Little <stevan.little@iinteractive.com>
o Dave Rolsky <autarch@urth.org>
o Jesse Luehrs <doy@tozt.net>
o Shawn M Moore <code@sartak.org>
o XXXX XXX'XX (Yuval Kogman) <nothingmuch@woobling.org>
o Karen Etheridge <ether@cpan.org>
o Florian Ragwitz <rafl@debian.org>
o Hans Dieter Pearcey <hdp@weftsoar.net>
o Chris Prather <chris@prather.org>
o Matt S Trout <mst@shadowcat.co.uk>
COPYRIGHT AND LICENSE
This software is copyright (c) 2006 by Infinity Interactive, Inc..
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.
perl v5.18.2 2014-01-19 Moose::Cookbook::Roles::ApplicationToInstance(3)