Sponsored Content
Top Forums Shell Programming and Scripting In memory grep of a large file. Post 302960518 by RudiC on Monday 16th of November 2015 01:02:55 PM
Old 11-16-2015
Did you try the proposals on a reduced data set (just to prove the applicability)?

Last edited by RudiC; 11-16-2015 at 02:32 PM.. Reason: typo...
 

10 More Discussions You Might Find Interesting

1. Linux

shmat() Failure While Using a Large Amount of Shared Memory

Hi, I'm developing a data processing pipeline with multiple stages, with data being moved between the stages using shared memory segments. The size of the data is typically of the order of hundreds of megabytes, and there are typically a few tens of main shared memory segments each of size... (2 Replies)
Discussion started by: theicarusagenda
2 Replies

2. HP-UX

How can I get memory usage or anything that show memory used from sar file?

Refer from title: How can i get memory used or anything that can show memory from sar file example on solaris:- we can use sar with option to show memory used at time that sar crontab run. on HP-UX, it not has option to see memory used. But i think it may be have some parameter or some... (1 Reply)
Discussion started by: panithat
1 Replies

3. UNIX for Dummies Questions & Answers

Grep alternative to handle large numbers of files

I am looking for a file with 'MCR0000000716214' in it. I tried the following command: grep MCR0000000716214 * The problem is that the folder I am searching in has over 87000 files and I am getting the following: bash: /bin/grep: Arg list too long Is there any command I can use that can... (6 Replies)
Discussion started by: runnerpaul
6 Replies

4. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

5. Shell Programming and Scripting

grep error: range endpoint too large

Hi, my problem: gzgrep "^.\{376\}8301685001120" filename /dev/null ###ERROR ### grep: RE error 11: Range endpoint too large. Whats my mistake? Is the position 376 to large for grep??? Thanks. (2 Replies)
Discussion started by: Timmää
2 Replies

6. Shell Programming and Scripting

grep/fgrep/egrep for a very large matrix

All, I have a problem with grep/fgrep/egrep. Basically I am building a 200 times 200 correlation matrix. The entries of this matrix need to be retrieved from another very large matrix (~100G). I tried to use the grep/fgrep/egrep to locate each entry and put them into one file. It looks very... (1 Reply)
Discussion started by: realwindfly
1 Replies

7. UNIX for Advanced & Expert Users

Out of Memory error when free memory size is large

I was running a program and it stopped and showed "Out of Memory!". at that time, the RAM used by this process is around 4G and the free memory size of the machine is around 30G. Does anybody know what maybe the reason? this program is written with Perl. the OS of the machine is Solaris U8. And I... (1 Reply)
Discussion started by: lilili07
1 Replies

8. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

9. UNIX for Dummies Questions & Answers

virtual memory and diff'ing very large files

(0 Replies)
Discussion started by: uiop44
0 Replies

10. Shell Programming and Scripting

Large search replace using sed results in memory problem.

I have one big file of size 9GB (big_file.txt). This big file has sentences and paragraphs like any usual English document. I have another file consisting of replacement strings for sed to use. The file name is replace.sed and each entry in one line looks like this: s/\<shout\>/shout/g s/\<b is... (2 Replies)
Discussion started by: shoaibjameel123
2 Replies
App::Prove(3pm) 					User Contributed Perl Documentation					   App::Prove(3pm)

NAME
App::Prove - Implements the "prove" command. VERSION
Version 3.25 DESCRIPTION
Test::Harness provides a command, "prove", which runs a TAP based test suite and prints a report. The "prove" command is a minimal wrapper around an instance of this module. SYNOPSIS
use App::Prove; my $app = App::Prove->new; $app->process_args(@ARGV); $app->run; METHODS
Class Methods "new" Create a new "App::Prove". Optionally a hash ref of attribute initializers may be passed. "state_class" Getter/setter for the name of the class used for maintaining state. This class should either subclass from "App::Prove::State" or provide an identical interface. "state_manager" Getter/setter for the instance of the "state_class". "add_rc_file" $prove->add_rc_file('myproj/.proverc'); Called before "process_args" to prepend the contents of an rc file to the options. "process_args" $prove->process_args(@args); Processes the command-line arguments. Attributes will be set appropriately. Any filenames may be found in the "argv" attribute. Dies on invalid arguments. "run" Perform whatever actions the command line args specified. The "prove" command line tool consists of the following code: use App::Prove; my $app = App::Prove->new; $app->process_args(@ARGV); exit( $app->run ? 0 : 1 ); # if you need the exit code "require_harness" Load a harness replacement class. $prove->require_harness($for => $class_name); "print_version" Display the version numbers of the loaded TAP::Harness and the current Perl. Attributes After command line parsing the following attributes reflect the values of the corresponding command line switches. They may be altered before calling "run". "archive" "argv" "backwards" "blib" "color" "directives" "dry" "exec" "extensions" "failures" "comments" "formatter" "harness" "ignore_exit" "includes" "jobs" "lib" "merge" "modules" "parse" "plugins" "quiet" "really_quiet" "recurse" "rules" "show_count" "show_help" "show_man" "show_version" "shuffle" "state" "state_class" "taint_fail" "taint_warn" "test_args" "timer" "verbose" "warnings_fail" "warnings_warn" "tapversion" "trap" PLUGINS
"App::Prove" provides support for 3rd-party plugins. These are currently loaded at run-time, after arguments have been parsed (so you can not change the way arguments are processed, sorry), typically with the "-Pplugin" switch, eg: prove -PMyPlugin This will search for a module named "App::Prove::Plugin::MyPlugin", or failing that, "MyPlugin". If the plugin can't be found, "prove" will complain & exit. You can pass an argument to your plugin by appending an "=" after the plugin name, eg "-PMyPlugin=foo". You can pass multiple arguments using commas: prove -PMyPlugin=foo,bar,baz These are passed in to your plugin's "load()" class method (if it has one), along with a reference to the "App::Prove" object that is invoking your plugin: sub load { my ($class, $p) = @_; my @args = @{ $p->{args} }; # @args will contain ( 'foo', 'bar', 'baz' ) $p->{app_prove}->do_something; ... } Note that the user's arguments are also passed to your plugin's "import()" function as a list, eg: sub import { my ($class, @args) = @_; # @args will contain ( 'foo', 'bar', 'baz' ) ... } This is for backwards compatibility, and may be deprecated in the future. Sample Plugin Here's a sample plugin, for your reference: package App::Prove::Plugin::Foo; # Sample plugin, try running with: # prove -PFoo=bar -r -j3 # prove -PFoo -Q # prove -PFoo=bar,My::Formatter use strict; use warnings; sub load { my ($class, $p) = @_; my @args = @{ $p->{args} }; my $app = $p->{app_prove}; print "loading plugin: $class, args: ", join(', ', @args ), " "; # turn on verbosity $app->verbose( 1 ); # set the formatter? $app->formatter( $args[1] ) if @args > 1; # print some of App::Prove's state: for my $attr (qw( jobs quiet really_quiet recurse verbose )) { my $val = $app->$attr; $val = 'undef' unless defined( $val ); print "$attr: $val "; } return 1; } 1; SEE ALSO
prove, TAP::Harness perl v5.14.2 2012-06-05 App::Prove(3pm)
All times are GMT -4. The time now is 07:50 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy