Sponsored Content
Top Forums Shell Programming and Scripting Want to extract certain lines from big file Post 302965064 by Don Cragun on Friday 22nd of January 2016 10:09:59 PM
Old 01-22-2016
In addition to what RudiC already said, if every transaction contains the literal line:
Code:
%%blah~transnum~blah~blah~blah

how do you know which transnum set you want? We might guess that blah isn't literal and we might guess that transnum isn't literal and that transnum is different in each set, but you haven't given us enough information to make a reasonable guess at a BRE that will match the transum set you want.

Please give us:
  1. some more realistic sample data,
  2. a description of any file(s) that your script is expected to read,
  3. a description of any file(s) that your script is expected to write,
  4. a description of any arguments you intend to pass to your script,
  5. the operating system and shell you're using, and
  6. the exact output you want your script to produce with the sample data provided in #1 above and sample arguments you provided in $4 above.
(And, don't forget to use CODE tags when showing us your sample input, sample output, and your attempts at writing a script to perform these tasks.)
This User Gave Thanks to Don Cragun For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to view a big file(143M big)

1 . Thanks everyone who read the post first. 2 . I have a log file which size is 143M , I can not use vi open it .I can not use xedit open it too. How to view it ? If I want to view 200-300 ,how can I implement it 3 . Thanks (3 Replies)
Discussion started by: chenhao_no1
3 Replies

2. UNIX for Dummies Questions & Answers

How big is too big a config.log file?

I have a 5000 line config.log file with several "maybe" errors. Any reccomendations on finding solvable problems? (2 Replies)
Discussion started by: NeedLotsofHelp
2 Replies

3. Shell Programming and Scripting

Print #of lines after search string in a big file

I have a command which prints #lines after and before the search string in the huge file nawk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r;print;c=a}b{r=$0}' b=0 a=10 s="STRING1" FILE The file is 5 gig big. It works great and prints 10 lines after the lines which contains search string in... (8 Replies)
Discussion started by: prash184u
8 Replies

4. Shell Programming and Scripting

Re: Deleting lines from big file.

Hi, I have a big (2.7 GB) text file. Each lines has '|' saperator to saperate each columns. I want to delete those lines which has text like '|0|0|0|0|0' I tried: sed '/|0|0|0|0|0/d' test.txt Unfortunately, it scans the file but does nothing. file content sample:... (4 Replies)
Discussion started by: dipeshvshah
4 Replies

5. Shell Programming and Scripting

Extract some lines from one file and add those lines to current file

hi, i have two files. file1.sh echo "unix" echo "linux" file2.sh echo "unix linux forums" now the output i need is $./file2.sh unix linux forums (3 Replies)
Discussion started by: snreddy_gopu
3 Replies

6. UNIX for Advanced & Expert Users

Delete first 100 lines from a BIG File

Hi, I need a unix command to delete first n (say 100) lines from a log file. I need to delete some lines from the file without using any temporary file. I found sed -i is an useful command for this but its not supported in my environment( AIX 6.1 ). File size is approx 100MB. Thanks in... (18 Replies)
Discussion started by: unohu
18 Replies

7. Shell Programming and Scripting

Extract certain entries from big file:Request to check

Hi all I have a big file which I have attached here. And, I have to fetch certain entries and arrange in 5 columns Name Drug DAP ID disease approved or notIn the attached file data is arranged with tab separated columns in this way: and other data is... (2 Replies)
Discussion started by: manigrover
2 Replies

8. Shell Programming and Scripting

Extract certain columns from big data

The dataset I'm working on is about 450G, with about 7000 colums and 30,000,000 rows. I want to extract about 2000 columns from the original file to form a new file. I have the list of number of the columns I need, but don't know how to extract them. Thanks! (14 Replies)
Discussion started by: happypoker
14 Replies

9. UNIX for Beginners Questions & Answers

How to copy only some lines from very big file?

Dear all, I have stuck with this problem for some days. I have a very big file, this file can not open by vi command. There are 200 loops in this file, in each loop will have one line like this: GWA quasiparticle energy with Z factor (eV) And I need 98 lines next after this line. Is... (6 Replies)
Discussion started by: phamnu
6 Replies

10. Shell Programming and Scripting

Extract Big and continuous regions

Hi all, I have a file like this I want to extract only those regions which are big and continous chr1 3280000 3440000 chr1 3440000 3920000 chr1 3600000 3920000 # region coming within the 3440000 3920000. so i don't want it to be printed in output chr1 3920000 4800000 chr1 ... (2 Replies)
Discussion started by: amrutha_sastry
2 Replies
Template::Test(3pm)					User Contributed Perl Documentation				       Template::Test(3pm)

NAME
Template::Test - Module for automating TT2 test scripts SYNOPSIS
use Template::Test; $Template::Test::DEBUG = 0; # set this true to see each test running $Template::Test::EXTRA = 2; # 2 extra tests follow test_expect()... # ok() can be called any number of times before test_expect ok( $true_or_false ) # test_expect() splits $input into individual tests, processes each # and compares generated output against expected output test_expect($input, $template, \%replace ); # $input is text or filehandle (e.g. DATA section after __END__) test_expect( $text ); test_expect( *DATA ); # $template is a Template object or configuration hash my $template_cfg = { ... }; test_expect( $input, $template_cfg ); my $template_obj = Template->new($template_cfg); test_expect( $input, $template_obj ); # $replace is a hash reference of template variables my $replace = { a => 'alpha', b => 'bravo' }; test_expect( $input, $template, $replace ); # ok() called after test_expect should be declared in $EXTRA(2) ok( $true_or_false ) ok( $true_or_false ) DESCRIPTION
The "Template::Test" module defines the test_expect() and other related subroutines which can be used to automate test scripts for the Template Toolkit. See the numerous tests in the t sub-directory of the distribution for examples of use. PACKAGE SUBROUTINES
text_expect() The "test_expect()" subroutine splits an input document into a number of separate tests, processes each one using the Template Toolkit and then compares the generated output against an expected output, also specified in the input document. It generates the familiar "ok"/"not ok" output compatible with "Test::Harness". The test input should be specified as a text string or a reference to a filehandle (e.g. "GLOB" or "IO::Handle") from which it can be read. In particular, this allows the test input to be placed after the "__END__" marker and read via the "DATA" filehandle. use Template::Test; test_expect(*DATA); __END__ # this is the first test (this is a comment) -- test -- blah blah blah [% foo %] -- expect -- blah blah blah value_of_foo # here's the second test (no surprise, so is this) -- test -- more blah blah [% bar %] -- expect -- more blah blah value_of_bar Blank lines between test sections are generally ignored. Any line starting with "#" is treated as a comment and is ignored. The second and third parameters to "test_expect()" are optional. The second may be either a reference to a Template object which should be used to process the template fragments, or a reference to a hash array containing configuration values which should be used to instantiate a new Template object. # pass reference to config hash my $config = { INCLUDE_PATH => '/here/there:/every/where', POST_CHOMP => 1, }; test_expect(*DATA, $config); # or create Template object explicitly my $template = Template->new($config); test_expect(*DATA, $template); The third parameter may be used to reference a hash array of template variable which should be defined when processing the tests. This is passed to the Template process() method. my $replace = { a => 'alpha', b => 'bravo', }; test_expect(*DATA, $config, $replace); The second parameter may be left undefined to specify a default Template configuration. test_expect(*DATA, undef, $replace); For testing the output of different Template configurations, a reference to a list of named Template objects also may be passed as the second parameter. my $tt1 = Template->new({ ... }); my $tt2 = Template->new({ ... }); my @tts = [ one => $tt1, two => $tt1 ]; The first object in the list is used by default. Other objects may be switched in with a '"-- use $name --"' marker. This should immediately follow a '"-- test --"' line. That object will then be used for the rest of the test, or until a different object is selected. -- test -- -- use one -- [% blah %] -- expect -- blah, blah -- test -- still using one... -- expect -- ... -- test -- -- use two -- [% blah %] -- expect -- blah, blah, more blah The "test_expect()" sub counts the number of tests, and then calls ntests() to generate the familiar ""1..$ntests "" test harness line. Each test defined generates two test numbers. The first indicates that the input was processed without error, and the second that the output matches that expected. Additional test may be run before "test_expect()" by calling ok(). These test results are cached until ntests() is called and the final number of tests can be calculated. Then, the ""1..$ntests"" line is output, along with ""ok $n"" / ""not ok $n"" lines for each of the cached test result. Subsequent calls to ok() then generate an output line immediately. my $something = SomeObject->new(); ok( $something ); my $other = AnotherThing->new(); ok( $other ); test_expect(*DATA); If any tests are to follow after "test_expect()" is called then these should be pre-declared by setting the $EXTRA package variable. This value (default: 0) is added to the grand total calculated by ntests(). The results of the additional tests are also registered by calling ok(). $Template::Test::EXTRA = 2; # can call ok() any number of times before test_expect() ok( $did_that_work ); ok( $make_sure ); ok( $dead_certain ); # <some> number of tests... test_expect(*DATA, $config, $replace); # here's those $EXTRA tests ok( defined $some_result && ref $some_result eq 'ARRAY' ); ok( $some_result->[0] eq 'some expected value' ); If you don't want to call "test_expect()" at all then you can call "ntests($n)" to declare the number of tests and generate the test header line. After that, simply call ok() for each test passing a true or false values to indicate that the test passed or failed. ntests(2); ok(1); ok(0); If you're really lazy, you can just call ok() and not bother declaring the number of tests at all. All tests results will be cached until the end of the script and then printed in one go before the program exits. ok( $x ); ok( $y ); You can identify only a specific part of the input file for testing using the '"-- start --"' and '"-- stop --"' markers. Anything before the first '"-- start --"' is ignored, along with anything after the next '"-- stop --"' marker. -- test -- this is test 1 (not performed) -- expect -- this is test 1 (not performed) -- start -- -- test -- this is test 2 -- expect -- this is test 2 -- stop -- ... ntests() Subroutine used to specify how many tests you're expecting to run. ok($test) Generates an ""ok $n"" or ""not ok $n"" message if $test is true or false. not_ok($test) The logical inverse of ok(). Prints an ""ok $n"" message is $test is false and vice-versa. callsign() For historical reasons and general utility, the module also defines a "callsign()" subroutine which returns a hash mapping the letters "a" to "z" to their phonetic alphabet equivalent (e.g. radio callsigns). This is used by many of the test scripts as a known source of variable values. test_expect(*DATA, $config, callsign()); banner() This subroutine prints a simple banner including any text passed as parameters. The $DEBUG variable must be set for it to generate any output. banner('Testing something-or-other'); example output: #------------------------------------------------------------ # Testing something-or-other (27 tests completed) #------------------------------------------------------------ PACKAGE VARIABLES
$DEBUG The $DEBUG package variable can be set to enable debugging mode. $PRESERVE The $PRESERVE package variable can be set to stop the test_expect() from converting newlines in the output and expected output into the literal strings ' '. HISTORY
This module started its butt-ugly life as the "t/texpect.pl" script. It was cleaned up to became the "Template::Test" module some time around version 0.29. It underwent further cosmetic surgery for version 2.00 but still retains some remarkable rear-end resemblances. Since then the "Test::More" and related modules have appeared on CPAN making this module mostly, but not entirely, redundant. BUGS
/ KNOWN "FEATURES" Imports all methods by default. This is generally a Bad Thing, but this module is only used in test scripts (i.e. at build time) so a) we don't really care and b) it saves typing. The line splitter may be a bit dumb, especially if it sees lines like "-- this --" that aren't supposed to be special markers. So don't do that. AUTHOR
Andy Wardley <abw@wardley.org> <http://wardley.org/> COPYRIGHT
Copyright (C) 1996-2007 Andy Wardley. All Rights Reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO
Template perl v5.14.2 2011-12-20 Template::Test(3pm)
All times are GMT -4. The time now is 06:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy