Sponsored Content
Top Forums Shell Programming and Scripting Match paragraph between two patterns, delete the duplicate paragraphs Post 302831055 by sb245 on Wednesday 10th of July 2013 06:28:20 AM
Old 07-10-2013
thanks vidyadhar85,

It removed all the duplicate lines. But the "BEGIN;" and "COMMIT;" were also needed which is needed in the file. The output came in below syntax.

Code:
BEGIN;
replace into domains (name,type) values ('blah blah 1,'MASTER');
COMMIT;
replace into domains (name,type) values ('blah blah 2','MASTER');
replace into domains (name,type) values ('blah blah 3','MASTER');
replace into domains (name,type) values ('blah blah 4','MASTER');

what I need is as below.


Code:
BEGIN;
replace into domains (name,type) values ('blah blah 1,'MASTER');
COMMIT;
BEGIN;
replace into domains (name,type) values ('blah blah 2','MASTER');
COMMIT;
BEGIN;
replace into domains (name,type) values ('blah blah 3','MASTER');
replace into domains (name,type) values ('blah blah 3','MASTER');
replace into domains (name,type) values ('blah blah 3','MASTER');
COMMIT;
BEGIN;
replace into domains (name,type) values ('blah blah 4','MASTER');
replace into domains (name,type) values ('blah blah 4','MASTER');
COMMIT;

Note: In the original file some paragraphs contain single line. Some contain multiple lines. The example I had pasted had multiple lines. Your script works perfectly for the example I had pasted.

TIA
sb
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

removing certain paragraphs for matching patterns

Hi, I have a log file which might have certain paragraphs. Switch not possible Error code 1234 Process number 678 Log not available Error code 567 Process number 874 ..... ...... ...... Now I create an exception file like this. cat text.exp Error code 1234 Process number 874 (7 Replies)
Discussion started by: kaushys
7 Replies

2. Shell Programming and Scripting

print lines which match multiple patterns

Hi, I have a text file as follows: 11:38:11.054 run1_rdseq avg_2-5 999988.0000 1024.0000 11:50:52.053 run3_rdrand 999988.0000 1135.0 128.0417 11:53:18.050 run4_wrrand avg_2-5 999988.0000 8180.5833 11:55:42.051 run4_wrrand avg_2-5 999988.0000 213.8333 11:55:06.053... (2 Replies)
Discussion started by: annazpereira
2 Replies

3. Shell Programming and Scripting

script to match patterns in 2 different files.

I am new to shell scripting and need some help. I googled, but couldn't find a similar scenario. Basically, I need to rename a datafile. This is the scenario - I have a file, readonly.txt that has 2 columns - file# and name. I have another file,missing_files.txt that has id and name. Both the... (3 Replies)
Discussion started by: mathews
3 Replies

4. Shell Programming and Scripting

Find files that do not match specific patterns

Hi all, I have been searching online to find the answer for getting a list of files that do not match certain criteria but have been unsuccessful. I have a directory that has many jpg files. What I need to do is get a list of the files that do not match both of the following patterns (I have... (21 Replies)
Discussion started by: nikos-koutax
21 Replies

5. Shell Programming and Scripting

Using AWK to match CSV files with duplicate patterns

Dear awk users, I am trying to use awk to match records across two moderately large CSV files. File1 is a pattern file with 173,200 lines, many of which are repeated. The order in which these lines are displayed is important, and I would like to preserve it. File2 is a data file with 456,000... (3 Replies)
Discussion started by: isuewing
3 Replies

6. Shell Programming and Scripting

Match 2 different patterns and print the lines

Hi, i have been trying to extract multiple lines based on two different patterns as below:- file1 @jkm|kdo|aas012|192.2.3.1 blablbalablablkabblablabla sjfdsakfjladfjefhaghfagfkafagkjsghfalhfk fhajkhfadjkhfalhflaffajkgfajkghfajkhgfkf jahfjkhflkhalfdhfwearhahfl @jkm|sdf|wud08q|168.2.1.3... (8 Replies)
Discussion started by: redse171
8 Replies

7. UNIX for Dummies Questions & Answers

Match patterns from another file and tag

Hi all, I have a file , which has 6 tab delimited fields, with $3 and $4 subfielded with spaces. I wamt to match cols $2,$3,$4 of tmp1 with tmp2, ..and then flag the 5th col if found. tmp1 1756 Xerm XermA XermB XermC XermD AA TT AA GG A 1 1763 Xerm XermA XermB XermC... (3 Replies)
Discussion started by: senhia83
3 Replies

8. Shell Programming and Scripting

Match 2 patterns together

How can I quickly print out lines in a datafile which has presence of both patterns in a row of another file. Maybe awk can do it much faster than bash. Patternfile ID1 PAT11 PAT12 ID1 PAT21 PAT22 ID2 PAT31 PAT32 datafile headerline... (2 Replies)
Discussion started by: abh.kumar
2 Replies

9. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

10. Shell Programming and Scripting

awk to print match or non-match and select fields/patterns for non-matches

In the awk below I am trying to output those lines that Match between file1 and file2, those Missing in file1, and those missing in file2. Using each $1,$2,$4,$5 value as a key to match on, that is if those 4 fields are found in both files the match, but if those 4 fields are not found then missing... (0 Replies)
Discussion started by: cmccabe
0 Replies
Template::Test(3)					User Contributed Perl Documentation					 Template::Test(3)

NAME
Template::Test - Module for automating TT2 test scripts SYNOPSIS
use Template::Test; $Template::Test::DEBUG = 0; # set this true to see each test running $Template::Test::EXTRA = 2; # 2 extra tests follow test_expect()... # ok() can be called any number of times before test_expect ok( $true_or_false ) # test_expect() splits $input into individual tests, processes each # and compares generated output against expected output test_expect($input, $template, \%replace ); # $input is text or filehandle (e.g. DATA section after __END__) test_expect( $text ); test_expect( *DATA ); # $template is a Template object or configuration hash my $template_cfg = { ... }; test_expect( $input, $template_cfg ); my $template_obj = Template->new($template_cfg); test_expect( $input, $template_obj ); # $replace is a hash reference of template variables my $replace = { a => 'alpha', b => 'bravo' }; test_expect( $input, $template, $replace ); # ok() called after test_expect should be declared in $EXTRA (2) ok( $true_or_false ) ok( $true_or_false ) DESCRIPTION
The "Template::Test" module defines the test_expect() and other related subroutines which can be used to automate test scripts for the Template Toolkit. See the numerous tests in the t sub-directory of the distribution for examples of use. PACKAGE SUBROUTINES
text_expect() The "test_expect()" subroutine splits an input document into a number of separate tests, processes each one using the Template Toolkit and then compares the generated output against an expected output, also specified in the input document. It generates the familiar "ok"/"not ok" output compatible with "Test::Harness". The test input should be specified as a text string or a reference to a filehandle (e.g. "GLOB" or "IO::Handle") from which it can be read. In particular, this allows the test input to be placed after the "__END__" marker and read via the "DATA" filehandle. use Template::Test; test_expect(*DATA); __END__ # this is the first test (this is a comment) -- test -- blah blah blah [% foo %] -- expect -- blah blah blah value_of_foo # here's the second test (no surprise, so is this) -- test -- more blah blah [% bar %] -- expect -- more blah blah value_of_bar Blank lines between test sections are generally ignored. Any line starting with "#" is treated as a comment and is ignored. The second and third parameters to "test_expect()" are optional. The second may be either a reference to a Template object which should be used to process the template fragments, or a reference to a hash array containing configuration values which should be used to instantiate a new Template object. # pass reference to config hash my $config = { INCLUDE_PATH => '/here/there:/every/where', POST_CHOMP => 1, }; test_expect(*DATA, $config); # or create Template object explicitly my $template = Template->new($config); test_expect(*DATA, $template); The third parameter may be used to reference a hash array of template variable which should be defined when processing the tests. This is passed to the Template process() method. my $replace = { a => 'alpha', b => 'bravo', }; test_expect(*DATA, $config, $replace); The second parameter may be left undefined to specify a default Template configuration. test_expect(*DATA, undef, $replace); For testing the output of different Template configurations, a reference to a list of named Template objects also may be passed as the second parameter. my $tt1 = Template->new({ ... }); my $tt2 = Template->new({ ... }); my @tts = [ one => $tt1, two => $tt1 ]; The first object in the list is used by default. Other objects may be switched in with a '"-- use $name --"' marker. This should immediately follow a '"-- test --"' line. That object will then be used for the rest of the test, or until a different object is selected. -- test -- -- use one -- [% blah %] -- expect -- blah, blah -- test -- still using one... -- expect -- ... -- test -- -- use two -- [% blah %] -- expect -- blah, blah, more blah The "test_expect()" sub counts the number of tests, and then calls ntests() to generate the familiar ""1..$ntests "" test harness line. Each test defined generates two test numbers. The first indicates that the input was processed without error, and the second that the output matches that expected. Additional test may be run before "test_expect()" by calling ok(). These test results are cached until ntests() is called and the final number of tests can be calculated. Then, the ""1..$ntests"" line is output, along with ""ok $n"" / ""not ok $n"" lines for each of the cached test result. Subsequent calls to ok() then generate an output line immediately. my $something = SomeObject->new(); ok( $something ); my $other = AnotherThing->new(); ok( $other ); test_expect(*DATA); If any tests are to follow after "test_expect()" is called then these should be pre-declared by setting the $EXTRA package variable. This value (default: 0) is added to the grand total calculated by ntests(). The results of the additional tests are also registered by calling ok(). $Template::Test::EXTRA = 2; # can call ok() any number of times before test_expect() ok( $did_that_work ); ok( $make_sure ); ok( $dead_certain ); # <some> number of tests... test_expect(*DATA, $config, $replace); # here's those $EXTRA tests ok( defined $some_result && ref $some_result eq 'ARRAY' ); ok( $some_result->[0] eq 'some expected value' ); If you don't want to call "test_expect()" at all then you can call "ntests($n)" to declare the number of tests and generate the test header line. After that, simply call ok() for each test passing a true or false values to indicate that the test passed or failed. ntests(2); ok(1); ok(0); If you're really lazy, you can just call ok() and not bother declaring the number of tests at all. All tests results will be cached until the end of the script and then printed in one go before the program exits. ok( $x ); ok( $y ); You can identify only a specific part of the input file for testing using the '"-- start --"' and '"-- stop --"' markers. Anything before the first '"-- start --"' is ignored, along with anything after the next '"-- stop --"' marker. -- test -- this is test 1 (not performed) -- expect -- this is test 1 (not performed) -- start -- -- test -- this is test 2 -- expect -- this is test 2 -- stop -- ... ntests() Subroutine used to specify how many tests you're expecting to run. ok($test) Generates an ""ok $n"" or ""not ok $n"" message if $test is true or false. not_ok($test) The logical inverse of ok(). Prints an ""ok $n"" message is $test is false and vice-versa. callsign() For historical reasons and general utility, the module also defines a "callsign()" subroutine which returns a hash mapping the letters "a" to "z" to their phonetic alphabet equivalent (e.g. radio callsigns). This is used by many of the test scripts as a known source of variable values. test_expect(*DATA, $config, callsign()); banner() This subroutine prints a simple banner including any text passed as parameters. The $DEBUG variable must be set for it to generate any output. banner('Testing something-or-other'); example output: #------------------------------------------------------------ # Testing something-or-other (27 tests completed) #------------------------------------------------------------ PACKAGE VARIABLES
$DEBUG The $DEBUG package variable can be set to enable debugging mode. $PRESERVE The $PRESERVE package variable can be set to stop the test_expect() from converting newlines in the output and expected output into the literal strings ' '. HISTORY
This module started its butt-ugly life as the "t/texpect.pl" script. It was cleaned up to became the "Template::Test" module some time around version 0.29. It underwent further cosmetic surgery for version 2.00 but still retains some remarkable rear-end resemblances. Since then the "Test::More" and related modules have appeared on CPAN making this module mostly, but not entirely, redundant. BUGS
/ KNOWN "FEATURES" Imports all methods by default. This is generally a Bad Thing, but this module is only used in test scripts (i.e. at build time) so a) we don't really care and b) it saves typing. The line splitter may be a bit dumb, especially if it sees lines like "-- this --" that aren't supposed to be special markers. So don't do that. AUTHOR
Andy Wardley <abw@wardley.org> <http://wardley.org/> COPYRIGHT
Copyright (C) 1996-2007 Andy Wardley. All Rights Reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO
Template perl v5.16.3 2011-12-20 Template::Test(3)
All times are GMT -4. The time now is 05:16 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy