Sponsored Content
Top Forums Shell Programming and Scripting Extract multiple line text from log between special words Post 303045618 by sast1a on Friday 3rd of April 2020 08:33:39 PM
Old 04-03-2020
Extract multiple line text from log between special words

Hi all,

I have a text file (log), it prints some SQL statements in the log. It is a bit irregular, I need to extract SQL statement into as separate file.

1. Some statements are printed as just SQL statements, in multiple lines.
2. Some SQL statements are printed line by line but after log details at the beginning of the line. So somehow I need to ignore the first part of the line but need to keep query.

What I found as a pattern is, all SQL statements have "DISTINCT" keyword in the beginning
and after each statement there are "quit;" statements.
This doesn't have to be at the beginning of the lines, it's enough for me to extract text between DISTINCT and quit.

This is where I confused and couldn't find a way to solve it with sed or awk.

Sample log structure 1:

Code:
/* clear syscc before new query */
%let syscc=0;
%let sysrc=0;
/* MAmodel_C */
SELECT  DISTINCT
        table0.SUBSCRIBER_KEY AS SUBSCRIBER_KEY LABEL='Maestro Subscriber Key' FORMAT=F13.0
FROM
        CMD.ACCOUNTS_MAESTRO table0 Inner join CMD.PRODUCT_MAESTRO table1 on  ( ( (table1.ACCOUNT_ID = table0.ACCOUNT_ID AND table1.ACCOUNT_KEY = table0.ACCOUNT_KEY)  AND table1.COMPANY_NUMBER = table0.COMPANY_NUMBER)  AND table1.SUBSCRIBER_KEY = table0.SUBSCRIBER_KEY)
WHERE
        table1.PRODUCT_DISCOUNT_CODE IS NULL
) table2 ;
quit;
-----------------------------------------------------
2020-03-17 09:46:11,531 INFO

Structure 2: (each line is after log details)

Code:
2020-03-17 09:45:57,733 INFO  [Exec-11] [02205e834:-68e0] [user1140] .execution.instructions - 2FD66078-1B0B: SELECT  DISTINCT
2020-03-17 09:45:57,733 INFO  [Exec-11] [02205e834:-68e0] [user1140] .execution.instructions - 2FD66078-1B0B:        table0.SUBSCRIBER_KEY AS SUBSCRIBER_KEY LABEL='Maestro Subscriber Key' FORMAT=F13.0
2020-03-17 09:45:57,733 INFO  [Exec-11] [02205e834:-68e0] [user1140] .execution.instructions - 2FD66078-1B0B: FROM
2020-03-17 09:45:57,733 INFO  [Exec-11] [02205e834:-68e0] [user1140] .execution.instructions - 2FD66078-1B0B:        CMDM.ACCOUNTS_MAESTRO table0 Inner join CMDM.PRODUCT_MAESTRO table1 on  ( ( (table1.ACCOUNT_ID = table0.ACCOUNT_ID AND table1.ACCOUNT_KEY = table0.ACCOUNT_KEY)  AND table1.COMPANY_NUMBER = table0.COMPANY_NUMBER)  AND table1.SUBSCRIBER_KEY = table0.SUBSCRIBER_KEY)
2020-03-17 09:45:57,734 INFO  [Exec-11] [02205e834:-68e0] [user1140] .execution.instructions - 2FD66078-1B0B: WHERE
2020-03-17 09:45:57,734 INFO  [Exec-11] [02205e834:-68e0] [user1140] .execution.instructions - 2FD66078-1B0B:        table1.PRODUCT_DISCOUNT_CODE IS NULL
2020-03-17 09:45:57,734 INFO  [Exec-11] [02205e834:-68e0] [user1140] .execution.instructions - 2FD66078-1B0B: ) table2 ;
2020-03-17 09:45:57,734 INFO  [Exec-11] [02205e834:-68e0] [user1140] .execution.instructions - 2FD66078-1B0B: quit;

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

grep on multiple words to match text template

hi, I have few text templates as a simple ex: template 1 city Name: zip code: state Name: template2: employee Name: Phone number: I wish to grep on given text file and make sure the text file matches one of these templates. Please give your ideas. (6 Replies)
Discussion started by: rider29
6 Replies

2. Shell Programming and Scripting

grep multiple words in a single line

Hi.. How to search for multiple words in a single line using grep?. Eg: Jack and Jill went up the hill Jack and Jill were best friends Humpty and Dumpty were good friends too ---------- I want to extract the 2nd statement(assuming there are several statements with... (11 Replies)
Discussion started by: anduzzi
11 Replies

3. Shell Programming and Scripting

Extract X words from end of line, minus last keynumber X

The file contains one line of text followed by a number. I want to take the number X at the end, take it out and display the last X words. X is the key telling me how many words from the end that I want and X will always be less than the number of words, so no problem there. Example input and... (4 Replies)
Discussion started by: fubaya
4 Replies

4. UNIX for Dummies Questions & Answers

extract text between two words on a single line

Hi Guys, Can someone help me with a way to extract text between two words on a single line. For example if the file has below content I want to extract all text between b and f inclusive of b and f. Aparently sed does this but does it line by line and I guess it cannot read word by word. ... (11 Replies)
Discussion started by: krishnaux
11 Replies

5. UNIX for Dummies Questions & Answers

Searching for multiple words on a line in any order issue

Hi again I have figured out how to be able to sort through lines in a file with multiple words in any order and display them using this command: cat file | grep -i $OPTION1 | grep -i $OPTION2 | grep -i $OPTION3 OPTION1 is 2008, OPTION2 is Mar, OPTION 3 is Tue Result: Tue Mar 25... (4 Replies)
Discussion started by: semaj
4 Replies

6. Shell Programming and Scripting

Combine multiple unique lines from event log text file into one line, use PERL or AWK?

I can't decide if I should use AWK or PERL after pouring over these forums for hours today I decided I'd post something and see if I couldn't get some advice. I've got a text file full of hundreds of events in this format: Record Number : 1 Records in Seq : ... (3 Replies)
Discussion started by: Mayday22
3 Replies

7. Shell Programming and Scripting

I need to extract uique words from text file

Hello programmers, I need to create a list of unique words from a text file using PERL...may i have the code for that please? Thank you (1 Reply)
Discussion started by: alsohari
1 Replies

8. Shell Programming and Scripting

USING sed to remove multiple strings/words from a line

Hi I use sed comnand to remove occurance of one workd from a line. However I need to removed occurance of dufferent words in ne line. Original-1 Hi this is the END of my begining Comand sed s/"END"/"start"/g Output-1 Hi this is the start of my beginig But I have more... (9 Replies)
Discussion started by: mnassiri
9 Replies

9. Shell Programming and Scripting

Search words in multiple file line by line

Hi All I have to search servers name say like 1000+ "unique names" line by line in child.txt files in another file that is a master file where all server present say "master.txt",if child.txt's server name matches with master files then it print yes else no with server name. (4 Replies)
Discussion started by: netdbaind
4 Replies

10. UNIX for Beginners Questions & Answers

Shell - Read a text file with two words and extract data

hi I made this simple script to extract data and pretty much is a list and would like to extract data of two words separated by commas and I would like to make a new text file that would list these extracted data into a list and each in a new line. Example that worked for me with text file... (5 Replies)
Discussion started by: dandaryll
5 Replies
MYSQLDIFF(1)							  MySQL Utilities						      MYSQLDIFF(1)

NAME
mysqldiff - Identify differences among database objects SYNOPSIS
mysqldiff [options] {db1[:db1] | db1.obj1[:db2.obj2]} ... DESCRIPTION
This utility reads the definitions of objects and compares them using a diff-like method to determine whether they are the same. The util- ity displays the differences for objects that are not the same. Use the notation db1:db2 to name two databases to compare, or, alternatively just db1 to compare two databases with the same name. The latter case is a convenience notation for comparing same-named databases on different servers. The comparison may be run against two databases of different names on a single server by specifying only the --server1 option. The user can also connect to another server by specifying the --server2 option. In this case, db1 is taken from server1 and db2 from server2. When a database pair is specified, all objects in one database are compared to the corresponding objects in the other. Any objects not appearing in either database produce an error. To compare a specific pair of objects, add an object name to each database name in db.obj format. For example, use db1.obj1:db2.obj2 to compare two named objects, or db1.obj1 to compare an object with the same name in databases with the same name. It is not legal to mix a database name with an object name. For example, db1.obj1:db2 and db1:db2.obj2 are illegal. The comparison may be run against a single server for comparing two databases of different names on the same server by specifying only the --server1 option. Alternatively, you can also connect to another server by specifying the --server2 option. In this case, the first object to compare is taken from server1 and the second from server2. By default, the utilty generates object differences as a difference report. However, you can generate a transformation report containing SQL statements for transforming the objects for conformity instead. Use the 'sql' value for the --difftype option to produce a listing that contains the appropriate ALTER commands to conform the object definitions for the object pairs specified. If a transformation cannot be formed, the utility reports the diff of the object along with a warning statement. See important limitations in the NOTES section. To specify how to display diff-style output, use one of the following values with the --difftype option: unified (default) Display unified format output. context Display context format output. differ Display differ-style format output. sql Display SQL transformation statement output. The --changes-for option controls the direction of the difference (by specifying the object to be transformed) in either the difference report (default) or the transformation report (designated with the --difftype=sql option). Consider the following command: mysqldiff --server1=root@host1 --server2=root@host2 --difftype=sql db1.table1:dbx.table3 The leftmost database (db1) exists on the server designated by the --server1 option (host1). The rightmost database (dbx) exists on the server designated by the --server2 option (host2). o --changes-for=server1: Produce output that shows how to make the definitions of objects on server1 like the definitions of the corre- sponding objects on server2. o --changes-for=server2: Produce output that shows how to make the definitions of objects on server2 like the definitions of the corre- sponding objects on server1. The default direction is server1. For sql difference format, you can also see the reverse transformation by specifying the --show-reverse option. The utility stops on the first occurrence of missing objects or when an object does not match. To override this behavior, specify the --force option to cause the utility to attempt to compare all objects listed as arguments. OPTIONS
mysqldiff accepts the following command-line options: --help Display a help message and exit. --changes-for=<direction> Specify the server to show transformations to match the other server. For example, to see the transformation for transforming object definitions on server1 to match the corresponding definitions on server2, use --changes-for=server1. Permitted values are server1 and server2. The default is server1. --difftype=<difftype>, -d<difftype> Specify the difference display format. Permitted format values are unified, context, differ, and sql. The default is unified. --force Do not halt at the first difference found. Process all objects to find all differences. --quiet, -q Do not print anything. Return only an exit code of success or failure. --server1=<source> Connection information for the first server in <user>[:<passwd>]@<host>[:<port>][:<socket>] format. --server2=<source> Connection information for the second server in <user>[:<passwd>]@<host>[:<port>][:<socket>] format. --show-reverse Produce a transformation report containing the SQL statements to conform the object definitions specified in reverse. For example, if --changes-for is set to server1, also generate the transformation for server2. Note: The reverse changes are annotated and marked as comments. --verbose, -v Specify how much information to display. Use this option multiple times to increase the amount of information. For example, -v = verbose, -vv = more verbose, -vvv = debug. --version Display version information and exit. --width=<number> Change the display width of the test report. The default is 75 characters. NOTES
You must provide connection parameters (user, host, password, and so forth) for an account that has the appropriate privileges to access all objects to be compared. The SQL transformation feature has these known limitations: o When tables with partition differences are encountered, the utility generates the ALTER TABLE statement for all other changes but prints a warning and omits the partition differences. o If the transformation detects table options in the source table (specified with the --changes-for option) that are not changed or do not exist in the target table, the utility generates the ALTER TABLE statement for all other changes but prints a warning and omits the table option differences. o Rename for events is not supported. This is because mysqldiff compares objects by name. In this case, depending on the direction of the diff, the event is identified as needing to be added or a DROP EVENT statement is generated. o Changes in the definer clause for events are not supported. o SQL extensions specific to MySQL Cluster are not supported. For the --difftype option, the permitted values are not case sensitive. In addition, values may be specified as any unambiguous prefix of a valid value. For example, --difftype=d specifies the differ type. An error occurs if a prefix matches more than one valid value. EXAMPLES
To compare the employees and emp databases on the local server, use this command: $ mysqldiff --server1=root@localhost employees:emp1 # server1 on localhost: ... connected. WARNING: Objects in server1:employees but not in server2:emp1: EVENT: e1 Compare failed. One or more differences found. $ mysqldiff --server1=root@localhost employees.t1:emp1.t1 employees.t3:emp1.t3 # server1 on localhost: ... connected. # Comparing employees.t1 to emp1.t1 [PASS] # server1 on localhost: ... connected. # Comparing employees.t3 to emp1.t3 [PASS] Success. All objects are the same. $ mysqldiff --server1=root@localhost employees.salaries:emp1.salaries --differ # server1 on localhost: ... connected. # Comparing employees.salaries to emp1.salaries [FAIL] # Object definitions are not the same: CREATE TABLE `salaries` ( `emp_no` int(11) NOT NULL, `salary` int(11) NOT NULL, `from_date` date NOT NULL, `to_date` date NOT NULL, PRIMARY KEY (`emp_no`,`from_date`), KEY `emp_no` (`emp_no`) - ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ? ^^^^^ + ) ENGINE=MyISAM DEFAULT CHARSET=latin1 ? ++ ^^^ Compare failed. One or more differences found. The following examples show how to generate a transformation report. Assume the following object definitions: Host1: CREATE TABLE db1.table1 (num int, misc char(30)); Host2: CREATE TABLE dbx.table3 (num int, notes char(30), misc char(55)); To generate a set of SQL statements that transform the definition of db1.table1 to dbx.table3, use this command: $ mysqldiff --server1=root@host1 --server2=root@host2 --changes-for=server1 --difftype=sql db1.table1:dbx.table3 # server1 on host1: ... connected. # server2 on host2: ... connected. # Comparing db1.table1 to dbx.table3 [FAIL] # Transformation statments: ALTER TABLE db1.table1 ADD COLUMN notes char(30) AFTER a, CHANGE COLUMN misc misc char(55); Compare failed. One or more differences found. To generate a set of SQL statements that transform the definition of dbx.table3 to db1.table1, use this command: $ mysqldiff --server1=root@host1 --server2=root@host2 --changes-for=server2 --difftype=sql db1.table1:dbx.table3 # server1 on host1: ... connected. # server2 on host2: ... connected. # Comparing db1.table1 to dbx.table3 [FAIL] # Transformation statments: ALTER TABLE dbx.table3 DROP COLUMN notes, CHANGE COLUMN misc misc char(30); Compare failed. One or more differences found. To generate a set of SQL statements that transform the definitions of dbx.table3 and db1.table1 in both directions, use this command: $ mysqldiff --server1=root@host1 --server2=root@host2 --show-reverse --difftype=sql db1.table1:dbx.table3 # server1 on host1: ... connected. # server2 on host2: ... connected. # Comparing db1.table1 to dbx.table3 [FAIL] # Transformation statments: # --destination=server1: ALTER TABLE db1.table1 ADD COLUMN notes char(30) AFTER a, CHANGE COLUMN misc misc char(55); # --destination=server2: # ALTER TABLE dbx.table3 # DROP COLUMN notes, # CHANGE COLUMN misc misc char(30); Compare failed. One or more differences found. COPYRIGHT
Copyright (c) 2011, 2012, Oracle and/or its affiliates. All rights reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MER- CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA AUTHOR
MySQL Utilities Team COPYRIGHT
2010, Oracle and/or its affiliates. All rights reserved. 1.0.3 May 09, 2012 MYSQLDIFF(1)
All times are GMT -4. The time now is 06:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy