deleting the part of the file(overwrite) using start and end point


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting deleting the part of the file(overwrite) using start and end point
# 1  
Old 12-06-2011
deleting the part of the file(overwrite) using start and end point

here is the contents of bigfile.sql
Code:
CREATE TABLE `Table11` (
`id` int(11) NOT NULL ,
`entityName` enum('Lines','EndUsers') COLLATE utf8_unicode_ci NOT NULL,
`parentAllianceMigrationProjectId`  varchar(255) NOT NULL
) ENGINE=InnoDB AUTO_INCREMENT=2000 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `Table22` (
`empNo` int(13) NOT NULL AUTO_INCREMENT,
`channels` enum('one','two','three') COLLATE utf8_unicode_ci NOT NULL,
`reserved6` varchar(255),
`accountStatus` enum('Active','Locked','Expired','Deactivated') COLLATE utf8_unicode_ci NOT NULL DEFAULT 'Active',
PRIMARY KEY (`empno`)
) ENGINE=InnoDB AUTO_INCREMENT=2000 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `HOOOP` (
  `allianceSiteId` int(11) DEFAULT NULL,
  `trunkGroupsId` int(11) DEFAULT NULL,
  `lastModified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  UNIQUE KEY `allianceSiteId` (`allianceSiteId`,`trunkGroupsId`)
) ENGINE=InnoDB AUTO_INCREMENT=2000 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `Table33` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`allianceId` int(11) NOT NULL,
`migratedAt` datetime DEFAULT NULL,
`migrationDetail` text COLLATE utf8_unicode_ci,
`unigyId` int(11) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2000 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

here is the content of tmpfile.sql
Code:
CREATE TABLE `HOOOP` (
  `allianceSiteId` int(11) DEFAULT NULL,
  `trunkGroupsId` int(11) DEFAULT NULL,
  `lastModified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  UNIQUE KEY `allianceSiteId` (`allianceSiteId`,`trunkGroupsId`)
)

this tmpfile.sql is extracted contents from bigfile.sql... now what i want to do is delete these extracted content from main file that is bigfile.sql... this i want to do dynamically.. but for explaining purpose lets say we have

Code:
firstword : CREATE TABLE `AllianceSiteTrunkGroupTrunkGroupsMap` (
lastword : ) ENGINE=InnoDB AUTO_INCREMENT=2000 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

where firstword and lastword are the first and last line of the tmpfile.sql.. ofcourse the last line of tmpfile.sql has only ")" but i am appending "ENGINE=InnoDB AUTO........etc etc" to identify since main file (bigfile) has this.
i tried

Code:
sed '/"firstword"/,/""lastword/d' bigfile.sql

but this is not working.... can anyone help me... SmilieSmilieSmilie

---------- Post updated at 02:53 PM ---------- Previous update was at 02:50 PM ----------

spell mistale its this which i tried..
Code:
 sed '/"$firstword"/,/"$lastword"/d' bigfile.sql

# 2  
Old 12-06-2011
It's either
Code:
sed "/$firstword/,/$lastword/d" bigfile.sql

or
Code:
sed '/'$firstword'/,/'$lastword'/d' bigfile.sql

You could also do:

Code:
table=HOOOP 
sed '/CREATE TABLE `'$table'`/,/^(/ d' bigfile.sql

All that is providing your table block doesnt contain a slash.
This User Gave Thanks to mirni For This Post:
# 3  
Old 12-06-2011
thanks mirni... your code is working good :-) :-) thanks a lot...
i have one more doubt... its almost similiar....

---------- Post updated at 06:38 PM ---------- Previous update was at 06:33 PM ----------

consider bigfile.sql...
now i have two variables named
startpoint and endpoint..

Code:
 
tablename="HOOOP"
startpoint="CREATE TABLE `$tablename` ("
endpoint=") ENGINE=InnoDB AUTO_INCREMENT=2000 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;"

how to extract lines from startpoint to endpoint from bigfile.sql and paste it in a temporary file like tmp.sql...

after executing the output in tmp.sql should be

cat tmp.sql
Code:
CREATE TABLE `HOOOP` (
  `allianceSiteId` int(11) DEFAULT NULL,
  `trunkGroupsId` int(11) DEFAULT NULL,
  `lastModified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  UNIQUE KEY `allianceSiteId` (`allianceSiteId`,`trunkGroupsId`)
)

---------- Post updated at 06:39 PM ---------- Previous update was at 06:38 PM ----------

i have done the same function using while loop its working fine the problem is its efficiency is poor since it takes lot of cycles of while loop to find it... is there any one liner to do this function...

---------- Post updated at 06:58 PM ---------- Previous update was at 06:39 PM ----------

here is a code to extract lines from the dump on the basis of "CREATE TABLE" and "ENGINE"
Code:
perl -lne '(/^\s*CREATE\s+TABLE/i .. /^\s*\)\s+ENGINE=InnoDB/i) && print;' bigfile.sql >tmp.sql

but what i want to include is cut from CREATE TABLE $table_name to engine.. (that is include table name in the above command). can you guys help me out...?
# 4  
Old 12-06-2011
Careful, you don't have it right:
Code:
tablename="HOOOP" 
startpoint="CREATE TABLE `$tablename` (" 
endpoint=") ENGINE=InnoDB AUTO_INCREMENT=2000 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;"

Try to echo the $startpoint. You will not have the backticks there. It's because the backticks mean "output of this command" when in weak (double) quotes. So the shell is trying to execute command "HOOOP" and inserting the output (empty) in the var $startpoint.
What you want, is to escape the backticks:
Code:
tablename="HOOOP"
startpoint="CREATE TABLE \`$tablename\` ("
endpoint=") ENGINE=InnoDB AUTO_INCREMENT=2000 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;"

echo start $startpoint

sed "/$startpoint/,/$endpoint/ w tmp.sql" < mysql.big
sed "/$startpoint/,/$endpoint/ d" mysql.big > mysql.new

The 'w' command of sed will write the block to tmp.sql. Watch the spaces, as any extra space will be considered as part of the filename:
Code:
sed "/$startpoint/,/$endpoint/ w tmp.sql " < mysql.big

Will create a file named 'tmp.sql '.
This User Gave Thanks to mirni For This Post:
# 5  
Old 12-09-2011
thanks the code is working :-)... but the thing is i need to validate whether the table exists in the file or not.. if the particular startpoint is not there in the file this sed command should not be executed and flag must be set to 1..
like

Code:
 
if $startpoint exists in the file
then
     sed "/$startpoint/,/$endpoint/ w tmp.sql " < mysql.big
else
      flag=1
fi

thats is i am checking whether the table exists in the file or not...

also i have one more doubt... how to check how many tables present in the file or not.. currently i am using while loop to find it..
Code:
while read line1
do
        if [[ `expr match "$line1" ".*CREATE TABLE.*"` != "0" ]]
        then
                counter1=`expr $counter1 + 1`
        fi
done < onlytables1.sql

but this is consuming time.. is there any one line code to perform this...?

---------- Post updated at 11:52 AM ---------- Previous update was at 11:38 AM ----------

i found out how to find the number of tables (by trial and error ofcourse) :-)
Code:
cat 1.sql | grep -i CREATE | wc | awk -F' ' '{print $1}'

---------- Post updated at 01:45 PM ---------- Previous update was at 11:52 AM ----------

i found out how to find whether table name is there or not.. thanks anyway mirni :-)

Code:
cat testdump1.sql | grep -i "CREATE TABLE \`$tablename\` (" | wc | awk -F' ' '{print $1}'

the output will be 1 if present else 0.. thanks.. i might have few more doubts wil ask you gradually.. :-)
# 6  
Old 12-09-2011
Glad you are making progress.
Let's simplify things a bit:

Your test, whether the pattern was found can be done after the sed command. If pattern was not found, the file will be empty. So:
Code:
sed "/$startpoint/,/$endpoint/ w tmp.sql " < mysql.big
if [ -s tmp.sql ] ; then  #is file non-empty?
   echo pattern found
else
   echo not found  #empty or doesnt exist
fi

Counting how many table entries, this can be done by grep alone, just use the -c option:
Code:
grep -c "CREATE TABLE \`$tablename\` (" testdump1.sql

Just beware:
sed's pattern matching is greedy. So, the range /start/,/end/ will match all lines between the first 'start' and last 'end'.

Last edited by mirni; 12-10-2011 at 12:50 AM..
This User Gave Thanks to mirni For This Post:
# 7  
Old 12-12-2011
hi

as you can see in the attachment.. i have uploaded log as out.txt.. the output displayed at the promt.. the sed command is working fine for intial content of the files... but as you said its greedy, for the the last but one table its consider both the table of interest and next table... why is that.... how could it work fine initially and fail at the end...?.. oberserve tableextract1.sql and tableextract2.sql in the output...

and also when i run sed command.. its performs some function but the same time displays all the contents at the promt.. how to subdue this feature.. i mean it should perfrom the function but not display all the contents of the file ....

Last edited by vivek d r; 02-07-2012 at 05:17 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove lines between the start string and end string including start and end string Python

Hi, I am trying to remove lines once a string is found till another string is found including the start string and end string. I want to basically grab all the lines starting with color (closing bracket). PS: The line after the closing bracket for color could be anything (currently 'more').... (1 Reply)
Discussion started by: Dabheeruz
1 Replies

2. Shell Programming and Scripting

How to perform a hexdump using dd from start point to end point?

hi, I would like to ask or is it possible to dump a hex using dd from starting point to end point just like the "xxd -s 512 -l 512 <bin file>" I know the redirect hexdump -C but i can't figure it out the combination options of dd. Hope someone can share their knowledge.. Thanks in... (3 Replies)
Discussion started by: jao_madn
3 Replies

3. UNIX for Dummies Questions & Answers

Deleting part of file names

My server got messed up and the names of my files were not completely processed. As results I ended up getting a bunch of files with the following names: I need to remove the underscore and everything before it. Thus, the files will be renamed to something like this: Any help will be greatly... (3 Replies)
Discussion started by: Xterra
3 Replies

4. Shell Programming and Scripting

deleting blank lines ONLY at the end of the file

Hi Guys, I have a quetion which was already discussed in the forum, but for some reason all approches suggested fail for me. I have a file which have blank lines at the body of the text as well as at the end. I need to delete ONLY blank lines at the end. Unfortunatly the approach below does not... (5 Replies)
Discussion started by: aoussenko
5 Replies

5. Shell Programming and Scripting

Grepping from a point in a file to the end of the file

does any one know how to turn the equivalent of this command: awk '/2011 John Doe 8344/,0' /tmp/ops.log | egrep -c "received request" to something that would use egrep instead of awk? What the awk command does is, it searches the ops.log file for "2011 John Doe 8344". When it finds it,... (4 Replies)
Discussion started by: SkySmart
4 Replies

6. Shell Programming and Scripting

deleting the lines at the end of the file.

I have a text file with two coulmn first column is just used in to show the line number, these line number are not there in the real file. I want to delete the line 16(in this file) here, even tough there is no data inside it . this empty line is causing me a problem by throwing me garbage... (12 Replies)
Discussion started by: shashi792
12 Replies

7. UNIX for Dummies Questions & Answers

deleting word from this point to end of file in VI

Hi All i need to delete a recurring word from point "n" till end of the file. there are other words in this file so i cannot use `dG`, can anyone help me out? Kind regards Brian (4 Replies)
Discussion started by: brian112
4 Replies

8. UNIX for Advanced & Expert Users

Deleting end of line $ character in file

Hi, I've got a file where in the middle of the record is a $ end of line character, visible only when I open the file in vi and do :set list. How to I get rid of the character in the middle and keep it at the end. The middle $ character always appears after SW, so that can be used to tag it.... (3 Replies)
Discussion started by: bwrynz1
3 Replies

9. UNIX Desktop Questions & Answers

Deleting Junks at the end of each line in a file

Is there any way to delete the Junk Characters(Invalid Characters like ^,',",),(,&,# etc.,) at the end of each record in a file? I want to do this using a single line script. Thanks to all in advance!!! (5 Replies)
Discussion started by: dave_nithis
5 Replies

10. Shell Programming and Scripting

Deleting end line spaces for along file

How can i clear all space characteres for a long file at the end of each line? (3 Replies)
Discussion started by: osymad
3 Replies
Login or Register to Ask a Question