Today (Saturday) We will make some minor tuning adjustments to MySQL.

You may experience 2 up to 10 seconds "glitch time" when we restart MySQL. We expect to make these adjustments around 1AM Eastern Daylight Saving Time (EDT) US.


Split large xml into mutiple files and with header and footer in file


Login or Register to Reply

 
Thread Tools Search this Thread
# 22  
Hi Don Cragun,

Please forget about the above awk commands it would be confusing below is the sample xml file
i want string value JOB_ID to be extracted and assigned to a variable NEW_VAR

Output Expected:
Code:
 NEW_VAR ='30544,30545,30546'

This value i will pass to Database later

Code:
 <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns5:DoPublishFromImportResponse xmlns:ns12="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/response" xmlns:ns11="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/response" xmlns:ns10="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/response" xmlns:ns9="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1" xmlns:ns8="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/request" xmlns:ns7="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/request" xmlns:ns6="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1" xmlns:ns5="oracle/documaker/schema/ws/publishing" xmlns:ns4="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1" xmlns:ns3="oracle/documaker/schema/common" xmlns:ns2="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/request" xmlns="oracle/documaker/schema/ws/publishing/common"><ns5:DoPublishFromImportResponseV1><Result>0</Result><ServiceTimeMillis>13</ServiceTimeMillis><ns6:JobResponse CorrelationId="?"><ns11:JobPayloadType>1</ns11:JobPayloadType><ns11:JobPriority>10</ns11:JobPriority><ns11:JobStatus>111</ns11:JobStatus><ns11:JobUnique_Id>010d9363-6362-4f66-a48a-b3a1e4b90bc9</ns11:JobUnique_Id><ns11:Job_Id>30544</ns11:Job_Id></ns6:JobResponse><ns6:ServiceInfo><ns3:Operation>doPublishFromImport</ns3:Operation><ns3:Version><ns3:Number>1</ns3:Number><ns3:Used>true</ns3:Used></ns3:Version></ns6:ServiceInfo></ns5:DoPublishFromImportResponseV1></ns5:DoPublishFromImportResponse></S:Body></S:Envelope><?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns5:DoPublishFromImportResponse xmlns:ns12="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/response" xmlns:ns11="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/response" xmlns:ns10="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/response" xmlns:ns9="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1" xmlns:ns8="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/request" xmlns:ns7="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/request" xmlns:ns6="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1" xmlns:ns5="oracle/documaker/schema/ws/publishing" xmlns:ns4="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1" xmlns:ns3="oracle/documaker/schema/common" xmlns:ns2="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/request" xmlns="oracle/documaker/schema/ws/publishing/common"><ns5:DoPublishFromImportResponseV1><Result>0</Result><ServiceTimeMillis>14</ServiceTimeMillis><ns6:JobResponse CorrelationId="?"><ns11:JobPayloadType>1</ns11:JobPayloadType><ns11:JobPriority>10</ns11:JobPriority><ns11:JobStatus>111</ns11:JobStatus><ns11:JobUnique_Id>f8268dda-9357-45ec-baab-e6fbb30744bd</ns11:JobUnique_Id><ns11:Job_Id>30545</ns11:Job_Id></ns6:JobResponse><ns6:ServiceInfo><ns3:Operation>doPublishFromImport</ns3:Operation><ns3:Version><ns3:Number>1</ns3:Number><ns3:Used>true</ns3:Used></ns3:Version></ns6:ServiceInfo></ns5:DoPublishFromImportResponseV1></ns5:DoPublishFromImportResponse></S:Body></S:Envelope><?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns5:DoPublishFromImportResponse xmlns:ns12="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/response" xmlns:ns11="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/response" xmlns:ns10="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/response" xmlns:ns9="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1" xmlns:ns8="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/request" xmlns:ns7="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/request" xmlns:ns6="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1" xmlns:ns5="oracle/documaker/schema/ws/publishing" xmlns:ns4="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1" xmlns:ns3="oracle/documaker/schema/common" xmlns:ns2="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/request" xmlns="oracle/documaker/schema/ws/publishing/common"><ns5:DoPublishFromImportResponseV1><Result>0</Result><ServiceTimeMillis>12</ServiceTimeMillis><ns6:JobResponse CorrelationId="?"><ns11:JobPayloadType>1</ns11:JobPayloadType><ns11:JobPriority>10</ns11:JobPriority><ns11:JobStatus>111</ns11:JobStatus><ns11:JobUnique_Id>35b40e14-77b8-4f63-80c4-6ac0d8020985</ns11:JobUnique_Id><ns11:Job_Id>30546</ns11:Job_Id></ns6:JobResponse><ns6:ServiceInfo><ns3:Operation>doPublishFromImport</ns3:Operation><ns3:Version><ns3:Number>1</ns3:Number><ns3:Used>true</ns3:Used></ns3:Version></ns6:ServiceInfo></ns5:DoPublishFromImportResponseV1></ns5:DoPublishFromImportResponse></S:Body></S:Envelope><?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns5:DoPublishFromImportResponse xmlns:ns12="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/response" xmlns:ns11="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/response" xmlns:ns10="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/response" xmlns:ns9="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1" xmlns:ns8="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/request" xmlns:ns7="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/request" xmlns:ns6="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1" xmlns:ns5="oracle/documaker/schema/ws/publishing" xmlns:ns4="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1" xmlns:ns3="oracle/documaker/schema/common" xmlns:ns2="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/request" xmlns="oracle/documaker/schema/ws/publishing/common"><ns5:DoPublishFromImportResponseV1><Result>0</Result><ServiceTimeMillis>15</ServiceTimeMillis><ns6:JobResponse CorrelationId="?"><ns11:JobPayloadType>1</ns11:JobPayloadType><ns11:JobPriority>10</ns11:JobPriority><ns11:JobStatus>111</ns11:JobStatus><ns11:JobUnique_Id>9e4e8e04-167f-46dd-9801-27776728fe05</ns11:JobUnique_Id><ns11:Job_Id>30547</ns11:Job_Id></ns6:JobResponse><ns6:ServiceInfo><ns3:Operation>doPublishFromImport</ns3:Operation><ns3:Version><ns3:Number>1</ns3:Number><ns3:Used>true</ns3:Used></ns3:Version></ns6:ServiceInfo></ns5:DoPublishFromImportResponseV1></ns5:DoPublishFromImportResponse></S:Body></S:Envelope><?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><ns5:DoPublishFromImportResponse xmlns:ns12="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/response" xmlns:ns11="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/response" xmlns:ns10="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/response" xmlns:ns9="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1" xmlns:ns8="oracle/documaker/schema/ws/publishing/doGetPublishingInfo/v1/request" xmlns:ns7="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/request" xmlns:ns6="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1" xmlns:ns5="oracle/documaker/schema/ws/publishing" xmlns:ns4="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1" xmlns:ns3="oracle/documaker/schema/common" xmlns:ns2="oracle/documaker/schema/ws/publishing/doPublishFromFactory/v1/request" xmlns="oracle/documaker/schema/ws/publishing/common"><ns5:DoPublishFromImportResponseV1><Result>0</Result><ServiceTimeMillis>15</ServiceTimeMillis><ns6:JobResponse CorrelationId="?"><ns11:JobPayloadType>1</ns11:JobPayloadType><ns11:JobPriority>10</ns11:JobPriority><ns11:JobStatus>111</ns11:JobStatus><ns11:JobUnique_Id>cfd9fba3-bc37-4f2f-936e-7b38f7c59f57</ns11:JobUnique_Id><ns11:Job_Id>30548</ns11:Job_Id></ns6:JobResponse><ns6:ServiceInfo><ns3:Operation>doPublishFromImport</ns3:Operation><ns3:Version><ns3:Number>1</ns3:Number><ns3:Used>true</ns3:Used></ns3:Version></ns6:ServiceInfo></ns5:DoPublishFromImportResponseV1></ns5:DoPublishFromImportResponse></S:Body></S:Envelope>

Moderator's Comments:
Mod Comment Please use CODE tags when displaying sample input (which you did), sample output (which you did not), and code segments (which you did not).

Moderator's Comments:
Mod Comment Duplicated update message deleted to save space and to reduce confusion about possible differences in the given sample XML file contents.

Last edited by Don Cragun; 02-10-2019 at 11:25 PM.. Reason: Add missing CODE tags and remove duplicated message update.
# 23  
Hi Karthik,
PLEASE pay attention to what you are doing! There cannot be a <space> between the name of a shell variable and the <equals-sign> that follows it if you are trying to assign a value to that variable. This has been said several times in this thread and yet you still write that you want the result to be:
Code:
 NEW_VAR ='30544,30545,30546'

which, as stated before tells the shell to run a utility named NEW_VAR with one operand that is the string =30544,30545,30546 and note that that operand does not contain the <single-quote> characters that will be removed by the shell as it prepares the arguments to be passed to the NEW_VAR utility when it is invoked.

Note also that you have not told us what operating system you're using. With a sample file that is 8,157 bytes long and contains only a single line, that is not a text file on many BSD, Linux, and UNIX systems and the awk, sed, and most other standard text processing utilities have undefined behavior if the input files being processed are not text files.

Note also that you say that the output to be produced from your sample input should have three numbers (Job IDs) in the output, but there are five Job IDs in the sample input? Why shouldn't all five values be extracted from the XML file?

If we assume that the awk utility on your system can handle text files with unlimited line lengths, the following might do what you want:
Code:
NEW_VAR=$(awk -v sq="'" -F'<ns11:Job_Id>' '
		{	for(i = 2; i <= NF; i++) {
				sub(/<.*/, "", $i)
				printf("%s%s", cnt++ ? "," : sq, $i)
			}
		}
		END {	print sq
		}' file
	)

printf 'NEW_VAR has been assigned the value: %s\n' "$NEW_VAR"

which, on macOS Mojave version 10.14.3, produces the output:
Code:
NEW_VAR has been assigned the value: '30544,30545,30546,30547,30548'

if the file named file contains the sample data you provided in post #22 in this thread.
These 2 Users Gave Thanks to Don Cragun For This Post:
# 24  
I am using Linux OS and the file is .xml , And I need all the values of job_id as you mentioned not just 3.

--- Post updated at 04:42 AM ---

Thanks a lot it worked and my apologies for all the confusion.
# 25  
Quote:
Originally Posted by karthik
Iam using Linux OS and the file is .xml , And i need all the values of job_id as you mentioned not just 3.
OK. So does the code I suggested in post #23 produce the output you want if you change the name of the file in the script to match the name of your input file?
This User Gave Thanks to Don Cragun For This Post:
# 27  
Hi Don Cragun/Rudic ,

I have built the script based on all the inputs one last thing is renaming files it is still creating just one standard file name kindly assist

Below command is not creating unique names as expected

sample input:
Code:
filename:sampletest.xml
                                           sampletest_111.xml

Actual Output:
Code:
Extrfile001.xml just 1 file is getting created

Expected Output:
Code:
Extrfile001.xml
Extrfile002.xml

Code:
arr=($(ls | grep "../Inbound/Extrfile[0-9]*.xml"))

Code:
#!/bin/sh

# Add all Input files to array
FileList=($(ls | grep "../Inbound/sampletest*\\_[0-9]"))
  
echo  "$FileList"  

#loop array for Input files

for x in "${FileList[@]}"
do
 #for each element in array
 

#File Split Begin
awk -f xml_tag_handler.awk -f File_split.awk OUT=$x"" ROWS="500" $x $x
mv $x ../Staging
done

rm Response.xml Extr*.xml 


for f in ../Inbound/sampletest_*
  do    TMP="${f/sampletest_/Extrfile}"
         mv "$f" "${TMP%.*}"
  done

# add all files to array
arr=($(ls | grep "../Inbound/Extrfile[0-9]*.xml"))


Last edited by Don Cragun; 02-18-2019 at 12:19 AM.. Reason: Remove quote tags around update; get rid of earlier version of duplicated post.
# 28  
Quote:
Originally Posted by karthik
Hi Don Cragun/Rudic ,

I have built the script based on all the inputs one last thing is renaming files it is still creating just one standard file name kindly assist

Below command is not creating unique names as expected

sample input:
Code:
filename:sampletest.xml
                                           sampletest_111.xml

Actual Output:
Code:
Extrfile001.xml just 1 file is getting created

Expected Output:
Code:
Extrfile001.xml
Extrfile002.xml

Code:
arr=($(ls | grep "../Inbound/Extrfile[0-9]*.xml"))

Code:
#!/bin/sh

# Add all Input files to array
FileList=($(ls | grep "../Inbound/sampletest*\\_[0-9]"))
  
echo  "$FileList"  

#loop array for Input files

for x in "${FileList[@]}"
do
 #for each element in array
 

#File Split Begin
awk -f xml_tag_handler.awk -f File_split.awk OUT=$x"" ROWS="500" $x $x
mv $x ../Staging
done

rm Response.xml Extr*.xml 


for f in ../Inbound/sampletest_*
  do    TMP="${f/sampletest_/Extrfile}"
         mv "$f" "${TMP%.*}"
  done

# add all files to array
arr=($(ls | grep "../Inbound/Extrfile[0-9]*.xml"))

Hi karthik,
All of the code marked in red above will ALWAYS expand to nothing because the output from ls when invoked with no operands will NEVER yield any string containing ../. Therefore the script you showed us is logically equivalent to the script:
Code:
#!/bin/sh

# Add all Input files to array
FileList=()
  
echo  ""  
#loop array for Input files
#for each element in array
#File Split Begin

rm Response.xml Extr*.xml

for f in ../Inbound/sampletest_*
  do    TMP="${f/sampletest_/Extrfile}"
         mv "$f" "${TMP%.*}"
  done

# add all files to array
arr=()

I assume that you are not getting what you want because you never run any of the awk scripts in your shell script; you only move around and change the names of files that already existed before you started running this script.
This User Gave Thanks to Don Cragun For This Post:
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Is there a way to append both at header and footer of a file
jediwannabe
currently I've a file Insert into CD_CARD_TYPE (CODE, DESCRIPTION, LAST_UPDATE_BY, LAST_UPDATE_DATE) Values ('024', '024', 2, sysdate); Insert into CD_CARD_TYPE (CODE, DESCRIPTION, LAST_UPDATE_BY, LAST_UPDATE_DATE) Values ('032', '032', 2, sysdate); ........ is it...... Shell Programming and Scripting
3
Shell Programming and Scripting
Removing header or footer from file
sridhardwh
Hi Every one, what is the coomand to remove header or footer from a file. Please help me by providing command/syntax to remove header/footer from unix. Thanks in advance for all your support.... Shell Programming and Scripting
5
Shell Programming and Scripting
Add header and footer with record count in footer
itsranjan
This is my file(Target.txt) name|age|locaction abc|23|del xyz|24|mum jkl|25|kol The file should be like this 1|03252012 1|name|age|location 2|abc|23|del 2|xyz|24|mum 2|jkl|25|kol 2|kkk|26|hyd 3|4 Column 1 is row indicator for row 1 and 2, column indicator is 1,for data rows...... Shell Programming and Scripting
1
Shell Programming and Scripting
Split large file and add header and footer to each small files
ashish4422
I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file?... Shell Programming and Scripting
7
Shell Programming and Scripting
Split large file and add header and footer to each file
ashish4422
I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file?... Shell Programming and Scripting
1
Shell Programming and Scripting

Featured Tech Videos