Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Splitting huge XML Files into fixsized wellformed parts

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting huge XML Files into fixsized wellformed parts
# 1  
Old 03-17-2008
Splitting huge XML Files into fixsized wellformed parts
Hi,

I need to split xml-files with sizes greater than 2 gb into smaler chunks. As I dont want to end up with billions of files, I want those splitted files to have configurable sizes like 250 MB. Each file should be well formed having an exact copy of the header (and footer as the closing of the header) from the original file. Forthermore, a table should be generated were I can see, that the File X is seperated into Part N with timestamp:

The Original XML-Files look like this:

<?xml ...>
<Headerelement with some infos to be copied 1to1>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
</Headerelement>

-------------------------
Protocol Table:

Orginalfilename|Name of PartN|Size of PartN|Timestamp

-------------------------

All in all I ended up with reading the XML processing docus of gawk,
but as it seems I am lacking some deeper programming skills.. Could
someone please help?

Thx
Malapha
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Splitting the XML file and renaming the files

Hello Gurus, I have a requirement to split the xml file into different xml files. Can you please help me with that? Here is my Source XML file <jms-system-resource> <name>PS6SOAJMSModule</name> <target>soa_server1</target> <sub-deployment> ... (3 Replies)
Discussion started by: Siv51427882
3 Replies

2. Shell Programming and Scripting

Splitting the XML file into three different files

Hello Shell Guru's I have a requirement to split the source xml file into three different text file. And i need your valuable suggestion to finish this. Here is my source xml snippet, here i am using only one entry of <jms-system-resource>. There may be multiple entries in the source file. ... (5 Replies)
Discussion started by: Siv51427882
5 Replies

3. Shell Programming and Scripting

Splitting a single xml file into multiple xml files

Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Sample.xml consists multiple headers so how can we split these multiple headers into multiple files in unix. eg : <?xml version="1.0" encoding="UTF-8"?> <ml:individual... (3 Replies)
Discussion started by: Narendra921631
3 Replies

4. Shell Programming and Scripting

Splitting xml file into several xml files using perl

Hi Everyone, I'm new here and I was checking this old post: /shell-programming-and-scripting/180669-splitting-file-into-several-smaller-files-using-perl.html (cannot paste link because of lack of points) I need to do something like this but understand very little of perl. I also check... (4 Replies)
Discussion started by: mcosta
4 Replies

5. Shell Programming and Scripting

XML Splitting into multi files

Hi , I have a XML file like below file name : sample.xml <?xml version="1.0"?> <catalog> <author>Rajini</author> <title>XML Guide</title> <Text> </Text> <genre>Computer</genre> <price>44.95</price> </catalog> <?xml version="1.0"?> <catalog> ... (5 Replies)
Discussion started by: karthinvk
5 Replies

6. Shell Programming and Scripting

perl, splitting out specific parts of the string

Hi there, I have an output from a command like this # ypcat -k netgroup.byuser| grep steven steven.* users_main,users_sysadmin,users_global,users_backup_team and wanted to pull the 'users' netgroups returned into a perl array, that will look like this users_main... (2 Replies)
Discussion started by: rethink
2 Replies

7. Shell Programming and Scripting

splitting a file (xml) into multiple files

To split the files Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Test.xml --------- <?xml version="UTF_8"> <emp: ....> <name>a</name> <age>10</age> </emp> <?xml version="UTF_8"> <emp: ....> <name>b</name> <age>10</age>... (11 Replies)
Discussion started by: sasi_u
11 Replies

8. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

9. Shell Programming and Scripting

Splitting a file into unequal parts

How do I split a file into many parts but with different amounts of lines per part? I looked at the split command but that only splits evenly. I'd like a range specified to determine how many lines each output file should have. For example, if the input file has 1000 lines and the range is... (1 Reply)
Discussion started by: revax
1 Replies

10. Shell Programming and Scripting

splitting huge xml into multiple files

hi all i have a some huge html files (500MB to 1GB). Each file has multiple <html></html> tags <html> ................. .................... .................... </html> <html> ................. .................... .................... </html> <html> .................... (5 Replies)
Discussion started by: uttamhoode
5 Replies
Login or Register to Ask a Question

LEARN ABOUT SUSE

crm_diff

CRM_DIFF(8)							  [FIXME: manual]						       CRM_DIFF(8)

NAME

       crm_diff - identify changes to the cluster configuration and apply patches to the configuration files

SYNOPSIS

	   crm_diff [-?|-V] [-o filename] [-O string] [-p filename] [-n filename] [-N string]

DESCRIPTION

       The crm_diff command assists in creating and applying XML patches. This can be useful for visualizing the changes between two versions of
       the cluster configuration or saving changes so they can be applied at a later time using cibadmin.

OPTIONS

       --help, -?
	   Print a help message.

       --original filename, -o filename
	   Specify the original file against which to diff or apply patches.

       --new filename, -n filename
	   Specify the name of the new file.

       --original-string string, -O string
	   Specify the original string against which to diff or apply patches.

       --new-string string, -N string
	   Specify the new string.

       --patch filename, -p filename
	   Apply a patch to the original XML. Always use with -o.

       --cib, -c
	   Compare or patch the inputs as a CIB. Always specify the base version with -o and provide either the patch file or the second version
	   with -p or -n, respectively.

       --stdin, -s
	   Read the inputs from stdin.

EXAMPLES

       Use crm_diff to determine the differences between various CIB configuration files and to create patches. By means of patches, easily reuse
       configuration parts without having to use the cibadmin command on every single one of them.

	1. Obtain the two different configuration files by running cibadmin on the two cluster setups to compare:

	       cibadmin -Q > cib1.xml
	       cibadmin -Q > cib2.xml

	2. Determine whether to diff the entire files against each other or compare just a subset of the configurations.

	3. To print the difference between the files to stdout, use the following command:

	       crm_diff -o cib1.xml -n cib2.xml

	4. To print the difference between the files to a file and create a patch, use the following command:

	       crm_diff -o cib1.xml -n cib2.xml > patch.xml

	5. Apply the patch to the original file:

	       crm_diff -o cib1.xml -p patch.xml

FILES

       /var/lib/heartbeat/crm/cib.xml--the CIB (minus status section) on disk. Editing this file directly is strongly discouraged.

SEE ALSO

       ???

AUTHOR

       crm_diff was written by Andrew Beekhof.

[FIXME: source] 						    07/05/2010							       CRM_DIFF(8)