Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Splitting huge XML Files into fixsized wellformed parts

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting huge XML Files into fixsized wellformed parts
# 1  
Old 03-17-2008
Splitting huge XML Files into fixsized wellformed parts
Hi,

I need to split xml-files with sizes greater than 2 gb into smaler chunks. As I dont want to end up with billions of files, I want those splitted files to have configurable sizes like 250 MB. Each file should be well formed having an exact copy of the header (and footer as the closing of the header) from the original file. Forthermore, a table should be generated were I can see, that the File X is seperated into Part N with timestamp:

The Original XML-Files look like this:

<?xml ...>
<Headerelement with some infos to be copied 1to1>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
</Headerelement>

-------------------------
Protocol Table:

Orginalfilename|Name of PartN|Size of PartN|Timestamp

-------------------------

All in all I ended up with reading the XML processing docus of gawk,
but as it seems I am lacking some deeper programming skills.. Could
someone please help?

Thx
Malapha
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Splitting the XML file and renaming the files

Hello Gurus, I have a requirement to split the xml file into different xml files. Can you please help me with that? Here is my Source XML file <jms-system-resource> <name>PS6SOAJMSModule</name> <target>soa_server1</target> <sub-deployment> ... (3 Replies)
Discussion started by: Siv51427882
3 Replies

2. Shell Programming and Scripting

Splitting the XML file into three different files

Hello Shell Guru's I have a requirement to split the source xml file into three different text file. And i need your valuable suggestion to finish this. Here is my source xml snippet, here i am using only one entry of <jms-system-resource>. There may be multiple entries in the source file. ... (5 Replies)
Discussion started by: Siv51427882
5 Replies

3. Shell Programming and Scripting

Splitting a single xml file into multiple xml files

Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Sample.xml consists multiple headers so how can we split these multiple headers into multiple files in unix. eg : <?xml version="1.0" encoding="UTF-8"?> <ml:individual... (3 Replies)
Discussion started by: Narendra921631
3 Replies

4. Shell Programming and Scripting

Splitting xml file into several xml files using perl

Hi Everyone, I'm new here and I was checking this old post: /shell-programming-and-scripting/180669-splitting-file-into-several-smaller-files-using-perl.html (cannot paste link because of lack of points) I need to do something like this but understand very little of perl. I also check... (4 Replies)
Discussion started by: mcosta
4 Replies

5. Shell Programming and Scripting

XML Splitting into multi files

Hi , I have a XML file like below file name : sample.xml <?xml version="1.0"?> <catalog> <author>Rajini</author> <title>XML Guide</title> <Text> </Text> <genre>Computer</genre> <price>44.95</price> </catalog> <?xml version="1.0"?> <catalog> ... (5 Replies)
Discussion started by: karthinvk
5 Replies

6. Shell Programming and Scripting

perl, splitting out specific parts of the string

Hi there, I have an output from a command like this # ypcat -k netgroup.byuser| grep steven steven.* users_main,users_sysadmin,users_global,users_backup_team and wanted to pull the 'users' netgroups returned into a perl array, that will look like this users_main... (2 Replies)
Discussion started by: rethink
2 Replies

7. Shell Programming and Scripting

splitting a file (xml) into multiple files

To split the files Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Test.xml --------- <?xml version="UTF_8"> <emp: ....> <name>a</name> <age>10</age> </emp> <?xml version="UTF_8"> <emp: ....> <name>b</name> <age>10</age>... (11 Replies)
Discussion started by: sasi_u
11 Replies

8. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

9. Shell Programming and Scripting

Splitting a file into unequal parts

How do I split a file into many parts but with different amounts of lines per part? I looked at the split command but that only splits evenly. I'd like a range specified to determine how many lines each output file should have. For example, if the input file has 1000 lines and the range is... (1 Reply)
Discussion started by: revax
1 Replies

10. Shell Programming and Scripting

splitting huge xml into multiple files

hi all i have a some huge html files (500MB to 1GB). Each file has multiple <html></html> tags <html> ................. .................... .................... </html> <html> ................. .................... .................... </html> <html> .................... (5 Replies)
Discussion started by: uttamhoode
5 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

xml_split

XML_SPLIT(1p)						User Contributed Perl Documentation					     XML_SPLIT(1p)

NAME

	 xml_split - cut a big XML file into smaller chunks

DESCRIPTION

       "xml_split" takes a (presumably big) XML file and split it in several smaller files. The memory used is the memory needed for the biggest
       chunk (ie memory is reused for each new chunk).

       It can split at a given level in the tree (the default, splits children of the root), or on a condition (using the subset of XPath
       understood by XML::Twig, so "section" or "/doc/section").

       Each generated file is replaced by a processing instruction that will allow "xml_merge" to rebuild the original document. The processing
       instruction format is "<?merge subdocs=[01] :<filename> ?>"

       File names are <file>-<nb>.xml, with <file>-00.xml holding the main document.

OPTIONS

       -l <level>
	   level to cut at: 1 generates a file for each child of the root, 2 for each grand child

	   defaults to 1

       -c <condition>
	   generate a file for each element that passes the condition

	   xml_split -c <section> will put each "section" element in its own file (nested sections are handled too)

	   Note that at the moment this option is a lot slower than using "-l"

       -s <size>
	   generates files of (approximately) <size>. The content of each file is enclosed in a new element ("xml_split::root"), so it's well-
	   formed XML.	The size can be given in bytes, Kb, Mb or Gb.

       -g <nb>
	   groups <nb> elements in a single file. The content of each file is enclosed in a new element ("xml_split::root"), so it's well-formed
	   XML.

       -b <name>
	   base name for the output, files will be named <base>-<nb><.ext>

	   <nb> is a sequence number, see below "--nb_digits" <ext> is an extension, see below "--extension"

	   defaults to the original file name (if available) or "out" (if input comes from the standard input)

       -n <nb>
	   number of digits in the sequence number for each file

	   if more digits than <nb> are needed, then they are used: if "--nb_digits 2" is used and 112 files are generated they will be named
	   "<file>-01.xml" to "<file>-112.xml"

	   defaults to 2

       -e <ext>
	   extension to use for generated files

	   defaults to the original file extension or ".xml"

       -i  use XInclude elements instead of Processing Instructions to mark where sub files need to be included

       -v  verbose output

	   Note that this option can slow down processing considerably (by an order of magnitude) when generating lots of small documents

       -V  outputs version and exit

       -h  short help

       -m  man (requires pod2text to be in the path)

EXAMPLES

	 xml_split foo.xml	       # split at level 1
	 xml_split -l 2 foo.xml        # split at level 2
	 xml_split -c section foo.xml  # a file is generated for each section element
				       # nested sections are split properly

SEE ALSO

       XML::Twig, xml_merge

TODO

       optimize the code
	   any idea welcome! I have already implemented most of what I thought would improve performances.

       provide other methods that PIs to keep merge information
	   XInclude is a good candidate (alpha support added in 0.04).

	   using entities, which would seem the natural way to do it, doesn't work, as they make it impossible to have both the main document and
	   the sub docs to be well-formed if the sub docs include sub-sub docs (you can't have entity declarations in an entity)

AUTHOR

       Michel Rodriguez <mirod@cpan.org>

LICENSE

       This tool is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.12.4							    2010-09-20							     XML_SPLIT(1p)