Splitting huge XML Files into fixsized wellformed parts


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting huge XML Files into fixsized wellformed parts
# 1  
Old 03-17-2008
Splitting huge XML Files into fixsized wellformed parts

Hi,

I need to split xml-files with sizes greater than 2 gb into smaler chunks. As I dont want to end up with billions of files, I want those splitted files to have configurable sizes like 250 MB. Each file should be well formed having an exact copy of the header (and footer as the closing of the header) from the original file. Forthermore, a table should be generated were I can see, that the File X is seperated into Part N with timestamp:

The Original XML-Files look like this:

<?xml ...>
<Headerelement with some infos to be copied 1to1>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
</Headerelement>

-------------------------
Protocol Table:

Orginalfilename|Name of PartN|Size of PartN|Timestamp

-------------------------

All in all I ended up with reading the XML processing docus of gawk,
but as it seems I am lacking some deeper programming skills.. Could
someone please help?

Thx
Malapha
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Splitting the XML file and renaming the files

Hello Gurus, I have a requirement to split the xml file into different xml files. Can you please help me with that? Here is my Source XML file <jms-system-resource> <name>PS6SOAJMSModule</name> <target>soa_server1</target> <sub-deployment> ... (3 Replies)
Discussion started by: Siv51427882
3 Replies

2. Shell Programming and Scripting

Splitting the XML file into three different files

Hello Shell Guru's I have a requirement to split the source xml file into three different text file. And i need your valuable suggestion to finish this. Here is my source xml snippet, here i am using only one entry of <jms-system-resource>. There may be multiple entries in the source file. ... (5 Replies)
Discussion started by: Siv51427882
5 Replies

3. Shell Programming and Scripting

Splitting a single xml file into multiple xml files

Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Sample.xml consists multiple headers so how can we split these multiple headers into multiple files in unix. eg : <?xml version="1.0" encoding="UTF-8"?> <ml:individual... (3 Replies)
Discussion started by: Narendra921631
3 Replies

4. Shell Programming and Scripting

Splitting xml file into several xml files using perl

Hi Everyone, I'm new here and I was checking this old post: /shell-programming-and-scripting/180669-splitting-file-into-several-smaller-files-using-perl.html (cannot paste link because of lack of points) I need to do something like this but understand very little of perl. I also check... (4 Replies)
Discussion started by: mcosta
4 Replies

5. Shell Programming and Scripting

XML Splitting into multi files

Hi , I have a XML file like below file name : sample.xml <?xml version="1.0"?> <catalog> <author>Rajini</author> <title>XML Guide</title> <Text> </Text> <genre>Computer</genre> <price>44.95</price> </catalog> <?xml version="1.0"?> <catalog> ... (5 Replies)
Discussion started by: karthinvk
5 Replies

6. Shell Programming and Scripting

perl, splitting out specific parts of the string

Hi there, I have an output from a command like this # ypcat -k netgroup.byuser| grep steven steven.* users_main,users_sysadmin,users_global,users_backup_team and wanted to pull the 'users' netgroups returned into a perl array, that will look like this users_main... (2 Replies)
Discussion started by: rethink
2 Replies

7. Shell Programming and Scripting

splitting a file (xml) into multiple files

To split the files Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Test.xml --------- <?xml version="UTF_8"> <emp: ....> <name>a</name> <age>10</age> </emp> <?xml version="UTF_8"> <emp: ....> <name>b</name> <age>10</age>... (11 Replies)
Discussion started by: sasi_u
11 Replies

8. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

9. Shell Programming and Scripting

Splitting a file into unequal parts

How do I split a file into many parts but with different amounts of lines per part? I looked at the split command but that only splits evenly. I'd like a range specified to determine how many lines each output file should have. For example, if the input file has 1000 lines and the range is... (1 Reply)
Discussion started by: revax
1 Replies

10. Shell Programming and Scripting

splitting huge xml into multiple files

hi all i have a some huge html files (500MB to 1GB). Each file has multiple <html></html> tags <html> ................. .................... .................... </html> <html> ................. .................... .................... </html> <html> .................... (5 Replies)
Discussion started by: uttamhoode
5 Replies
Login or Register to Ask a Question
DBAUTOSPLIT(1)						      DocBook Splitting tools						    DBAUTOSPLIT(1)

NAME
dbautosplit - automaticaly split DocBook document SYNOPSIS
dbautosplit [options] input_file [output_template] dbautosplit --help DESCRIPTION
dbautosplit is tool for automaticaly splitting DocBook documents. It use XInclude (see <http://www.w3.org/TR/xinclude/>) for include docu- ment fragments. This allow split large XML file to smaler files withal preserve validation. Splitting is possible in any element. Default is "book", "part", "chapter", "article", "preface", "reference" and "refentry". Morewer you may specify maximum depth level. Output file names are full customizable. It is possible to get tree structure corespond to hiearchy of sections in DocBook document. All "xml:base" attributes (see <http://www.w3.org/TR/xmlbase/>) are processed and them removed. Relative references in "fileref" and "url" attributes are properly rewrited. So it is posible to use dbautosplit (with maximum level depth set to zero) as inteligent "xml:base" attribute remover and so as intelignet DocBook copyier. If no output teplate is specified, default out/index.xml is used. See TEMPALTE EXPANSION below. OPTIONS
--split=ELEMENTS Coma-separated list of XML elements in which is required spliting. Asterisk ('*') means any tag. The default is "book", "part", "chap- ter", "article", "preface" "reference" and "refentry". --split-add=ELEMENTS Coma-separate list of XML elements which are addeded to previously specified tags list or to the default list. See --split otpion. --split-del=ELEMENTS Coma-separated list of XML elements which are removed from previously specified tags list or from the default list. See --split option. -l DEPTH, --level=DEPTH Specify recursion maximum depth level to DEPTH. The default maximum depth is 1. --lo-dirs=PATHS Coma-separated list of paths in which are located local objects (see LOCAL OBJECTS below). All paths are expanded as the standart Unix shell would do, see File::Glob(3perl) for details. Relative paths is related to curent working directory. If path contains $XML_BASE at the begining it is replaced by real base of input document, see option --xmlbase below. Default paths are LocalObjects, MediaOb- jects, ImageObjects and #Pictures in base directory of input document. --lo-dirs-add=PATHS Coma-separeted list of paths which are addeded to preiously specified or the default local objects paths. See --lo-dirs option. --lo-dir=DIRECTORY Name of local objects directory for newly created documents (see LOCAL OBJECTS below). Relative path is related to newly created docu- ments. Default directory is LocalObjects. --lo-move Move local objects instead copy them. See LOCAL OBJECTS below. -o=TEMPLATE, --output-other=TEMPLATE TEMPLATE for other (included) output files (see TEMPALTE EXPANSIONS below). Relative path is related to parent. The default is %attr(id)?:%text(%name()-%02index())/index.xml. --xmlbase=URI XML base URI (Uniform Resource Identifier) for input document. See <http://www.w3.org/TR/xmlbase/> for details. You may need change default if you simple copy an DocBook document without copying referenced files nor changing "fileref" attributes. On this case set XML base to orginal location. Note, URI is required not (unix) file name. --xi-dtd[=FILE] Append minimal internal subset for support "xi:include" element. If FILE is specified then entiry reference to this external subset is inserted instead. Relative path is related to curent work directory. --encoding=ENCODING Enforce encoding for all output files. -v, --verbose Increase verbosity level. -q, --quiet Decrease verbosity level. -V, --version Dump version information and exit. -h, -?, --help Dump help screen and exit. --man Show this manual page and exit. LOCAL OBJECTS
Local objects are special files (pictures, audio files, ...) in directories specified by --lo-dirs option. If DocBook document refer (via "fileref" attribute) to this directory, the files are copied (or moved) to local "Local Objects Directory" specified by --lo-dir option. Moreower all files with same name and differ suffix (after last dot) are copied (or moved) too. TEMPLATE EXPANSIONS
In output file names are recognized special templates with following syntax: TEMPLATE := %[FORMAT]COMMAND(OPTIONS)[?TEMPLATE[:TEMPLATE]] where FORMAT is format modifier as for "%s" format in "printf" command, and COMMAND(OPTIONSB) is one of (case insensible): Attribute(NAME), Attr(NAME) Replace template by attribute value. NodeName(), Name() Replace template by name of node. Depth() Replace template by current depth. Index() Replace template by an unique increasing integer. Text(STRING), Txt(STRING) Replace template by arbritrary string. All templates in STRING are expanded at first. PerlEval(CODE), Eval(CODE) Replace template by result of evalution arbritraty perl code. All templates in CODE are expanded at first. Optionaly the first template can by folloved question mark, second temlate, colon mark and third temlpate. On this case if expansion of the first template is emty the second template is expanded otherwise third template is expanded. EXAMPLES
dbautosplit --level=0 --om=new/mydoc.xml old/mydoc.xml Copy file mydoc.xml from old/ to new/ directory, remove all xml:base attributes and properly rewrite all relative fileref references so they refer to same files. If they are reference to files in local objects directoryes, they will be copyed. dbautosplit --om=new/mydoc.xml --split-add="section" old/mydoc.xml Same as above, but new/mydox.xml is now splited in defaults elements plus in element "section". Name of output files according to default output template (except main output file which is specified) which is %attr(id)?:%text(%name()-%02index())/index.xml. That mean every splited part create own directory in new directory and file index.xml in this directory. Name of this directory will be the value of "id" attribute or name of splited element followed by dash and an integer if there is no "id" attribute specified. dbautosplit --oo='%eval(uc("%attr(id)?:%text(%name()-%index())/index.xml"))' old/mydoc.xml The output files names will have the same name as in default case but in uper case. BUGS AND TODO
o How about cross-document ID/IDREF? Help me. o Magnle entity referenced by entityref and targetdoent attributes. SEE ALSO
dbmerge(3), "DocBook: The Definitive Guide" <http://www.docbook.org/tdg/en/html/docbook.html> AUTHOR
Martin Lazar <mlazar@suse.cz> 0.6 2004-01-27 DBAUTOSPLIT(1)