Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Splitting huge XML Files into fixsized wellformed parts

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting huge XML Files into fixsized wellformed parts
# 1  
Old 03-17-2008
Splitting huge XML Files into fixsized wellformed parts
Hi,

I need to split xml-files with sizes greater than 2 gb into smaler chunks. As I dont want to end up with billions of files, I want those splitted files to have configurable sizes like 250 MB. Each file should be well formed having an exact copy of the header (and footer as the closing of the header) from the original file. Forthermore, a table should be generated were I can see, that the File X is seperated into Part N with timestamp:

The Original XML-Files look like this:

<?xml ...>
<Headerelement with some infos to be copied 1to1>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
</Headerelement>

-------------------------
Protocol Table:

Orginalfilename|Name of PartN|Size of PartN|Timestamp

-------------------------

All in all I ended up with reading the XML processing docus of gawk,
but as it seems I am lacking some deeper programming skills.. Could
someone please help?

Thx
Malapha
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Splitting the XML file and renaming the files

Hello Gurus, I have a requirement to split the xml file into different xml files. Can you please help me with that? Here is my Source XML file <jms-system-resource> <name>PS6SOAJMSModule</name> <target>soa_server1</target> <sub-deployment> ... (3 Replies)
Discussion started by: Siv51427882
3 Replies

2. Shell Programming and Scripting

Splitting the XML file into three different files

Hello Shell Guru's I have a requirement to split the source xml file into three different text file. And i need your valuable suggestion to finish this. Here is my source xml snippet, here i am using only one entry of <jms-system-resource>. There may be multiple entries in the source file. ... (5 Replies)
Discussion started by: Siv51427882
5 Replies

3. Shell Programming and Scripting

Splitting a single xml file into multiple xml files

Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Sample.xml consists multiple headers so how can we split these multiple headers into multiple files in unix. eg : <?xml version="1.0" encoding="UTF-8"?> <ml:individual... (3 Replies)
Discussion started by: Narendra921631
3 Replies

4. Shell Programming and Scripting

Splitting xml file into several xml files using perl

Hi Everyone, I'm new here and I was checking this old post: /shell-programming-and-scripting/180669-splitting-file-into-several-smaller-files-using-perl.html (cannot paste link because of lack of points) I need to do something like this but understand very little of perl. I also check... (4 Replies)
Discussion started by: mcosta
4 Replies

5. Shell Programming and Scripting

XML Splitting into multi files

Hi , I have a XML file like below file name : sample.xml <?xml version="1.0"?> <catalog> <author>Rajini</author> <title>XML Guide</title> <Text> </Text> <genre>Computer</genre> <price>44.95</price> </catalog> <?xml version="1.0"?> <catalog> ... (5 Replies)
Discussion started by: karthinvk
5 Replies

6. Shell Programming and Scripting

perl, splitting out specific parts of the string

Hi there, I have an output from a command like this # ypcat -k netgroup.byuser| grep steven steven.* users_main,users_sysadmin,users_global,users_backup_team and wanted to pull the 'users' netgroups returned into a perl array, that will look like this users_main... (2 Replies)
Discussion started by: rethink
2 Replies

7. Shell Programming and Scripting

splitting a file (xml) into multiple files

To split the files Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Test.xml --------- <?xml version="UTF_8"> <emp: ....> <name>a</name> <age>10</age> </emp> <?xml version="UTF_8"> <emp: ....> <name>b</name> <age>10</age>... (11 Replies)
Discussion started by: sasi_u
11 Replies

8. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

9. Shell Programming and Scripting

Splitting a file into unequal parts

How do I split a file into many parts but with different amounts of lines per part? I looked at the split command but that only splits evenly. I'd like a range specified to determine how many lines each output file should have. For example, if the input file has 1000 lines and the range is... (1 Reply)
Discussion started by: revax
1 Replies

10. Shell Programming and Scripting

splitting huge xml into multiple files

hi all i have a some huge html files (500MB to 1GB). Each file has multiple <html></html> tags <html> ................. .................... .................... </html> <html> ................. .................... .................... </html> <html> .................... (5 Replies)
Discussion started by: uttamhoode
5 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

avisplit

avisplit(1)						      General Commands Manual						       avisplit(1)

NAME

       avisplit - split AVI-files into chunks of a maximum size

SYNOPSIS

       avisplit [ -i file -o base [ -s size ] [ -H num ] [ -t s1-s2[,s3-s4,..]	-c -m -b num -f commentfile ] ] [ -v ]

COPYRIGHT

       avisplit is Copyright (C) by Thomas Oestreich.

DESCRIPTION

       avisplit splits a single AVI-file into chunks of size size.
       Each of the created chunks will be an independent file, i.e. it can be played without needing any other of the chunk.

OPTIONS

       -i file
	      Specify the filename of the file to split into chunks.

       -o base
	      Specify the base of the output filename(s) avisplit will then split to base-%04d.avi

       -s size
	      Use this option to specify the maximum size (in units of MB) of the chunks avisplit should create. 0 means dechunk, create as many
	      files as possible.

       -H num Create only the first num chunks then exit.

       -t s1-s2[,s3-s4,..]
	      Split the input file based on time/framecode (hh:mm:ss.ms)

       -c     Together with -t. Merge all segments into one AVI-File again instead generating seperate files.

       -m     Together with -t. Force split at upper bondary instead of lower border.

       -b num Specify if avisplit should write an VBR mp3 header into the AVI file. Default is 1 because it does not hurt. num is either 1 or 0.

       -f commentfile
	      Read AVI tombstone data for header comments from commentfile. See /docs/avi_comments.txt for a sample.

       -v     Print only version information and exit.

EXAMPLES

       The command

       avisplit -s 700 -i my_file.avi

       will split the file my_file.avi into chunks which's maximum size will not exceed 700 MB, i.e. they will fit onto a CD, each.  The created
       chunks will be named my_file.avi-0000, my_file.avi-0001, etc.

       avisplit -i my_file.avi -c -o out.avi -t 00:10:00-00:11:00,00:13:00-00:14:00

       will grab Minutes 10 to 11 and 13 to 14 from my_file.avi and merge it into out.avi

BAD SYNCH

       When you split a file with avisplit and the A/V sync for the first file is OK but the sync on all successive files is bad then have a look
       at the output of tcprobe(1) (shortend).

	| V: 25.000 fps, codec=dvsd, frames=250, width=720, height=576
	| A: 48000 Hz, format=0x01, bits=16, channels=2, bitrate=1536 kbps,
	|    10 chunks, 1920000 bytes

       You'll see the AVI file has only 10 Audio chunks but 250 video chunks. That means one audio chunk spans several video frames.  avisplit can
       not cut a chunk in half, it only handles complete chunks. If you do, say, avisplit -s 20, it is possible that the first file will have 6
       audio chunks and the second one only 4 meaning there is too much audio in the first AVI file.

       The solution is to remux the AVI file with
	      transcode -i in.avi -P1 -N 0x1 -y raw -o out.avi
       (of course -N 0x1 is not correct for all AVI files).  Now look at tcprobe again

	| V: 25.000 fps, codec=dvsd, frames=250, width=720, height=576
	| A: 48000 Hz, format=0x01, bits=16, channels=2, bitrate=1536 kbps,
	|   250 chunks, 1920000 bytes

       The data in this file is exactly the same (its bit-identical) as it was in in.avi; the AVI file was just written in a different way, we do
       now have 250 audio chunks which makes splitting much easier and more accurate for avisplit.

AUTHORS

       avisplit was written by Thomas Oestreich
       <ostreich@theorie.physik.uni-goettingen.de> with contributions from many others.  See AUTHORS for details.

SEE ALSO

       aviindex(1), avifix(1), avimerge(1), tccat(1), tcdecode(1), tcdemux(1), tcextract(1), tcprobe(1), tcscan(1), transcode(1)

avisplit(1)							  25th June 2003						       avisplit(1)