Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Splitting huge XML Files into fixsized wellformed parts

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting huge XML Files into fixsized wellformed parts
# 1  
Old 03-17-2008
Splitting huge XML Files into fixsized wellformed parts
Hi,

I need to split xml-files with sizes greater than 2 gb into smaler chunks. As I dont want to end up with billions of files, I want those splitted files to have configurable sizes like 250 MB. Each file should be well formed having an exact copy of the header (and footer as the closing of the header) from the original file. Forthermore, a table should be generated were I can see, that the File X is seperated into Part N with timestamp:

The Original XML-Files look like this:

<?xml ...>
<Headerelement with some infos to be copied 1to1>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
<OfferInfo>
<OfferID></OfferID>
...
</OfferInfo>
</Headerelement>

-------------------------
Protocol Table:

Orginalfilename|Name of PartN|Size of PartN|Timestamp

-------------------------

All in all I ended up with reading the XML processing docus of gawk,
but as it seems I am lacking some deeper programming skills.. Could
someone please help?

Thx
Malapha
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Splitting the XML file and renaming the files

Hello Gurus, I have a requirement to split the xml file into different xml files. Can you please help me with that? Here is my Source XML file <jms-system-resource> <name>PS6SOAJMSModule</name> <target>soa_server1</target> <sub-deployment> ... (3 Replies)
Discussion started by: Siv51427882
3 Replies

2. Shell Programming and Scripting

Splitting the XML file into three different files

Hello Shell Guru's I have a requirement to split the source xml file into three different text file. And i need your valuable suggestion to finish this. Here is my source xml snippet, here i am using only one entry of <jms-system-resource>. There may be multiple entries in the source file. ... (5 Replies)
Discussion started by: Siv51427882
5 Replies

3. Shell Programming and Scripting

Splitting a single xml file into multiple xml files

Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Sample.xml consists multiple headers so how can we split these multiple headers into multiple files in unix. eg : <?xml version="1.0" encoding="UTF-8"?> <ml:individual... (3 Replies)
Discussion started by: Narendra921631
3 Replies

4. Shell Programming and Scripting

Splitting xml file into several xml files using perl

Hi Everyone, I'm new here and I was checking this old post: /shell-programming-and-scripting/180669-splitting-file-into-several-smaller-files-using-perl.html (cannot paste link because of lack of points) I need to do something like this but understand very little of perl. I also check... (4 Replies)
Discussion started by: mcosta
4 Replies

5. Shell Programming and Scripting

XML Splitting into multi files

Hi , I have a XML file like below file name : sample.xml <?xml version="1.0"?> <catalog> <author>Rajini</author> <title>XML Guide</title> <Text> </Text> <genre>Computer</genre> <price>44.95</price> </catalog> <?xml version="1.0"?> <catalog> ... (5 Replies)
Discussion started by: karthinvk
5 Replies

6. Shell Programming and Scripting

perl, splitting out specific parts of the string

Hi there, I have an output from a command like this # ypcat -k netgroup.byuser| grep steven steven.* users_main,users_sysadmin,users_global,users_backup_team and wanted to pull the 'users' netgroups returned into a perl array, that will look like this users_main... (2 Replies)
Discussion started by: rethink
2 Replies

7. Shell Programming and Scripting

splitting a file (xml) into multiple files

To split the files Hi, I'm having a xml file with multiple xml header. so i want to split the file into multiple files. Test.xml --------- <?xml version="UTF_8"> <emp: ....> <name>a</name> <age>10</age> </emp> <?xml version="UTF_8"> <emp: ....> <name>b</name> <age>10</age>... (11 Replies)
Discussion started by: sasi_u
11 Replies

8. Shell Programming and Scripting

Splitting the Huge file into several files...

Hi I have to write a script to split the huge file into several pieces. The file columns is | pipe delimited. The data sample is as: 6625060|1420215|07308806|N|20100120|5572477081|+0002.79|+0000.00|0004|0001|......... (3 Replies)
Discussion started by: lakteja
3 Replies

9. Shell Programming and Scripting

Splitting a file into unequal parts

How do I split a file into many parts but with different amounts of lines per part? I looked at the split command but that only splits evenly. I'd like a range specified to determine how many lines each output file should have. For example, if the input file has 1000 lines and the range is... (1 Reply)
Discussion started by: revax
1 Replies

10. Shell Programming and Scripting

splitting huge xml into multiple files

hi all i have a some huge html files (500MB to 1GB). Each file has multiple <html></html> tags <html> ................. .................... .................... </html> <html> ................. .................... .................... </html> <html> .................... (5 Replies)
Discussion started by: uttamhoode
5 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

createrepo

createrepo(8)															     createrepo(8)

NAME

       createrepo - Create repomd (xml-rpm-metadata) repository

SYNOPSIS

       createrepo [options] <directory>

DESCRIPTION

       createrepo is a program that creates a repomd (xml-based rpm metadata) repository from a set of rpms.

OPTIONS

       -u --baseurl <url>
	      Optional base url location for all files. (not used by any clients at this time)

       -o --outputdir <url>
	      Optional output directory (useful for read only media).

       -x --exclude <package>
	      File globs to exclude, can be specified multiple times.

       -i --pkglist <filename>
	      specify a text file which contains the complete list of files to include in the repository from the set found in the directory. File
	      format is one package per line, no wildcards or globs.

       -q --quiet
	      Run quietly.

       -g --groupfile <groupfile>
	      A precreated xml filename to point to for group information.
	      See examples section below for further explanation.

       -v --verbose
	      Run verbosely.

       -c --cachedir <path>
	      Specify a directory to use as a cachedir. This allows createrepo to create a cache of checksums of packages in  the  repository.	In
	      consecutive  runs  of  createrepo  over  the  same  repository  of files that do not have a complete change out of all packages this
	      decreases the processing time dramatically.

       --update
	      If metadata already exists in the outputdir and an rpm is unchanged (based on file size and mtime) since the metadata was generated,
	      reuse  the  existing  metadata  rather than recalculating it. In the case of a large repository with only a few new or modified rpms
	      this can significantly reduce I/O and processing time.

       -C --checkts
	      Don't generate repo metadata, if their timestamps are newer than its rpms.  This option decreases the  processing  time  drastically
	      again, if you happen to run it on an unmodified repo, but it is (currently) mutual exclusive with the --split option.

       --split
	      Run  in  split  media  mode.  Rather than pass a single directory, take a set of directories corresponding to different volumes in a
	      media set.

       -p --pretty
	      Output xml files in pretty format.

       -V --version
	      Output version.

       -h --help
	      Show help menu.

       -d --database
	      Generate sqlite databases for use with yum.

EXAMPLES

       Here is an example of a repository with a groups file. Note that the groups file should be in the same directory as the rpm packages  (i.e.
       /path/to/rpms/comps.xml).

       createrepo -g comps.xml /path/to/rpms

FILES

       repodata/filelists.xml.gz
       repodata/other.xml.gz
       repodata/primary.xml.gz
       repodata/repomd.xml

SEE ALSO

       yum (8) yum.conf (5)

AUTHORS

       Seth Vidal <skvidal@phy.duke.edu>

BUGS

       Any bugs which are found should be emailed to the mailing list: rpm-metadata@linux.duke.edu

Seth Vidal							    2005 Jan 2							     createrepo(8)