Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Split a 30GB XML file into 16 pieces
Top Forums Shell Programming and Scripting Split a 30GB XML file into 16 pieces Post 302551025 by yazu on Monday 29th of August 2011 05:37:30 AM
Old 08-29-2011
It's easy enough with low-level programming with C (or Perl). The principle is the same as in "tail" command. You rewind the file pointer to FILESIZE/16 and go back to find the first <page> and remember the byte position, then rewind to 2*FILESIZE/16 and so on. When you get your positions you split the file with dd (or in this program).
If no one comes with a solution, I'll write a program but a little later.
This User Gave Thanks to yazu For This Post:
shoaibjameel123
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to split XML file

Hi, I'm experiencing difficulty in loading an XML file to an Oracle destination table.I keep running into a memory problem due to the large size of the file. I want to split the XML file into several smaller files based on the keyword(s)/tags : '' and '' and would like to use a Unix shell... (2 Replies)
Discussion started by: bayflash27
2 Replies

2. Shell Programming and Scripting

How do I split file into pieces with PERL?

How do I split file into pieces with PERL? IE file.txt head 1 2 3 4 end head 5 6 7 8 9 end n so on (7 Replies)
Discussion started by: 3junior
7 Replies

3. Shell Programming and Scripting

Need to split a xml file in proper format

Hi, I have a file which has xml data but all in single line Ex - <?xml version="1.0"?><User><Name>Robert</Name><Location>California</Location><Occupation>Programmer</Occupation></User> I want to split the data in proper xml format Ex- <?xml version="1.0"?> <User> <Name>Robert</Name>... (6 Replies)
Discussion started by: avishek007
6 Replies

4. UNIX for Dummies Questions & Answers

How to split a huge file into small pieces (per 2000 columns)?

Dear all, I have a big file:2879(rows)x400,170 (columns) like below. I 'd like to split the file into small pieces:2879(rows)x2000(columns) per file (the last small piece will be 2879x170. So far, I only know how to create one samll piece at one time. But actually I need to repeat this work... (6 Replies)
Discussion started by: forevertl
6 Replies

5. Shell Programming and Scripting

Split xml file into many

Hi, I had a scenario need a help as I am new to this. I have a xml file employee.xml with the below content. <Organisation><employee>xxx</employee><employee>yyy</employee><employee>zzz</employee></Organisation> I want to split the file into multiple file as below. Is there a specifice way... (5 Replies)
Discussion started by: mankuar
5 Replies

6. Shell Programming and Scripting

Split XML file

Hi Experts, Can you please help me to split following XML file based on new Order ? Actual file is very big. I have taken few lines of it. <?xml version="1.0" encoding="utf-8" standalone="yes"?> <Orders xmlns='http://www.URL.com/Orders'> <Order> <ORDNo>450321</ORDNo> ... (3 Replies)
Discussion started by: meetmedude
3 Replies

7. Shell Programming and Scripting

Perl : to split the tags from xml file

I do have an xml sheet as below where I need the perl script to filter only the hyperlink tags. <cols><col min="1" max="1" width="30.5703125" customWidth="1"/><col min="2" max="2" width="7.140625" bestFit="1" customWidth="1"/> <col min="3" max="3" width="32.28515625" bestFit="1"... (3 Replies)
Discussion started by: scriptscript
3 Replies

8. Shell Programming and Scripting

Split XML file based on tags

Hello All , Please help me with below requirement I want to split a xml file based on tag.here is the file format <data-set> some-information </data-set> <data-set1> some-information </data-set1> <data-set2> some-information </data-set2> I want to split the above file into 3... (5 Replies)
Discussion started by: Pratik4891
5 Replies

9. Shell Programming and Scripting

Split xml file into multiple xml based on letterID

Hi All, We need to split a large xml into multiple valid xml with same header(2lines) and footer(last line) for N number of letterId. In the example below we have first 2 lines as header and last line as footer.(They need to be in each split xml file) Header: <?xml version="1.0"... (5 Replies)
Discussion started by: vx04
5 Replies

10. Shell Programming and Scripting

Split Big XML file Base on tag

HI I want to split file base on tag name. I have few header and footer on file <?xml version="1.33" encing="UTF-8"?> <bulkCmConfigDataFile" <xn:SubNetwork id="ONRM_ROOT"> <xn:MeContext id="PPP04156"> ... (4 Replies)
Discussion started by: pareshkp
4 Replies

LEARN ABOUT HPUX

norewind_close_disabled

norewind_close_disabled(5)					File Formats Manual					norewind_close_disabled(5)

NAME

       norewind_close_disabled - determine whether access to tape device special files without the no-rewind-on-close option is allowed

VALUES

   Default
   Allowed values
DESCRIPTION

       Note: this tunable was named in releases prior to HP-UX 11i v3.

       This  tunable  notifies the tape driver whether it should allow access to device special files without the no-rewind-on-close option.  When
       this tunable is on, the tape driver will fail any open on a tape device special file that does not have the no-rewind-on-close option (that
       is, or with a status of

       The  tunable  allows  the  safe	sharing of tape devices in SAN tape backup solution configurations.  The disabling of device special files
       without no-rewind-on-close in a SAN tape backup solution configuration prevents the accidental sending of a rewind command  on  close  from
       HP-UX systems via standard tape commands like which would corrupt a backup in process.

   Who Is Expected to Change This Tunable?
       Anyone.

   Restrictions on Changing
       Changes to this tunable take effect immediately.

   When Should the Tunable Be Turned On?
       This tunable should be turned on if the system is going to be used in a SAN tape backup solution configuration.

   What Are the Side Effects of Turning the Tunable On?
       The  tape  driver  will	fail any open to a tape device special file that does not have the no-rewind-on-close option with a status of This
       behavior prevents the usage of any tape device special file without the no-rewind-on-close option.

   When Should the Tunable Be Turned Off?
       This tunable should be turned off if the system is not part of a SAN tape backup solution configuration or the  system  utilizes  will  not
       function unless it has access to device special files without the no-rewind-on-close option.

   What Are the Side Effects of Turning the Tunable Off?
       A rewind may be issued accidentally from this system on close, possibly corrupting a backup being issued from another system on the SAN.

   What Other Tunables Should Be Changed at the Same Time?
       None.

EXAMPLES

       Query the current value of the tunable:

       Set the value of the tunable to

       To save the value of the tunable across reboots:

WARNINGS

       All  HP-UX kernel tunable parameters are release specific.  This parameter may be removed or have its meaning changed in future releases of
       HP-UX.

       Installation of optional kernel software, from HP or other vendors, may cause changes to tunable  parameter  values.   After  installation,
       some  tunable  parameters may no longer be at the default or recommended values.  For information about the effects of installation on tun-
       able values, consult the documentation for the kernel software being installed.	For information about optional kernel  software  that  was
       factory installed on your system, see at

AUTHOR

       was developed by Hewlett-Packard.

SEE ALSO

       scsi_tape(7), scsimgr_estape(7).

							     Tunable Kernel Parameters					norewind_close_disabled(5)
All times are GMT -4. The time now is 08:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy