Sponsored Content
Top Forums Shell Programming and Scripting Split a 30GB XML file into 16 pieces Post 302551101 by Corona688 on Monday 29th of August 2011 11:50:20 AM
Old 08-29-2011
Nice-looking program, though I would note one problem:
Code:
char BUF[NBUF+1];
BUF[NBUF+1] = '\0';

Replace NBUF with 4 and follow along:

Code:
char BUF[4+1];
buf[0]=0; // first element
buf[1]=1; // second element
buf[2]=2; // third element
buf[3]=3; // fourth element
buf[4]=4; // fifth element
buf[5]=5; // SIXTH element!  buf[4+1] is beyond the end!

If you're lucky, this will do nothing.

If you're unlucky, it will crash your program.

If you're very unlucky, it will corrupt stack values in strange ways that alter other local variables and cause unpredictable misbehavior.

This often results in programs that work fine when compiled for debugging, but do strange things when optimized -- suddenly memory values which didn't matter get stripped out and you're only stomping on ones that do.

Code:
char BUF[NBUF+1];
    BUF[NBUF] = '\0';

That should be all it needs I think.
These 2 Users Gave Thanks to Corona688 For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to split XML file

Hi, I'm experiencing difficulty in loading an XML file to an Oracle destination table.I keep running into a memory problem due to the large size of the file. I want to split the XML file into several smaller files based on the keyword(s)/tags : '' and '' and would like to use a Unix shell... (2 Replies)
Discussion started by: bayflash27
2 Replies

2. Shell Programming and Scripting

How do I split file into pieces with PERL?

How do I split file into pieces with PERL? IE file.txt head 1 2 3 4 end head 5 6 7 8 9 end n so on (7 Replies)
Discussion started by: 3junior
7 Replies

3. Shell Programming and Scripting

Need to split a xml file in proper format

Hi, I have a file which has xml data but all in single line Ex - <?xml version="1.0"?><User><Name>Robert</Name><Location>California</Location><Occupation>Programmer</Occupation></User> I want to split the data in proper xml format Ex- <?xml version="1.0"?> <User> <Name>Robert</Name>... (6 Replies)
Discussion started by: avishek007
6 Replies

4. UNIX for Dummies Questions & Answers

How to split a huge file into small pieces (per 2000 columns)?

Dear all, I have a big file:2879(rows)x400,170 (columns) like below. I 'd like to split the file into small pieces:2879(rows)x2000(columns) per file (the last small piece will be 2879x170. So far, I only know how to create one samll piece at one time. But actually I need to repeat this work... (6 Replies)
Discussion started by: forevertl
6 Replies

5. Shell Programming and Scripting

Split xml file into many

Hi, I had a scenario need a help as I am new to this. I have a xml file employee.xml with the below content. <Organisation><employee>xxx</employee><employee>yyy</employee><employee>zzz</employee></Organisation> I want to split the file into multiple file as below. Is there a specifice way... (5 Replies)
Discussion started by: mankuar
5 Replies

6. Shell Programming and Scripting

Split XML file

Hi Experts, Can you please help me to split following XML file based on new Order ? Actual file is very big. I have taken few lines of it. <?xml version="1.0" encoding="utf-8" standalone="yes"?> <Orders xmlns='http://www.URL.com/Orders'> <Order> <ORDNo>450321</ORDNo> ... (3 Replies)
Discussion started by: meetmedude
3 Replies

7. Shell Programming and Scripting

Perl : to split the tags from xml file

I do have an xml sheet as below where I need the perl script to filter only the hyperlink tags. <cols><col min="1" max="1" width="30.5703125" customWidth="1"/><col min="2" max="2" width="7.140625" bestFit="1" customWidth="1"/> <col min="3" max="3" width="32.28515625" bestFit="1"... (3 Replies)
Discussion started by: scriptscript
3 Replies

8. Shell Programming and Scripting

Split XML file based on tags

Hello All , Please help me with below requirement I want to split a xml file based on tag.here is the file format <data-set> some-information </data-set> <data-set1> some-information </data-set1> <data-set2> some-information </data-set2> I want to split the above file into 3... (5 Replies)
Discussion started by: Pratik4891
5 Replies

9. Shell Programming and Scripting

Split xml file into multiple xml based on letterID

Hi All, We need to split a large xml into multiple valid xml with same header(2lines) and footer(last line) for N number of letterId. In the example below we have first 2 lines as header and last line as footer.(They need to be in each split xml file) Header: <?xml version="1.0"... (5 Replies)
Discussion started by: vx04
5 Replies

10. Shell Programming and Scripting

Split Big XML file Base on tag

HI I want to split file base on tag name. I have few header and footer on file <?xml version="1.33" encing="UTF-8"?> <bulkCmConfigDataFile" <xn:SubNetwork id="ONRM_ROOT"> <xn:MeContext id="PPP04156"> ... (4 Replies)
Discussion started by: pareshkp
4 Replies
MPI_File_write_shared(3OpenMPI) 										   MPI_File_write_shared(3OpenMPI)

NAME
MPI_File_write_shared - Writes a file using the shared file pointer (blocking, noncollective). SYNTAX
C Syntax #include <mpi.h> int MPI_File_write_shared(MPI_File fh, void *buf, int count, MPI_Datatype datatype, MPI_Status *status) Fortran Syntax INCLUDE 'mpif.h' MPI_FILE_WRITE_SHARED(FH, BUF, COUNT, DATATYPE, STATUS, IERROR) <type> BUF(*) INTEGER FH, COUNT, DATATYPE, STATUS(MPI_STATUS_SIZE), IERROR C++ Syntax #include <mpi.h> void MPI::File::Write_shared(const void* buf, int count, const MPI::Datatype& datatype, MPI::Status& status) void MPI::File::Write_shared(const void* buf, int count, const MPI::Datatype& datatype) INPUT
/OUTPUT PARAMETER fh File handle (handle). INPUT PARAMETERS
buf Initial address of buffer (choice). count Number of elements in buffer (integer). datatype Data type of each buffer element (handle). OUTPUT PARAMETERS
status Status object (status). IERROR Fortran only: Error status (integer). DESCRIPTION
MPI_File_write_shared is a blocking routine that uses the shared file pointer to write files. The order of serialization is not determinis- tic for this noncollective routine. ERRORS
Almost all MPI routines return an error value; C routines as the value of the function and Fortran routines in the last argument. C++ func- tions do not return errors. If the default error handler is set to MPI::ERRORS_THROW_EXCEPTIONS, then on error the C++ exception mechanism will be used to throw an MPI:Exception object. Before the error value is returned, the current MPI error handler is called. For MPI I/O function errors, the default error handler is set to MPI_ERRORS_RETURN. The error handler may be changed with MPI_File_set_errhandler; the predefined error handler MPI_ERRORS_ARE_FATAL may be used to make I/O errors fatal. Note that MPI does not guarantee that an MPI program can continue past an error. Open MPI 1.2 September 2006 MPI_File_write_shared(3OpenMPI)
All times are GMT -4. The time now is 08:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy