Programming

Dear all,

I am having a huge XML file, as below structure
<EMPLOYEE>
<RECORD id =aaa>
<Salary>99999</Salary>
<section>ssss</section>
</RECORD>
<RECORD id =bbb>
<Salary>77777</Salary>
<section>ssss</section>
</RECORD>
</EMPLOYEE>

This is a 50 GB file I want to read this file in multithreading mode and write to a multiple files(one for each thread) with the Salary and section (salary~section) . Trying to do it in C++

After that i will have to merge this file, i am pretty new to threading concepts in C++. Can any one please suggest me a way in doing this

Thanks
Arun

For such simple processing as this, a single-threaded program is going to be far faster than your disk by far.

You can only process a file as fast as the disk can read it no matter how many threads you have. Files don't have a multithreading "go-faster" mode.

What exactly are you trying to do? What do you mean by "merge"? Describe what you're doing in more detail, we may be able to help track down the slow step.

P.S: Is this really what your XML looks like, or did you pretty it up for posting?

Thanks for your reply. below are the task I am asked to do

1.The file is about 100GB, I was asked to read the file by multiple thread using producer-consumer scenario and populate the salary~Section in multiple file to split the 100GB load.
2.After it done , I need to merge the file and I need to report the date in the order Section.


Thanks,
Arun

Your CPU's going to be faster than your disk whether you have one thread or 100 threads. Disks do not have a multithreading "go faster" mode.

If you were doing some difficult processing on the data, multithreading would make sense -- a multicore CPU can process several sets of data at once -- but there isn't. The toughest part is just splitting into records, which can't be multithreaded anyway since you must read the file in order to figure out where records begin and end.

Sounds a bit contrived honestly. Is this homework?

Thanks for your reply and BTW this is not a homework. If reading in multithreading in not a feasible solution can you suggest me a way and algorithm to do this in a single threaded mode

Thanks,
Arun

The data you have right now is extremely easy. If the data's actually different that could make it very hard. So please answer this question from earlier:

I know that reading the file record by record and using the xmlparcer we can print this into another file. The real thing is that i was given this file and asked to do in multithreading mode using XML parser. And then I will be given the original XML file which I have to modify this code and make that to work

Also to your note that this a 100GB file.so if we process by a single read it will take time , this is what my thinking I may be wrong.