Large XML to MySQL - fast way Post: 302833137

Sponsored Content

Top Forums Shell Programming and Scripting Large XML to MySQL - fast way Post 302833137 by stendelis on Tuesday 16th of July 2013 10:27:01 AM

07-16-2013

Registered User

Large XML to MySQL - fast way

Hello,

Sorry for my bad english.

I need to improve performance in project managing large data, these data are exported to a MySql from XML.

Now I use PHP (XMLReader ()) to do this job.

I need a faster way to do this process.

Which do you think is the best way?

Example:
(the item total count is 80000)

PHP Code:


<?xml version='1.0' encoding='ISO-8859-1' ?>
<items>
    <item>
        <id>xx</id>
        <date>xx/xx/xxxx</date>
        <aaa a="x" b="x" />
        <bbb a="x" b="x" />
        <ccc a="x" b="x" />
        <ddd a="x" b="x" />
        <eee a="x" b="x" />
        <fff a="x" b="x" />
        <ggg a="x" b="x" />
        <hhh a="x" b="x" />
        <iii a="x" b="x" />
        <jjj a="x" b="x" />
    </item>
    <item>...</item>
    <item>...</item>
    <item>...</item>
    <item>...</item>
    <item>...</item>
    <item>...</item>
</items>

stendelis

View Public Profile for stendelis

Find all posts by stendelis

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

parsing large CDR XML file

Dear Freind in the file attached how parse the data to be like a normal table :D

2. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that...

3. Shell Programming and Scripting

Using AWK to separate data from a large XML file into multiple files

I have a 500 MB XML file from a FileMaker database export, it's formatted horribly (no line breaks at all). The node structure is basically <FMPXMLRESULT> <METADATA> <FIELD att="............." id="..."/> </METADATA> <RESULTSET FOUND="1763457"> <ROW att="....." etc="...."> ...

4. Shell Programming and Scripting

Help with passing XML variables to MySQL DB via PHP

Hi everybody, I need the help of the Unix community once again :) I have some code which queries an XML feed and displays the results for me. I would like to enter the XML output in to my database, but I am having trouble passing the variables while INSERTing. Here is my code that I need to...

5. Shell Programming and Scripting

Help needed for parsing large XML with awk.

My XML structure looks like: <?xml version="1.0" encoding="UTF-8"?> <SearchRepository> <SearchItems> <SearchItem> ... </SearchItem> <SearchItem> ... ...

6. Programming

Fast string removal from large text collection

Hi All, I don't want any codes for this problem. Just suggestions: I have a huge collection of text files (around 300,000) which look like this: 1.fil orange apple dskjdsk computer skjks The entire text collection (referenced above) has about 1 billion words. I have created...

7. Shell Programming and Scripting

Curl download zip extract large xml file

Hi i have a php script that works 100% however i don't want this to run on php because of server limits etc. Ideally if i could convert this simple php script to a shell script i can set it up to run on a cron. My mac server has curl on it. So i am assuming i should be using this to download the...

8. Shell Programming and Scripting

Help with Splitting a Large XML file based on size AND tags

Hi All, This is my first post here. Hoping to share and gain knowledge from this great forum !!!! I've scanned this forum before posting my problem here, but I'm afraid I couldn't find any thread that addresses this exact problem. I'm trying to split a large XML file (with multiple tag...

9. Shell Programming and Scripting

Split large xml into mutiple files and with header and footer in file

Split large xml into mutiple files and with header and footer in file tried below it splits unevenly and also i need help in adding header and footer command : csplit -s -k -f my_XML_split.xml extrfile.xml "/<Document>/" {1} sample xml <?xml version="1.0" encoding="UTF-8"?><Recipient>...

LEARN ABOUT PHP

apache_child_terminate

APACHE_CHILD_TERMINATE(3)						 1						 APACHE_CHILD_TERMINATE(3)

apache_child_terminate - Terminate apache process after this request

SYNOPSIS

       bool apache_child_terminate (void  )

DESCRIPTION

       apache_child_terminate(3)  will register the Apache process executing the current PHP request for termination once execution of PHP code is
       completed. It may be used to terminate a process after a script with high memory consumption has been run as memory will  usually  only	be
       freed internally but not given back to the operating system.

RETURN VALUES

	Returns  TRUE  if  PHP is running as an Apache 1 module, the Apache version is non-multithreaded, and the child_terminate PHP directive is
       enabled (disabled by default). If these conditions are not met, FALSE is returned and an error of level E_WARNING is generated.

CHANGELOG

       +--------+---------------------------------------------------+
       |Version |						    |
       |	|						    |
       |	|		     Description		    |
       |	|						    |
       +--------+---------------------------------------------------+
       | 5.4.0	|						    |
       |	|						    |
       |	|  This function became  available  under  FastCGI. |
       |	| Previously,  it  was	supported only when PHP was |
       |	| installed as an Apache module.		    |
       |	|						    |
       +--------+---------------------------------------------------+
NOTES

       Note

	      This function is not implemented on Windows platforms.

SEE ALSO

       exit(3).

PHP Documentation Group 												 APACHE_CHILD_TERMINATE(3)