Large XML to MySQL - fast way


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Large XML to MySQL - fast way
# 1  
Old 07-16-2013
Large XML to MySQL - fast way

Hello,

Sorry for my bad english.

I need to improve performance in project managing large data, these data are exported to a MySql from XML.

Now I use PHP (XMLReader ()) to do this job.

I need a faster way to do this process.

Which do you think is the best way?

Example:
(the item total count is 80000)

PHP Code:
<?xml version='1.0' encoding='ISO-8859-1' ?>
<items>
    <item>
        <id>xx</id>
        <date>xx/xx/xxxx</date>
        <aaa a="x" b="x" />
        <bbb a="x" b="x" />
        <ccc a="x" b="x" />
        <ddd a="x" b="x" />
        <eee a="x" b="x" />
        <fff a="x" b="x" />
        <ggg a="x" b="x" />
        <hhh a="x" b="x" />
        <iii a="x" b="x" />
        <jjj a="x" b="x" />
    </item>
    <item>...</item>
    <item>...</item>
    <item>...</item>
    <item>...</item>
    <item>...</item>
    <item>...</item>
</items>
# 2  
Old 07-16-2013
Try load xml (MySQL 5.5+).

Last edited by radoulov; 07-16-2013 at 12:01 PM..
# 3  
Old 07-17-2013
You think this way would be faster to use a component executing it on a bash script.

For example something like this ...

vtd-xml
xml-shell
xmlsh
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split large xml into mutiple files and with header and footer in file

Split large xml into mutiple files and with header and footer in file tried below it splits unevenly and also i need help in adding header and footer command : csplit -s -k -f my_XML_split.xml extrfile.xml "/<Document>/" {1} sample xml <?xml version="1.0" encoding="UTF-8"?><Recipient>... (36 Replies)
Discussion started by: karthik
36 Replies

2. Shell Programming and Scripting

Help with Splitting a Large XML file based on size AND tags

Hi All, This is my first post here. Hoping to share and gain knowledge from this great forum !!!! I've scanned this forum before posting my problem here, but I'm afraid I couldn't find any thread that addresses this exact problem. I'm trying to split a large XML file (with multiple tag... (7 Replies)
Discussion started by: Aviktheory11
7 Replies

3. Shell Programming and Scripting

Curl download zip extract large xml file

Hi i have a php script that works 100% however i don't want this to run on php because of server limits etc. Ideally if i could convert this simple php script to a shell script i can set it up to run on a cron. My mac server has curl on it. So i am assuming i should be using this to download the... (3 Replies)
Discussion started by: timgolding
3 Replies

4. Programming

Fast string removal from large text collection

Hi All, I don't want any codes for this problem. Just suggestions: I have a huge collection of text files (around 300,000) which look like this: 1.fil orange apple dskjdsk computer skjks The entire text collection (referenced above) has about 1 billion words. I have created... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

5. Shell Programming and Scripting

Help needed for parsing large XML with awk.

My XML structure looks like: <?xml version="1.0" encoding="UTF-8"?> <SearchRepository> <SearchItems> <SearchItem> ... </SearchItem> <SearchItem> ... ... (1 Reply)
Discussion started by: jasonjustice
1 Replies

6. Shell Programming and Scripting

Help with passing XML variables to MySQL DB via PHP

Hi everybody, I need the help of the Unix community once again :) I have some code which queries an XML feed and displays the results for me. I would like to enter the XML output in to my database, but I am having trouble passing the variables while INSERTing. Here is my code that I need to... (0 Replies)
Discussion started by: o0110o
0 Replies

7. Shell Programming and Scripting

Using AWK to separate data from a large XML file into multiple files

I have a 500 MB XML file from a FileMaker database export, it's formatted horribly (no line breaks at all). The node structure is basically <FMPXMLRESULT> <METADATA> <FIELD att="............." id="..."/> </METADATA> <RESULTSET FOUND="1763457"> <ROW att="....." etc="...."> ... (16 Replies)
Discussion started by: JRy
16 Replies

8. Programming

Read/Write a fairly large amount of data to a file as fast as possible

Hi, I'm trying to figure out the best solution to the following problem, and I'm not yet that much experienced like you. :-) Basically I have to read a fairly large file, composed of "messages" , in order to display all of them through an user interface (made with QT). The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies

9. Shell Programming and Scripting

parsing large CDR XML file

Dear Freind in the file attached how parse the data to be like a normal table :D (3 Replies)
Discussion started by: saifsafaa
3 Replies
Login or Register to Ask a Question