Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

sh and MySQL LOAD DATA

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sh and MySQL LOAD DATA
# 1  
Old 03-11-2009
sh and MySQL LOAD DATA
I have a csv file in a folder that is roughly 500,000 rows long.

Rather than using PHP, I would like to use SH to run a MYSQL LOAD DATA command to load the data in, as I think it would be much faster and would not cause any memory problems associated with PHP.

But the problem is, I am not sure how to combine sh and my MySQL LOAD DATA command as it's on several lines and is very, very long.

The following is a sample of the actual load data I use.

How do I make sh to connect to mysql, and run this following command?

Any help on this would be great!

Code:
LOAD DATA LOCAL INFILE '/path/to/traveldata/traveldata.csv' 
INTO TABLE `offers` 
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 2 LINES
(
OfferProcessType,
TradingName,
OfferLabelDesc,
DestinationName,
DepAirportCode,
TelephoneNumber,
ReturnAirportCode,
ArrivalAirportCode,
ABTA,
ATOL,
OtherBonding,
BoardTypesShort,
AccomodationShort,
AllocationShort,
OperatorShort,
@DepartureDate,
Price,
ShareBasisAdult,
BulkLoaderRef,
Email,
PublicNotes,
CCC,
TOD,
DCC,
AMEX,
CDW,
BookingFee,
AccomodationName,
Duration,
ShareBasisChild,
Rating,
OfferLink,
AccomodationRef,
Transfers,
OperatorsRating,
QuoteRef,
Url,
OutDepTime,
OutArrTime,
InDepTime,
InArrTime
)
SET
DepartureDate = (str_to_date(@DepartureDate, '%d-%M-%y'))
;

# 2  
Old 03-11-2009
On the command line, you can just use mysqlimport. It's the direct equivalent of the command you want to run, and takes the same arguments.

If you're familiar with sqlldr for Oracle, it works in a similar fashion.
# 3  
Old 03-11-2009
Also, you must make sure that your MySQL configuration permits loading local data. I think this is normally the case, by default, but some folks turn it off with the following setting in my.cnf

Code:
local-infile=0

(off)
# 4  
Old 03-11-2009
This looks like a good solution, however... it states;

Quote:
...mysqlimport strips any extension from the file name and uses the result to determine the name of the table into which to import the file's contents. For example, files named patient.txt, patient.text, and patient all would be imported into a table named patient.
So this means that the CSV file has to be named the same as the table -- but what if I want to import into a specific table?

I don't want to be reliant upon the CSV filename, especially if the name suddenly changes then the mysqlimport code will result in an error.
# 5  
Old 03-11-2009
No, you have to rename the file to match the table. That's just the way mysqlimport works.
# 6  
Old 03-11-2009
Quote:
No, you have to rename the file to match the table. That's just the way mysqlimport works.
Oh... I guess I'll just have to use PHP or another solution as I don't have any control over the incoming filename.
# 7  
Old 03-11-2009
Quote:
Originally Posted by worchyld
Oh... I guess I'll just have to use PHP or another solution as I don't have any control over the incoming filename.
Sure you do. If you're writing a shell script, have the script rename it. Of course you can always use a programming language to do the inserts, but it's probably going to be less efficient.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel sqlldr to load data.

I am using SQLLDR to load data in DB.For parallel loading I'm using nohup command. The requirement is: I have different files within a directories. Ex: 1) Dir/folder_A/AE.txt 2) Dir/folder_A/DM.txt 3) Dir/folder_B/CM.txt I need to loop through directories and load the data... (1 Reply)
Discussion started by: Pratiksha Mehra
1 Replies

2. UNIX for Dummies Questions & Answers

Load UNIX data into excel sheet

Hi, i have some data in a temporary file in Unix (the data is taken from the result of an SQL query). Now i want to dump that data into an excel sheet. How to do that. Someone please advise. Thanks Regards, Vinit (3 Replies)
Discussion started by: vinit raj
3 Replies

3. Shell Programming and Scripting

Load data from a flat file to oracle.

I have a flat file with records like Header 123 James Williams Finance2000 124 Pete Pete HR 1500 125 PatrickHeather Engg 3000 Footer The structure is: Eno:4 characters Name:8 characters Surname : 9 characters Dept:7 characters Sal:4characters These are sample... (1 Reply)
Discussion started by: Shivdatta
1 Replies

4. Web Development

script to load data from csv file

hello i want a script to load the data line by line from a csv file into a mysql table (3 Replies)
Discussion started by: srpa01red
3 Replies

5. Shell Programming and Scripting

executing mysql load statement from shell script

Hi, I have a piece of shell script which will connect to mysql database and execute a load statement(which will load datas in a file to the database table).The code is working and the data is in the tables. Now my requirement is, i need to grab the output from the load statement... (4 Replies)
Discussion started by: DILEEP410
4 Replies

6. Shell Programming and Scripting

How to load different type of data in a file to two arrays

Hi, I have tried to find some sort of previous similar thread on this but not quite close to what I want to achieve. Basically I have two class of data in my file..e.g 1,1,1,1,1,2,yes 1,2,3,4,5,5,yes 2,3,4,5,5,5,no 1,2,3,4,4,2,no 1,1,3,4,5,2,no I wanted to read the "yes" entry to an... (5 Replies)
Discussion started by: ahjiefreak
5 Replies

7. Shell Programming and Scripting

Automatically Load data from all files in directory

I'm new in Unix shell scripting and i need someone to help me to make Script that run all time watching my directory that files uploaded to it via FTP (/mydir/incoming_files), if any files exists in it then (if many files exists, then sort files and load them ascending) it‘ll checks the size of the... (1 Reply)
Discussion started by: m_fighter
1 Replies

8. Programming

Load text file into a MySQL field

Hello, maybe this post is offtopic, sorry for the inconveniencies. I found examples about how to populate different fields from a text file (with MySQL, the LOAD DATA INFILE sentece), but not about how to load a complete plain text file into a concrete database field. Could you please tell me if... (1 Reply)
Discussion started by: aristegui
1 Replies

9. Shell Programming and Scripting

Need help in wrting Load Script for a Load-Resume type of load.

hi all need your help. I am wrting a script that will load data into the table. then on another load will append the data into the existing table. Regards Ankit (1 Reply)
Discussion started by: ankitgupta
1 Replies
Login or Register to Ask a Question