MYSQL merge csv data with exisiting table


 
Thread Tools Search this Thread
Top Forums Programming MYSQL merge csv data with exisiting table
# 1  
Old 05-10-2017
MYSQL merge csv data with exisiting table

I have a MYSQL table with demographic data. The addresses contained in the table were entered manually before the advent of online postcode lookup software and, hence, there are a lot of errors when using automated online mailing checking software. I plan to export the data as csv file for a 3rd party to do a bulk correction and then reimport the data. The problem is that the database will continue to be used and the demographic table also includes other data (such as dates and times, etc) than might change in the meantime, for example each row will have the following fields:
Code:
ID, Surname, firstname, dob, street, address2, town, county, postcode, tel, email, dateseen, nextdate

Obviously the address fields for some records will change in the csv file and the date fields may change in the original database table. So how do I go about importing the corrected data fields while leaving the unchanged fields that same? Would it be sensible to export only the address fields?
# 2  
Old 05-12-2017
Quote:
Originally Posted by barrydocks
I have a MYSQL table with demographic data. The addresses contained in the table were entered manually before the advent of online postcode lookup software and, hence, there are a lot of errors when using automated online mailing checking software. I plan to export the data as csv file for a 3rd party to do a bulk correction and then reimport the data. The problem is that the database will continue to be used and the demographic table also includes other data (such as dates and times, etc) than might change in the meantime, for example each row will have the following fields:
Code:
ID, Surname, firstname, dob, street, address2, town, county, postcode, tel, email, dateseen, nextdate

Obviously the address fields for some records will change in the csv file and the date fields may change in the original database table. So how do I go about importing the corrected data fields while leaving the unchanged fields that same? Would it be sensible to export only the address fields?
The most important requirement here would be a primary key on your table - a non-NULL column or set of columns that uniquely identify each record in your table.
If you have a primary key, your task will be much easier. If not, then it will be fiendishly difficult.
Looking at your column list, I assume that "ID" is the primary key column i.e. it has a value in each row and that value is unique across the entire table.
So you could do the following:
  1. First determine which columns will be changed by the 3rd party. Is it just address2, or a bunch of columns like (street, address2, town) etc.?
  2. Export the ID and the set of columns that could be updated by the 3rd party. If they need the other columns like (surname, firstname) etc. for context, then you could export them as well, but mention that any changes to those columns will not be applied in your original table. Essentially, you need to draw the line on changes to which columns are acceptable and which ones are not. Of course, the ID column shall not be changed. The primary key is immutable.
  3. Your 3rd party gets the CSV file, updates the address column(s) and sends it back to you. I am assuming that they will not add any new records with new IDs; they will only update the existing records.
  4. You load the updated records in a separate table in your database.
  5. Join the original table with the updated table on the ID column and update only the address column(s) in the original table picking their new values from the updated table.

I do not have MySQL to work with at the moment, but I have created a few SQL statements and tested them in SQLFiddle. (www.sqlfiddle.com)

Code:
-- The original table
create table demographics1 (id int primary key, name varchar(10), dob date, city varchar(20), country varchar(2));
insert into demographics1 (id, name, dob, city, country) values
(1, 'Aaron', DATE'1980-01-01', 'London', 'UK'),
(2, 'Beth', DATE'1981-01-01', 'Chicago', 'US'),
(3, 'Colin', DATE'1982-01-01', 'Berlin', 'DE')
;
  
-- The new table loaded with the updated data in the CSV file obtained from the 3rd party.
create table demographics2 (id int primary key, name varchar(10), dob date, city varchar(20), country varchar(2));
insert into demographics2 (id, name, dob, city, country) values
(1, 'Aaron', DATE'1980-01-01', 'London', 'UK'),
(2, 'Beth', DATE'1981-01-01', 'New York', 'US'),
(3, 'Calvin', DATE'1982-11-11', 'Zurich', 'CH'),
(4, 'Dennis', DATE'1983-01-01', 'Tokyo', 'JP'),
(5, 'Emma', DATE'1984-01-01', 'Paris', 'FR')
;
  
--
select * from demographics1;
--
select * from demographics2;
 
-- Original data is in "lft" table alias, new data is in "rgt" table alias.
-- Only the updates to "city" and "country" will be applied to the original table.
-- Note that the change in "name" and "dob" for ID=3 will *NOT* be applied to the original table.
-- Also note that the new IDs (4, 5) in the "rgt" table will *NOT* be added to the original table.
update demographics1 as lft
inner join demographics2 as rgt on lft.id = rgt.id
set lft.city = rgt.city, lft.country = rgt.country;
  
--
select * from demographics1;
--
select * from demographics2;


Last edited by rbatte1; 05-15-2017 at 07:33 AM.. Reason: Converted textual numbered list to formatted numbered list
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Send Data to MySQL Table Columns

I have two scripts, each script reads an individual data file and copies specific lines of data and sends to MySQL table. Only difference is, each script sends data to a separate column on the same DB. I want to use one script to populate DB table and have data look horizontal, with no overlapping.... (3 Replies)
Discussion started by: SysAdminRialto
3 Replies

2. UNIX for Beginners Questions & Answers

Merge cells in all rows of a HTML table dynamically.

Hello All, I have visited many pages in Unix.com and could find out one solution for merging the HTML cells in the 1st row. (Unable to post the complete URL as I should not as per website rules). But, however I try, I couldn't achieve this merging to happen for all other rows of HTML... (17 Replies)
Discussion started by: Mounika
17 Replies

3. Linux

Parsing - export html table data as .csv file?

Hi all, Is there any out there have a brilliant idea on how to export html table data as .csv or write to txt file with separated comma and also get the filename of link from every table and put one line per rows each table. Please see the attached html and PNG of what it looks like. ... (7 Replies)
Discussion started by: lxdorney
7 Replies

4. UNIX for Dummies Questions & Answers

Need help combining txt files w/ multiple lines into csv single cell - also need data merge

:confused:Hello -- i just joined the forums. I am a complete noob -- only about 1 week into learning how to program anything... and starting with linux. I am working in Linux terminal. I have a folder with a bunch of txt files. Each file has several lines of html code. I want to combine... (2 Replies)
Discussion started by: jetsetter
2 Replies

5. Shell Programming and Scripting

Merge Two Tables with duplicates in first table

Hi.. File 1: 1 aa rep 1 dd rep 1 kk rep 2 bb sad 2 ss sad 3 ee dam File 2 1 apple fruit 2 mango tree 3 lilly flower output: 1 aaple fruit aa,dd,kk rep (7 Replies)
Discussion started by: empyrean
7 Replies

6. UNIX and Linux Applications

mysql table disappear

I have set a mysql file to excute everyday morning to generate a html file displayng 2 tables from the database. Sometime they cannot be shown, and it shows the tables are not existed. I have not drop any table, and those 2 tables are not used by any other excution. Anybody know what is happening?... (0 Replies)
Discussion started by: c203040
0 Replies

7. Shell Programming and Scripting

how to print out data from mysql table in perl script

I'm having trouble with this code. if i do .\read.pl -u user it prints out 2010-12-20 12:00:00 host1 <cmd>a 2010-12-20 12:00:01 host1 <cmd> <execute> 2010-12-20 12:00:02 host1 <cmd>b 2010-12-20 12:00:03 host1 <cmd>c however, if i enter .\read.pl -h host1 it should... (3 Replies)
Discussion started by: kpddong
3 Replies

8. UNIX for Dummies Questions & Answers

Storing data from a table into a csv file

Hi I need to write a bash script to take the data stored in 3 oracle tables .. and filter them and store the results in a csv file. It is an Oracle database Thank you (1 Reply)
Discussion started by: ladyAnne
1 Replies

9. Shell Programming and Scripting

select data from oracle table and save the output as csv file

Hi I need to execute a select statement in a solaris environment with oracle database. The select statement returns number of rows of data. I need the data to be inserted into a CSV file with proper format. For that we normally use "You have to select all your columns as one big string,... (2 Replies)
Discussion started by: rdhanek
2 Replies

10. Shell Programming and Scripting

Help Inserting data in mysql table

Cant understand the error #!/bin/bash temp="" A="" D=$(date +"%Y-%m-%d") H=$(date +"%R") temp=$(wget -q -O - website | grep -o "Temperature:]**" | grep \-E -o "+") mysql -D "weather_wise" -e "INSERT INTO weather (Date, Hour, Degrees) VALUES ($D,$H, $temp)"; my data types for... (11 Replies)
Discussion started by: vadharah
11 Replies
Login or Register to Ask a Question