Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Using bash script : How to Import data from a dsv file into multiple tables in mysql Post 303030988 by tera on Wednesday 20th of February 2019 09:04:29 AM
Old 02-20-2019
Using bash script : How to Import data from a dsv file into multiple tables in mysql

HI

I have a dsv file that looks like:

Code:
<<BOF>>
record_number|id_number|first name|last name|msisdn|network|points|card number|gender
312|9101011234011|Test Junior|Smith|071 123 4321|MTN|73|1241551413214444|M
313|9012023213011|Bob|Smith|27743334321|Vodacom|3|1231233232323244|M
314|8706055678011|Frank|Frankinson|2771 156 1567|8ta|0|1231123453214444|M
315|9102078765011|Mary|Van Niekerk|+27(0)711236677|CellC|2|1278933213214444|F

316|9005074545011|Susan|Wilson|0821121124|CellC|705|1231233216544444|F
317|9101013232011|Katherine|Jeeves|+271233214|8ta|112|1231233678214444|F
318|9101011234011|Matthew|Matthias|0711111111|MTN||1231555213214444|M
319|9103126666011|Michael|Bay|085-6122-161|8ta|63|1231244413214444|M
320|7506023232300|Tyrone|Olivier|711234322|CellC|89|1234563213214444|M
321|8901020304055|Burt|Jackson|071 4566544|Vodacom|1|4567233213214444|M
<<EOF>>

I also have tables in Mysql looking like this :
Code:
CREATE TABLE `tUSER` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`id_number` varchar(20) NOT NULL,
`first_names` varchar(100) NOT NULL,
`last_name` varchar(100) NOT NULL,
PRIMARY KEY (`id`),
INDEX(`id_number`,`first_names`,`last_name`)
);
CREATE TABLE `tPROFILE` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`tUSER_id` bigint(20) DEFAULT NULL,
`tTYPES_id` bigint(20) DEFAULT NULL,
`value` varchar(100) NOT NULL,
PRIMARY KEY (`id`),
INDEX(`tUSER_id`,`tTYPES_id`,`value`)
);
CREATE TABLE `tTYPES` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`type` varchar(100) NOT NULL DEFAULT '',
`description` varchar(255) NOT NULL,
`deleted` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
INDEX(`type`,`description`,`deleted`)
);

How would i go about importing the data from the dsv file to the 3 tables above using a bash script
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Import data from compressed file

HI I need to import data from a file which is in comressed format but system doesn't have enough space to uncompress file Is there any way so that i can do import from compressed file. (4 Replies)
Discussion started by: ap_gore79
4 Replies

2. Shell Programming and Scripting

Converting tables of row data into columns of tables

I am trying to transpose tables listed in the format into format. Any help would be greatly appreciated. Input: test_data_1 1 2 90% 4 3 91% 5 4 90% 6 5 90% 9 6 90% test_data_2 3 5 92% 5 4 92% 7 3 93% 9 2 92% 1 1 92% ... Output:... (7 Replies)
Discussion started by: justthisguy
7 Replies

3. Shell Programming and Scripting

Data Import perl script

Hi, I have a requirement for creating a Perl Script which will perform Data Import process in an automated way and I am elaborating herewith : Section 1 ) - use the following command line format : "./import.pl -h hostname -p port -f datafile.txt" Section 2) datafile.txt will... (3 Replies)
Discussion started by: scott_apc
3 Replies

4. Shell Programming and Scripting

Reading data from multiple tables from Oracle DB

Hi , I want to read the data from 9 tables in oracle DB into 9 different files in the same connection instance (session). I am able to get data from one table to one file with below code : X=`sqlplus -s user/pwd@DB <<eof select col1 from table1; EXIT; eof` echo $X>myfile Can anyone... (2 Replies)
Discussion started by: net
2 Replies

5. Shell Programming and Scripting

Shell snip to import CSV data into BASH array

I have been trying to write a simple snip of bash shell code to import from 1 to 100 records into a BASH array. I have a CSV file that is structured like: record1,item1,item2,item3,item4,etc.,etc. .... (<= 100 items) record2,item1,item2,item3,item4,etc.,etc. .... (<= 100 items)... (5 Replies)
Discussion started by: dstrout
5 Replies

6. Web Development

mysql query for multiple columns from multiple tables in a DB

Say I have two tables like below.. status HId sName dName StartTime EndTime 1 E E 9:10 10:10 2 E F 9:15 10:15 3 G H 9:17 10:00 logic Id devName capacity free Line 1 E 123 34 1 2 E 345 ... (3 Replies)
Discussion started by: ilan
3 Replies

7. Shell Programming and Scripting

Bash script with python slicing on multiple data files

I have 2 files generated in linux that has common output and were produced across multiple hosts with the same setup/configs. These files do some simple reporting on resource allocation and user sessions. So, essentially, say, 10 hosts, with the same (2) system reporting in the files, so a... (0 Replies)
Discussion started by: jdubbz
0 Replies

8. Shell Programming and Scripting

Append data by looking up 2 tables for multiple files

I want to lookup values from two different tables based on common columns and append. The trick is the column to be looked up is not fixed and varies , so it has to be detected from the header. How can I achieve this at once, for multiple data files, but lookup tables fixed. The two lookup... (5 Replies)
Discussion started by: ritakadm
5 Replies

9. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

10. Shell Programming and Scripting

Shell script automation using cron which query's MySQL Tables

What I have: I have a input.sh (script which basically connect to mysql-db and query's multiple tables to write back the output to output1.out file in a directory) note: I need to pass an integer (unique_id = anything b/w 1- 1000) next to the script everytime I run the script which generates... (3 Replies)
Discussion started by: kkpand
3 Replies
SQL::Translator::Parser::MySQL(3pm)			User Contributed Perl Documentation		       SQL::Translator::Parser::MySQL(3pm)

NAME
SQL::Translator::Parser::MySQL - parser for MySQL SYNOPSIS
use SQL::Translator; use SQL::Translator::Parser::MySQL; my $translator = SQL::Translator->new; $translator->parser("SQL::Translator::Parser::MySQL"); DESCRIPTION
The grammar is influenced heavily by Tim Bunce's "mysql2ora" grammar. Here's the word from the MySQL site (http://www.mysql.com/doc/en/CREATE_TABLE.html): CREATE [TEMPORARY] TABLE [IF NOT EXISTS] tbl_name [(create_definition,...)] [table_options] [select_statement] or CREATE [TEMPORARY] TABLE [IF NOT EXISTS] tbl_name LIKE old_table_name; create_definition: col_name type [NOT NULL | NULL] [DEFAULT default_value] [AUTO_INCREMENT] [PRIMARY KEY] [reference_definition] or PRIMARY KEY (index_col_name,...) or KEY [index_name] (index_col_name,...) or INDEX [index_name] (index_col_name,...) or UNIQUE [INDEX] [index_name] (index_col_name,...) or FULLTEXT [INDEX] [index_name] (index_col_name,...) or [CONSTRAINT symbol] FOREIGN KEY [index_name] (index_col_name,...) [reference_definition] or CHECK (expr) type: TINYINT[(length)] [UNSIGNED] [ZEROFILL] or SMALLINT[(length)] [UNSIGNED] [ZEROFILL] or MEDIUMINT[(length)] [UNSIGNED] [ZEROFILL] or INT[(length)] [UNSIGNED] [ZEROFILL] or INTEGER[(length)] [UNSIGNED] [ZEROFILL] or BIGINT[(length)] [UNSIGNED] [ZEROFILL] or REAL[(length,decimals)] [UNSIGNED] [ZEROFILL] or DOUBLE[(length,decimals)] [UNSIGNED] [ZEROFILL] or FLOAT[(length,decimals)] [UNSIGNED] [ZEROFILL] or DECIMAL(length,decimals) [UNSIGNED] [ZEROFILL] or NUMERIC(length,decimals) [UNSIGNED] [ZEROFILL] or CHAR(length) [BINARY] or VARCHAR(length) [BINARY] or DATE or TIME or TIMESTAMP or DATETIME or TINYBLOB or BLOB or MEDIUMBLOB or LONGBLOB or TINYTEXT or TEXT or MEDIUMTEXT or LONGTEXT or ENUM(value1,value2,value3,...) or SET(value1,value2,value3,...) index_col_name: col_name [(length)] reference_definition: REFERENCES tbl_name [(index_col_name,...)] [MATCH FULL | MATCH PARTIAL] [ON DELETE reference_option] [ON UPDATE reference_option] reference_option: RESTRICT | CASCADE | SET NULL | NO ACTION | SET DEFAULT table_options: TYPE = {BDB | HEAP | ISAM | InnoDB | MERGE | MRG_MYISAM | MYISAM } or ENGINE = {BDB | HEAP | ISAM | InnoDB | MERGE | MRG_MYISAM | MYISAM } or AUTO_INCREMENT = # or AVG_ROW_LENGTH = # or [ DEFAULT ] CHARACTER SET charset_name or CHECKSUM = {0 | 1} or COLLATE collation_name or COMMENT = "string" or MAX_ROWS = # or MIN_ROWS = # or PACK_KEYS = {0 | 1 | DEFAULT} or PASSWORD = "string" or DELAY_KEY_WRITE = {0 | 1} or ROW_FORMAT= { default | dynamic | fixed | compressed } or RAID_TYPE= {1 | STRIPED | RAID0 } RAID_CHUNKS=# RAID_CHUNKSIZE=# or UNION = (table_name,[table_name...]) or INSERT_METHOD= {NO | FIRST | LAST } or DATA DIRECTORY="absolute path to directory" or INDEX DIRECTORY="absolute path to directory" A subset of the ALTER TABLE syntax that allows addition of foreign keys: ALTER [IGNORE] TABLE tbl_name alter_specification [, alter_specification] ... alter_specification: ADD [CONSTRAINT [symbol]] FOREIGN KEY [index_name] (index_col_name,...) [reference_definition] A subset of INSERT that we ignore: INSERT anything ARGUMENTS
This parser takes a single optional parser_arg "mysql_parser_version", which provides the desired version for the target database. Any statement in the processed dump file, that is commented with a version higher than the one supplied, will be stripped. The default "mysql_parser_version" is set to the conservative value of 40000 (MySQL 4.0) Valid version specifiers for "mysql_parser_version" are listed here More information about the MySQL comment-syntax: <http://dev.mysql.com/doc/refman/5.0/en/comments.html> AUTHOR
Ken Youens-Clark <kclark@cpan.org>, Chris Mungall <cjm@fruitfly.org>. SEE ALSO
Parse::RecDescent, SQL::Translator::Schema. perl v5.14.2 2012-05-01 SQL::Translator::Parser::MySQL(3pm)
All times are GMT -4. The time now is 01:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy