Sponsored Content
Top Forums Web Development Mysql question: Best way to update a column containing 8 million rows Post 302358012 by muay_tb on Thursday 1st of October 2009 06:20:45 AM
Old 10-01-2009
Quote:
Originally Posted by scottn
Hi.

Sorry, I'm not so familiar with mySQL, but as you didn't get an answer yet!

This is based on Oracle, but the same ideas should relate.

Firstly it's safe to update a table with however many rows. The worst that could happen is that the update fails and the uncommitted changes are rolled back.


You could:
  • split the updates in to smaller batches
  • add commit statements every few hundred / thousand statements
  • instruct your sql client to commit every few hundred / thousand statements
  • add more redo / undo logs or make your existing ones larger, or both
  • switch off logging on the table (alter table .. nologging if mySQL supports that) to reduce redo / undo (in most cases)
  • disable referential / unique constraint indexes (more for performance)
Smilie Many thanks Scottn

I think I am going down the road of creating a script to run in the background (as a cron task) and commit every X thousand records. I just didn't want to run the update all in one go as this would impact performance and increase logs made.
 

10 More Discussions You Might Find Interesting

1. Web Development

[MYSQL] problem with spaces in rows

Hello. I'm not sure how I can get around this, or what I am doing wrong, but I need some help. :) I want to do an select query looking like this: SELECT venue, SUM( amount ) FROM IWD WHERE venue = 'Foxy Hollow' Unfortunately I need to have spaces in the names in these fields, is... (10 Replies)
Discussion started by: noratx
10 Replies

2. Shell Programming and Scripting

awk command to print only selected rows in a particular column specified by column name

Dear All, I have a data file input.csv like below. (Only five column shown here for example.) Data1,StepNo,Data2,Data3,Data4 2,1,3,4,5 3,1,5,6,7 3,2,4,5,6 5,3,5,5,6 From this I want the below output Data1,StepNo,Data2,Data3,Data4 2,1,3,4,5 3,1,5,6,7 where the second column... (4 Replies)
Discussion started by: ks_reddy
4 Replies

3. UNIX for Dummies Questions & Answers

merging rows into new file based on rows and first column

I have 2 files, file01= 7 columns, row unknown (but few) file02= 7 columns, row unknown (but many) now I want to create an output with the first field that is shared in both of them and then subtract the results from the rest of the fields and print there e.g. file 01 James|0|50|25|10|50|30... (1 Reply)
Discussion started by: A-V
1 Replies

4. UNIX for Dummies Questions & Answers

[Solved] Deleting all rows where the first column equals the second column

Hi, I have a tab delimited text file where the first two columns equal numbers. I want to delete all rows where the value in the first column equals the second column. How do I go about doing that? Thanks! Input: 1 1 ABC DEF 2 2 IJK LMN 1 2 ZYX OPW Output: 1 2 ZYX OPW (2 Replies)
Discussion started by: evelibertine
2 Replies

5. Shell Programming and Scripting

Scrape 10 million pages and save the raw html data in mysql database

I have a list of 10 million page urls. I want those pages scraped and saved in the mysql database as raw html. I own a Linux VPS server with 1GB RAM and WHM/cPanel. I would like to scrape at least 100,000 urls in 24 hours. So can anyone give me some sample shell scripting code? (4 Replies)
Discussion started by: Viruthagiri
4 Replies

6. Shell Programming and Scripting

Converting Single Column into Multiple rows, but with strings to specific tab column

Dear fellows, I need your help. I'm trying to write a script to convert a single column into multiple rows. But it need to recognize the beginning of the string and set it to its specific Column number. Each Line (loop) begins with digit (RANGE). At this moment it's kind of working, but it... (6 Replies)
Discussion started by: AK47
6 Replies

7. Shell Programming and Scripting

MySql split rows

Dear community, I have to split string in table and list all values. I'll skip the code and jump directly to mysql query. This is the table: category title ======= ======= 7,3 title 1 1,3 title 2 1,2,3 title 3 Now, what I need is split category into single... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

8. Shell Programming and Scripting

Update a mysql column via bash script

Hello, I want to check the value of all MySQL columns.(column name is "status") via bash script. If value is "0" at I want to make only single column value to "1" I have many "0" values on mysql database(on "status" column) "0" means it is a draft post. I want to publish a post. I... (2 Replies)
Discussion started by: tara123
2 Replies

9. UNIX for Dummies Questions & Answers

Mysql: How to update value in 27000 rows?

Hello, some member created 27000 posts in wrong section (lol :D) so i need to edit all his entries to get new section ID. SELECT * FROM `phpbb_topics` WHERE `topic_first_poster_name` LIKE "%ozerway%"; this will select all his topics... the column with forum id is named "forum_id" and... (3 Replies)
Discussion started by: postcd
3 Replies

10. Programming

MYSQL - trigger update on record insert or update

Right I have a MYSQL database with table1 with 3 columns, colA, colB and colC. I want to combine the data in the 3 columns into a 4th column names col_comb. Here's the SQL command that works: UPDATE table1 SET `col_comb` = CONCAT( `colA` , ' - ', `colB` , ', ', `colC` ); So now I want this... (5 Replies)
Discussion started by: barrydocks
5 Replies
OCI_EXECUTE(3)															    OCI_EXECUTE(3)

oci_execute - Executes a statement

SYNOPSIS
bool oci_execute (resource $statement, [int $mode = OCI_COMMIT_ON_SUCCESS]) DESCRIPTION
Executes a $statement previously returned from oci_parse(3). After execution, statements like INSERT will have data committed to the database by default. For statements like SELECT, execution per- forms the logic of the query. Query results can subsequently be fetched in PHP with functions like oci_fetch_array(3). Each parsed statement may be executed multiple times, saving the cost of re-parsing. This is commonly used for INSERT statements when data is bound with oci_bind_by_name(3). PARAMETERS
o $statement - A valid OCI statement identifier. o $mode - An optional second parameter can be one of the following constants: Execution Modes +----------------------+---------------------------------------------------+ | Constant | | | | | | | Description | | | | +----------------------+---------------------------------------------------+ | | | |OCI_COMMIT_ON_SUCCESS | | | | | | | Automatically commit all outstanding changes for | | | this connection when the statement has succeeded. | | | This is the default. | | | | | | | | OCI_DESCRIBE_ONLY | | | | | | | Make query meta data available to functions like | | | oci_field_name(3) but do not create a result set. | | | Any subsequent fetch call such as | | | oci_fetch_array(3) will fail. | | | | | | | | OCI_NO_AUTO_COMMIT | | | | | | | Do not automatically commit changes. Prior to PHP | | | 5.3.2 (PECL OCI8 1.4) use OCI_DEFAULT which is | | | equivalent to OCI_NO_AUTO_COMMIT. | | | | +----------------------+---------------------------------------------------+ Using OCI_NO_AUTO_COMMIT mode starts or continues a transaction. Transactions are automatically rolled back when the connection is closed, or when the script ends. Explicitly call oci_commit(3) to commit a transaction, or oci_rollback(3) to abort it. When inserting or updating data, using transactions is recommended for relational data consistency and for performance reasons. If OCI_NO_AUTO_COMMIT mode is used for any statement including queries, and oci_commit(3) or oci_rollback(3) is not subsequently called, then OCI8 will perform a rollback at the end of the script even if no data was changed. To avoid an unnecessary rollback, many scripts do not use OCI_NO_AUTO_COMMIT mode for queries or PL/SQL. Be careful to ensure the appropriate transactional consis- tency for the application when using oci_execute(3) with different modes in the same script. RETURN VALUES
Returns TRUE on success or FALSE on failure. EXAMPLES
Example #1 oci_execute(3) for queries <?php $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'SELECT * FROM employees'); oci_execute($stid); echo "<table border='1'> "; while ($row = oci_fetch_array($stid, OCI_ASSOC+OCI_RETURN_NULLS)) { echo "<tr> "; foreach ($row as $item) { echo " <td>" . ($item !== null ? htmlentities($item, ENT_QUOTES) : "&nbsp;") . "</td> "; } echo "</tr> "; } echo "</table> "; ?> Example #2 oci_execute(3) without specifying a mode example <?php // Before running, create the table: // CREATE TABLE MYTABLE (col1 NUMBER); $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'INSERT INTO mytab (col1) VALUES (123)'); oci_execute($stid); // The row is committed and immediately visible to other users ?> Example #3 oci_execute(3) with OCI_NO_AUTO_COMMIT example <?php // Before running, create the table: // CREATE TABLE MYTABLE (col1 NUMBER); $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'INSERT INTO mytab (col1) VALUES (:bv)'); oci_bind_by_name($stid, ':bv', $i, 10); for ($i = 1; $i <= 5; ++$i) { oci_execute($stid, OCI_NO_AUTO_COMMIT); // use OCI_DEFAULT for PHP <= 5.3.1 } oci_commit($conn); // commits all new values: 1, 2, 3, 4, 5 ?> Example #4 oci_execute(3) with different commit modes example <?php // Before running, create the table: // CREATE TABLE MYTABLE (col1 NUMBER); $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'INSERT INTO mytab (col1) VALUES (123)'); oci_execute($stid, OCI_NO_AUTO_COMMIT); // data not committed $stid = oci_parse($conn, 'INSERT INTO mytab (col1) VALUES (456)'); oci_execute($stid); // commits both 123 and 456 values ?> Example #5 oci_execute(3) with OCI_DESCRIBE_ONLY example <?php $conn = oci_connect('hr', 'welcome', 'localhost/XE'); $stid = oci_parse($conn, 'SELECT * FROM locations'); oci_execute($s, OCI_DESCRIBE_ONLY); for ($i = 1; $i <= oci_num_fields($stid); ++$i) { echo oci_field_name($stid, $i) . "<br> "; } ?> NOTES
Note Transactions are automatically rolled back when connections are closed, or when the script ends, whichever is soonest. Explicitly call oci_commit(3) to commit a transaction. Any call to oci_execute(3) that uses OCI_COMMIT_ON_SUCCESS mode explicitly or by default will commit any previous uncommitted transaction. Any Oracle DDL statement such as CREATE or DROP will automatically commit any uncommitted transaction. Note Because the oci_execute(3) function generally sends the statement to the database, oci_execute(3) can identify some statement syn- tax errors that the lightweight, local oci_parse(3) function does not. SEE ALSO
oci_parse(3). PHP Documentation Group OCI_EXECUTE(3)
All times are GMT -4. The time now is 12:21 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy