damonparker.org » SQL

Quickly Copy MySQL Database to Remote Host

damonp — Fri, 31 Aug 2007 16:29:50 +0000

Copy an entire database:

mysqldump LOCAL_DBNAME | ssh USER@REMOTE_HOST mysql -p REMOTE_DBNAME

Copy a single table:

mysqldump LOCAL_DBNAME LOCAL_TABLE | ssh USER@REMOTE_HOST -p REMOTE_DBNAME REMOTE_TABLE

This shortcut only works if you can access the local DB without a password. If you have to login to both local and remote MySQL servers, the MySQL password prompts get mashed together. You could specify the password on the command line like

-pPASSWORD

but your shell may keep the password in its history so that anyone with access to your account could pick through your shell history and retrieve the password. Using the MySQL password prompt doesn’t do this.

Data Manipulation – Splitting Surname and Name

damonp — Thu, 21 Jun 2007 13:04:55 +0000

For a new client project I was provided with database of users consisting of names, email addresses and managers.

CREATE TABLE `users` (
`name` varchar(255) NOT NULL,
`email` varchar(255) NOT NULL,
`manager` varchar(255) NOT NULL,
`status` varchar(255) NOT NULL
)

The name column contained both first and last names formatted last name first like:

lastname, fistname
Parker, Damon

MySQL’s SUBSTRING_INDEX() and TRIM() functions allow us to make quick work of this data manipulation. Using the query below and a new table similar to the first except with separate fields for first and last name we can copy the data to the new table and split the names in the process.

INSERT INTO users_clean (first_name, last_name, email, manager, STATUS) SELECT TRIM(SUBSTRING_INDEX( `name` , ',', 1 )) AS first_name, TRIM(SUBSTRING_INDEX( SUBSTRING_INDEX( `name` , ',', 2 ) , ',', -1 )) AS last_name, email, STATUS
FROM `users`

Adding Human Readable Date to MySQL Query

damonp — Tue, 15 May 2007 21:43:40 +0000

SELECT *, FROM_UNIXTIME( 't_stamp_column') AS date FROM 'tablename'

For formatting the date see MySQL’s date functions information.

Easy MySQL Sorting Mistake

damonp — Thu, 05 Apr 2007 17:00:57 +0000

If you have a compound ORDER BY clause with a direction (ASC or DESC), make sure to specify the directions of all columns.

Correct:

ORDER BY col1 DESC, col2 DESC

Wrong:

ORDER BY col1, col2 DESC

The latter forces a filesort.

Quick and Dirty MySQL Backup Script

damonp — Wed, 14 Mar 2007 10:22:53 +0000

I previously posted a Quick and Dirty MySQL Backup snippet that has gotten a bit of traffic. It works well in a pinch, but it doesn’t have any features. I use it primarily as a safety backup when performing maintenance on a MySQL server.

Several of my clients have requested a script that they can use to do on demand one-off backups. I rolled the snippet up into a very quick-and-dirty MySQL backup.

Download mysql_quick_back.sh

Installation

Copy mysql_quick_back.sh to a directory in your path (~/bin/ for me).
Set execute bits with
chmod 755 mysql_quick_back.sh
Set MySQL user/pass, directory to save backups to and notification email in script configuration
# START configure
# MySQL user, needs SELECT privileges
# Create a new user with these permissions! More secure than using a user
# with full permissions
user=

# MySQL password
pass=

# admin email address
adminemail=

# backup location without trailing slash
# set to current directory
backuppath=.
# or set to your home directory
#backuppath=/home/admin/db_backs

Usage

Simply execute the script

NOTE

For a full MySQL backup application, check out the original MySQL Backup by Peter Falkenberg Brown. MySQL Backup is robust enough to use as a daily backup. I use it on most of the servers I manage.

ZenCart Manufacturers System Errors

damonp — Fri, 12 Jan 2007 17:46:54 +0000

In ZenCart versions immediately prior to 1.3.6 using many of the public side manufacturers functions may produce this error:

Warning: constant(): Couldn't find constant in .../includes/init_includes/init_add_crumbs.php on line 45

The error is caused by a missing database column that was added in the official 1.3.6 release. If upgrade is not immediately possible the following SQL snippet will correct the problem in most cases:

ALTER TABLE get_terms_to_filter ADD get_term_table varchar(64) NOT NULL;

ALTER TABLE get_terms_to_filter ADD get_term_name_field varchar(64) NOT NULL;

UPDATE get_terms_to_filter SET get_term_table = 'TABLE_MANUFACTURERS', get_term_name_field = 'manufacturers_name' WHERE get_term_name = 'manufacturers_id';

UPDATE get_terms_to_filter SET get_term_table = 'TABLE_MUSIC_GENRE', get_term_name_field = 'music_genre_name' WHERE get_term_name = 'music_genre_id';

UPDATE get_terms_to_filter SET get_term_table = 'TABLE_RECORD_COMPANY', get_term_name_field = 'record_company_name' WHERE get_term_name = 'record_company_id';

If you are using other product types, some additional UPDATE statements will be required for your new product types.

I found this solution after some searching on the ZenCart forums.

MySQL Slow Query Logging

damonp — Tue, 05 Dec 2006 23:52:27 +0000

Add to /etc/my.cnf under [mysqld] stanza

MySQL5

log_slow_queries = /var/log/mysqld-slow.log
long_query_time = 1

MySQL3, MySQL4

log-slow-queries=/var/log/mysqld-slow.log
set-variable = long_query_time=1

ZenCart Orders Total Weight Query

damonp — Wed, 29 Nov 2006 19:00:05 +0000

SELECT op.products_quantity * SUM( products_weight )
FROM products p
JOIN orders_products op
WHERE p.products_id
IN (
SELECT products_id
FROM orders_products
WHERE orders_id = 'ORDER_ID'
)
AND op.orders_id = 'ORDER_ID'

Replace ORDER_ID with the orders_id in question.

Requires MySQL 4.1 or better for subselect support

MySQL Database Performance – ZenCart Sales Report

damonp — Mon, 23 Oct 2006 19:13:36 +0000

One of my client’s ZenCart installs was loading the server down so much to generate sales reports with the Sales Report v1 contribution that the public side was noticeably less responsive for the couple of minutes it took to build the report. The site has been running for over three years and contains 9k orders, so there is a lot of data to weed through even for a weekly report. A quick look at the slow query log listed the following query as the culprit:

SELECT count( op.products_id ) AS attr_cnt, o.orders_id, opa.orders_products_id, opa.products_options, opa.products_options_values, opa.options_values_price, opa.price_prefix
FROM orders_products_attributes opa, orders o, orders_products op
WHERE o.orders_id = opa.orders_id
AND op.orders_products_id = opa.orders_products_id
AND o.date_purchased >= '2006-10-18T00:00:00'
AND o.date_purchased < '2006-10-19T00:00:00'
AND op.products_id =179
GROUP BY products_options_values
ORDER BY orders_products_id;

Running this query through EXPLAIN shows 20978 rows examined. In this case, thats 20978 rows per day included in the sales report.

The main problem with this query is the ORDER BY clause which cannot utilize an exisiting index. Adding the following index:

ALTER TABLE 'orders_products_attributes' ADD INDEX ( 'orders_products_id' )

limits the rows examined for each query to 68 (results).

This simple change allows the sales report to load in seconds as opposed to minutes with no noticeable lag on the public side.

The create table SQL for the current version of ZenCart (v1.3.5) contains the following index in orders_products_attributes:

...
KEY idx_orders_id_prod_id_zen (orders_id,orders_products_id)
...

This index is used in the query in question. In my tests it was only slightly slower than the single column index above. EXPLAIN showed about 270 rows examined with the two column index on my dataset while the single column index only examined 68 rows.

Quick and Dirty MySQL Backup

damonp — Fri, 04 Aug 2006 16:56:45 +0000

This is great for a quick backup of all databases before performing maintenance or check/repair.

mysqldump --all-databases -p | gzip > db_dump-YYYYMMDD.gz

Replace YYYYMMDD with Year, Month and Day.