Thursday, June 11, 2009

Converting an Oracle Schema to MySQL

Both Oracle and MySQL provide a number of sample schemas so users can build different schema environments and load them with data. So I thought I'd get one of the Oracle sample schemas and go through the exercise of converting it to a MySQL schema. I could have gone in either direction, since this is more of a MySQL blog I made MySQL the destination. Before I start let me introduce some of the Oracle and MySQL sample schemas.

Oracle Sample Schemas:
  • Human Resources: (HR) is a simple schema containing tables with basic primary and foreign key relationships such as: employees, departments, jobs, locations, regions, countrys, jobs, etc. This can be extended to support Oracle Internet Directory demos.
  • Order Entry (OE) is a more complex set of tables demonstrating the different datatypes supported by Oracle. Tables dealing with products, sales, customers, etc. are used in this schema.
  • Online Catalog (OC) is a supporting schema that demonstrates object-relational database objects built inside the OE schema.
  • Product Media (PM) is a schema that supports multimedia datatypes. Audio, video, images, large text data are all used in this schema.
  • Queued Shipping (QS) to demonstrate Oracle Advanced Queuing capabilities. Advanced Queuing and XML are used in this schema.
  • Sales History (SH) is a schema supporting a large number of tables. Features like partitioning and Oracle analytics can be used with this schema.

MySQL Sample Schemas:
  • World is a basic schema using three tables City, Country and CountryLanguages that use MyISAM tables.
  • Sakila is a schema that adds additional complexity using an online DVD store to demonstrate more of the MySQL 5.1 features.
  • Employees is a schema with a much larger data set containing employees, deparments, salaries, titles, etc that are very intuitive to work with. There are approximately 4 million records to work with. Different storage engines and partitioning are often benchmarked using this schema.
Summary of changes made to convert Oracle Schema to MySQL Schema

Here is a summary of changes made to convert the Oracle schema script to a MySQL schema script. The goal here is to provide a level of understanding of converting a schema.
  • Comment out SQL*Plus specific commands.
  • Change REM to --
  • Change VARCHAR2 to VARCHAR
  • Change NUMBER to INT
  • Change NUMBER(x,x) to DECIMAL(x,x)
  • Change CONSTRAINT keyword Syntax
  • For InnoDB tables, define PRIMARY KEY as part of the CREATE TABLE command and not a separate CREATE INDEX command.
  • Make sure tables are created as InnoDB. set storage_engine=innodb;
  • Modify constraint syntax for FOREIGN KEY definitions.
  • Fully quality PRIMARY KEY column in FOREIGN KEY constraint.
  • Remove READ ONLY clause with CREATE VIEW.

Converting the Oracle HR sample schema to MySQL

I decided the Oracle HR schema would be a good sample schema to convert to a MySQL schema to demonstrate things to consider in creating schema scripts. I started with the cre_hr.sql script and converted it to a cre_hr_mysql.sql script. The Oracle cre_hr.sql script I started with can be found at:
  • http://www.oracle.com/technology/obe/sqldev_obe/osdm/files/hr_cre.sql

MySQL equivalent of the Oracle HR schema script

Here is the corresponding MySQL create script. The only thing I did not address were the sequential incremental values used with Oracle sequences. I included the Oracle CREATE SEQUENCE commands in so it would show the adjustments I made to use autoincrement but also show that the incremental values would need to be addressed. A little more work needs to be done here to verify the data loads would create the same values in an Oracle environment. As I mentioned my main goal was to give someone a feel for the code differences in creating a schema between Oracle and MySQL.

-- hr_cre_mysql.sql


/*
Notes of changes:
Comment out SQL*Plus specific commands.
Change REM to --
Change VARCHAR to VARCHAR
Change INT to INT
Change INT(x,x) to DECIMAL(x,x)
Change CONSTRAINT keyword Syntax
For InnoDB tables, define PRIMARY KEY as part of the CREATE TABLE command and not a separate CREATE INDEX command.
Make sure tables are created as InnoDB.
set storage_engine=innodb;
Modify constraint syntax for FOREIGN KEY definitions.
Fully quality PRIMARY KEY column in FOREIGN KEY constraint.
Remove READ ONLY clause with CREATE VIEW.
*/

DROP DATABASE IF EXISTS hr;
CREATE DATABASE hr;
USE hr;

CREATE TABLE regions
( region_id INT NOT NULL
, region_name varchar(25),
primary key (region_id)
);


CREATE TABLE countries
( country_id CHAR(2) NOT NULL
, country_name varchar(40)
, region_id INT
, PRIMARY KEY (country_id)
) ;


ALTER TABLE countries
ADD (FOREIGN KEY (region_id)
REFERENCES regions(region_id)
) ;

CREATE TABLE locations
( location_id INT(4)
, street_address varchar(40)
, postal_code varchar(12)
, city varchar(30) NOT NULL
, state_province varchar(25)
, country_id CHAR(2)
, PRIMARY KEY (location_id)
) auto_increment=3300;

/*
CREATE SEQUENCE locations_seq\
START WITH 3300\
INCREMENT BY 100\
MAXVALUE 9900\
NOCACHE\
NOCYCLE;\
*/

ALTER TABLE locations
ADD ( FOREIGN KEY (country_id)
REFERENCES countries(country_id)
) ;

CREATE TABLE departments
( department_id INT(4)
, department_name varchar(30) NOT NULL
, manager_id INT(6)
, location_id INT(4)
, PRIMARY KEY (department_id)
) auto_increment=280;

ALTER TABLE departments
ADD ( FOREIGN KEY (location_id)
REFERENCES locations (location_id)
) ;

/*
CREATE SEQUENCE departments_seq\
START WITH 280\
INCREMENT BY 10\
MAXVALUE 9990\
NOCACHE\
NOCYCLE;\
*/

CREATE TABLE jobs
( job_id VARCHAR(10)
, job_title VARCHAR(35) NOT NULL
, min_salary INT(6)
, max_salary INT(6)
, PRIMARY KEY (job_id)
) ;


CREATE TABLE employees
( employee_id INT(6)
, first_name varchar(20)
, last_name varchar(25) NOT NULL
, email varchar(25) NOT NULL
, phone_INT varchar(20)
, hire_date DATE NOT NULL
, job_id varchar(10) NOT NULL
, salary DECIMAL(8,2)
, commission_pct DECIMAL(2,2)
, manager_id INT(6)
, department_id INT(4)
, UNIQUE KEY (email)
, PRIMARY KEY (employee_id)
) auto_increment=207;

DELIMITER //
CREATE TRIGGER emp_sal_min BEFORE INSERT ON employees
FOR EACH ROW
BEGIN
IF (salary <= 0 ) THEN call unknown(); END IF; END// DELIMITER ; ALTER TABLE employees ADD ( FOREIGN KEY (department_id) REFERENCES departments(department_id)); ALTER TABLE employees ADD ( FOREIGN KEY (job_id) REFERENCES jobs (job_id)); ALTER TABLE employees ADD ( FOREIGN KEY (manager_id) REFERENCES employees(employee_id)) ; ALTER TABLE departments ADD ( FOREIGN KEY (manager_id) REFERENCES employees (employee_id)) ; /* CREATE SEQUENCE employees_seq\ START WITH 207\ INCREMENT BY 1\ NOCACHE\ NOCYCLE;\ */ CREATE TABLE job_history ( employee_id INT(6) NOT NULL , start_date DATE NOT NULL , end_date DATE NOT NULL , job_id VARCHAR(10) NOT NULL , department_id INT(4) , PRIMARY KEY (employee_id, start_date) ) ; DELIMITER // CREATE TRIGGER jobhist_dates BEFORE INSERT ON job_history FOR EACH ROW BEGIN IF (end_date > start_date ) THEN call unknown();
END IF;
END//

DELIMITER ;

ALTER TABLE job_history
ADD (FOREIGN KEY (job_id)
REFERENCES jobs(job_id));


ALTER TABLE job_history
ADD ( FOREIGN KEY (employee_id)
REFERENCES employees(employee_id));

ALTER TABLE job_history
ADD (FOREIGN KEY (department_id)
REFERENCES departments(department_id)) ;


CREATE OR REPLACE VIEW emp_details_view
(employee_id, job_id, manager_id,
department_id, location_id, country_id,
first_name, last_name, salary,
commission_pct, department_name, job_title,
city, state_province, country_name, region_name)
AS SELECT
e.employee_id, e.job_id, e.manager_id,
e.department_id, d.location_id, l.country_id,
e.first_name, e.last_name, e.salary,
e.commission_pct, d.department_name, j.job_title,
l.city, l.state_province, c.country_name, r.region_name
FROM
employees e, departments d, jobs j,locations l,
countries c, regions r
WHERE e.department_id = d.department_id
AND d.location_id = l.location_id
AND l.country_id = c.country_id
AND c.region_id = r.region_id
AND j.job_id = e.job_id ;

Tuesday, June 9, 2009

MySQL Workbench - Export Features

MySQL Workbench has some nice export features. They include:
  • Forward engineer SQL Create Script
  • Forward Engineer SQL ALTER Script
  • Synchronize With SQL Create Script
  • Export as PNG
  • Export as SVG
  • Export as Single Page PDF
  • Export as Single Page PostScript File
Export options (just choose File Menu then Export option):












MySQL Workbench 5.2 - Some nice features

MySQL Workbench 5.2 Alpha

My understanding is the MySQL Workbench 5.2 has the core 5.1 modeling features as well as the new Query Browser. The Query Browser has been easy to use but the feature functionality is still in the alpha phase. Adding the Query Browser in the 5.2 release adds a nice dimension of being able to look at the data model, physical table and data very easily. So far I have found the tool very intuitive and have not had to spend any time figuring things out. So far I have not done any benchmarking on the product.

New Query Browser




















Reverse Engineering Feature


I used the reverse engineering feature with the Sakila sample schema and it was point and click and the data model was built in just a second. Notice the ERD diagram in the top left, hierarchy of schemas and objects on right side and detailed table definition at bottom left.


MySQL Workbench - Visual Database Design Tool

What is MySQL Workbench?

MySQL Workbench is a visual database design tool available on Window, Linux and Mac OS. The MySQL Administrator and Query Browser tools have been deprecated. The database GUI interfaces will be rolled into the MySQL Workbench 5.2 release (currently alpha). MySQL Workbench is available in a Community (free) and Enterprise edition (paid license). Below are a few good links to look at:

Always looking at new tools

I've been looking for a better tool to work in a MySQL environment when I want to demo relationships between data models, physical tables, data and SQL code. It has to be a tool that the students can pick up easily without spending a lot of time understanding how it works. It's always an investment to learn another tool. So I've downloaded the MySQL Workbench 5.2 alpha release and really like what I've seen so far. I like it so much I'm going to start giving demonstrations for my work using the Workbench 5.2 release. I understand it's alpha but for my needs it was really easy to pick up and start using it right away. Anyone using Workbench for a production environment should be using 5.0 (GA) or 5.1 (Beta but soon to go GA).

Why I was looking for a new visual database design tool

I actually did not have it on my top three list of tools to look at initially. I just had some spare time this afternoon and downloaded the latest release of MySQL Workbench. I picked the alpha release because I wanted to look at the latest features. I just wanted to play with it but it was so easy to use that I starting playing with it seriously. I was doing a complex example this last week of looking at a customer issue from a number of perspectives to show problems that come up in database environments.

Really easy to build a data model, physical schema and use query browser

Before I knew it, I had built a data model, the physical schema, populated the key tables with data and did some query benchmarks. I realized in this personal exercise I had built an easy to follow demo that I wanted to use to show students some specific examples of database servers and troubleshooting problems. I also seen that I could have students new to MySQL Workbench be able to easily follow me by using the tool as well. I then started reading the whitepapers and documentation to understand the tool better. Before I knew it I was liking the product more and more. I realized that in about ten minutes I can easily convert a lot of users using SQL GUI tools in a MySQL environment easily to the MySQL Workbench 5.2 alpha release. But for how I am using it, its okay that it is alpha. Obviously before using any alpha release make sure you do your due diligence and understand if it is okay to use an alpha product for your situation. I'm just doing demos so I want a tool that meets my needs for easy to use demonstrations and a tool that is easy for students to use as well in following my demos.

MySQL Workbench Features

MySQL Workbench is a visual database design tool. Some of the key features available in this release:
  • Manage database schema objects: tables, views, indexes, stored procedures, functions and triggers.
  • Easy to use query browser tool.
  • Choice of diagramming notation.
  • Model validation.
  • Schema validation.
  • Forward and reverse engineering.
  • Change management.
  • Reporting functionality.
I've liked what I've seen in MySQL Workbench so I will be demoing with it and using the functionality in my future classes.

Thursday, May 28, 2009

What is MySQL?

It's been an incredible journey coming from the Oracle world (which I love) and having the privilege of joining MySQL and watching MySQL be on the verge of going public and then being part of the Sun acquisition and now watching the Oracle acquisition move forward. To say its been one of the richest experiences in my professional career would be an understatement.

Which makes me think, what is MySQL? If you asked 20 different people you would get 20 different answers. Some of the most common I believe would be:
  • A small open source company that wanted to disrupt the IT industry and did!
  • A company with incredible leadership that had a vision that every single employee was committed and dedicated to that vision.
  • A free open source database that helped redefine the database industry.
  • A very successful database product that has incredible scale out capability that leverages open source and the LAMP stack.
  • A product that opened up more options for organizations during a time of an economic down turn.
Today and twenty years from now, when I think of MySQL I do not just think of a database software product that has disrupted the IT industry. I think of:
  • A culture based on sharing, teamwork, commitment to a vision and love of open source.
  • A community that is entirely team based with an incredible sense of loyalty to each other and the goals defined by the leadership.
  • A leadership team that believed in speaking to the employees and not at them.
  • An environment built on empowering individuals to make the sum greater than the parts.
  • The incredible dedication and selflessness of the employees.
  • An environment that cultivated incredible talent and was able to define a vision that people believe in.
The spirit, dedication, belief in the vision and camaraderie is as strong in the MySQL community as it was over a year and a half when I joined MySQL. The culture, Qua, karma, kismit or what ever you want to call it, is I believe growing stronger.

I'm a big reader of business books and I believe we will see some successful business books on the MySQL story. Yet if these books talk about the MySQL success story or the MySQL product they will be capturing a small part of what MySQL is. To me, the MySQL story has to include the vision, the incredible culture and the unique group of people that have come together to disrupt the IT industry.

I write this blog entry because I always find it interesting the people that like to tear down our heroes, spread Fear Uncertainty and Doubt (FUD) and scare people with worse case scenarios. I on the other hand believe we should believe its okay to have heroes, believe that good things can happen and never underestimate what individuals that believe in a vision can accomplish.

The Big Misperception by new MySQL Customers

When meeting with MySQL customers I find they usually have some very strong misperceptions of using the free version (Community Version) of MySQL and that they have no need to purchase the Enterprise Version of MySQL.

First of all, most new MySQL users do not install, configure, tune or manage the MySQL database server properly. MySQL is the easiest database server in the world to install, but if used in a production environment it needs to be set up properly. When an organization purchases an Oracle license for six or seven figures they understand they had better get the appropriate expertise to manage their Oracle database server. When downloading MySQL for free, they think anybody can manage the MySQL server because it is free and anybody can install it. There is a big difference between installing and playing with a MySQL database server and running it in a production environment.

MySQL is extremely popular with web applications. Often traditional OLTP and DW DBAs are not familiar with architectures surrounding the MySQL database server or how web applications need to be configured differently than a traditional OLTP database or data warehouse application.

When using the MySQL server in a production environment it needs to be configured properly for it to run right. Problems new customers can have when using MySQL:
  • Database server running slow, hanging and possibly crashing because it was not configured properly for the production server environment it will be running in.
  • Batch operations taking 40 - 60 hours when they should be taking 20 - 60 minutes because the database server is not tuned properly.
  • High cost of using MySQL because customers do not understand all the features of MySQL.
  • Not using MySQL for the right type of application or using the wrong architecture for their MySQL solution.
New customers talk about how they do not need to purchase the Enterprise version of MySQL and how support is not necessary. MySQL support is arguably the best support in the IT industry and MySQL support people are well recognized experts in MySQL. MySQL support people are not call takers, they are experienced experts in MySQL. So often one support can justify the cost of purchasing a MySQL Enterprise License.

When customers ask if they should be purchasing the MySQL Enterprise edition, I ask them these following questions.
  1. If your database server crashes and it costs your company money or impacts your business you should be looking at support. Production database servers should be running in a supported environment.
  2. Do you have the expertise to configure and tune your database server?
  3. Do you understand the ramifications of how you are configuring MySQL, the architecture it will run under and what features you are using? If not, you could be setting yourself up for problems down the road?
I guess the question is, can your business afford to run the free version of MySQL or should you be purchasing the supported Enterprise release of MySQL.? It's so funny, because no one would every run an important Oracle database server without some type of support. So I always wonder why a customer would run the community version of MySQL and not the supported version of MySQL for critical applications?

Thursday, May 21, 2009

Brian Miezejewski speaking at the Dallas MySQL Users Group

Brian Miezejewski at Dallas MySQL Users Group

Brian Miezejewski will be speaking at the Dallas MySQL Users Group meeting June 1st in Dallas. Brian is a top respected MySQL consultant. If Brian is talking about performance tuning MySQL, then any MySQL DBAs in the area should be attending to listen to what he has to say.

I'm planning on attending the Dallas MySQL group meeting and listening to Brian.




Dallas MySQL Users Group Meeting June 1st, 2009

Meeting: June 1st
7:00 PM
Suite 700 16000
Dallas Tollway

Setting the MySQL Temporary Directory

One of the common mistakes made by new MySQL DBAs is forgetting to consider the MySQL temporary directory. If an internal heap table exceeds the size of the MySQL parameter TMP_TABLE_SIZE, MySQL will create a temporary MyISAM file on the disk to buffer the data.

MySQL will use the TMPDIR environmental variable to set the default location for storing temporary files. If this environmental variable is not set, the OS default will be used (i.e. /tmp, /usr/tmp or /var/tmp, etc.).

The mysqld --tmpdir can be set to multiple paths used in a rotating method. Paths are separated by colon characters (Unix) or semi-colons (Windows).

Remember that temporary files are created as hidden files. This makes sure that temporary files are deleted if mysqld is terminated.

Make sure to NOT set the TMPDIR path to the MySQL data directory filesystem.

Removing a MySQL Service FROM Windows

After shutting down and removing a MySQL database server from a Windows platform it is important to remember to remove the Windows service associated with it. There are two different ways to remove a MySQL server.
  • MySQL way: c:> mysqld --remove servicename
  • Windows way: c:> sc delete servicename

Saturday, May 16, 2009

Oracle SQL Developer with MySQL: Looks great so far.
















So far I am pretty excited about using Oracle SQL Developer with MySQL. I spent about ten minutes installing Oracle SQL Developer, the MySQL J Connector and setting up SQL Developer to connect to MySQL.

I've only been using it a few minutes and so far its been great to work with. Oracle SQL Developer has matured into a nice interface so its been really fun to work with. In terms of ease of use and being a nice interface to work with its been awesome! Now I need to start testing all the features and doing some migrations between Oracle and MySQL.

Installing and Configuring Oracle SQL Developer with MySQL

Installing Oracle SQL Developer was way too easy!

Installing Oracle SQL Developer is pretty straight forward to install. You will need to install the Oracle SQL Developer software and the MySQL J Connector (JDBC driver). It then requires about two minutes to point and click through the setup of the MySQL connection using the JDBC driver in SQL Developer. Oracle SQL Developer has the following downloads:
  • Oracle SQL Developer for Windows.
  • Oracle SQL Developer for Mac OS X platforms.
  • Oracle SQL Developer RPM for Linux.
  • Oracle SQL Developer for other platforms.

Install MySQL J Connector:
You will first need to download the MySQL J Connector (JDBC Driver) at the MySQL download site..

Install Oracle SQL Developer
Then download Oracle SQL Developer software from Oracle OTN. . You need to go to the Oracle Technology Network (OTN) and set up a user account (it is free). I installed it on Mac OS. I just unzipped the zip file in the Applications directory and started Oracle SQL Developer. Here is the main window in SQL Developer.

















After starting Oracle SQL Developer, go to the Tools Menu and choose Preferences. Then open up the Database Hierarchy and choose Third Party JDBC Drivers.
















Then click on the Add Entry button to add the JAR file containing the JDBC Connector. I put the J Connector file in /Users/George/Library/Application Support/SQL Developer directory. In the File Browswer I then choose: Library | Application Support | SQL Developer, then selected the mysql-connector-java-5.1.7-bin.jar file. You will then see the jar file added.



















Then right click on the Connections Icon in the main window. It will then bring up a tab window showing Oracle and MySQL. Choose the MySQL tab. Enter a connection name, username and password. Then test the connection. You will then see your new MySQL connection below the Connections Icon in the main window. Double click on your MySQL connection and start writing queries. Then hit the green arrow button to execute the SQL statement.

















Oracle has some great tutorials on how to get started with SQL Developer.


Have fun!

Wednesday, May 13, 2009

The Importance of Peripheral DBA Skills

At http://trubix.blogspot.com I've written a blog on the importance of peripheral DBA skills. In the MySQL world, the following peripheral skills are very helpful for managing MySQL environments:
  • LAMP skills - Linux, Apache, PHP/Perl/Python.
  • Application Servers - knowledge of Apache, JBoss, LightHTTP, connection pooling, etc.
  • Monitoring skills - Enterprise Monitor, Nagios, Cacti, etc.
  • Ability to support developers - PHP, Java, .NET.
  • Networking skills.

Saturday, May 9, 2009

New Things to Look at: OEM Grid Control with MySQL Plug-in

My customers tell me Oracle is winning the war with the Oracle Enterprise Manager (OEM) as the tool of choice for managing the Oracle Infrastructure across multiple tiers. With Oracle 11g, Oracle will be adding more RAC integration/functionality into the middle-tier with OEM. Part of the power of OEM is that it manages Oracle Database Servers, Application Servers, Services, Applications and the environment surrounding the Oracle Infrastructure.

OEM provides a number of plug-ins so I need to look into how having OEM monitor and manage MySQL with the OEM MySQL Plug-in. OEM also provides plug-ins for monitoring other database servers as well as other application servers. The capability to manage multiple tiers of heterogenous software is some very serious functionality. So seeing how well it manages MySQL is going to be fun.

New Things to Look at: InnoDB Plug-in

Just about every week I am asked by customers what storage engine they should be using. As well as, what storage engine should we be paying attention to in the future (i.e. Falcon, Maria, InnoDB, etc.).

It's finally time to start looking very seriously at the InnoDB plug-in. It's in the final beta releases while Maria and Falcon are still in alpha stage. The really nice thing about the InnoDB storage engine is it works very similar to how the Oracle works with a database buffer cache, redo log cache and undo data. It's always very easy for me to explain how InnoDB works to Oracle DBAs. There is also an InnoDB Hot Backup utility that I always hear positive things from customers using it.

I will be following with my thoughts and impressions of the Oracle InnoDB Plug-in.

Friday, May 8, 2009

New Things to Look: Oracle SQL Developer with MySQL

GUI Interfaces to use with MySQL

When meeting with MySQL customers I always get asked about GUI interfaces.
  1. I am always asked, what GUI to you recommend with MySQL? Two of the most popular are SQLYog and Navicat for MySQL customers.
Looking at Oracle SQL Developer

I think it is time to see how well SQL Developer works with MySQL. Oracle's SQL Developer has matured into a very nice tool and its free. I also want to see how well it migrates objects between MySQL and Oracle databases. SQL Developer has a lot of very cool functionality and all the interfaces are well designed. SQL Developer is also a Java application so you have the same look and feel on Linux, Mac OS and Windows.


Thursday, May 7, 2009

Collaborate 09 - Wrapup

The Oracle users Collaborate 09 was an absolutely fantastic technical conference. The presentations were exceptional in their quality. This conference was invaluable for networking and the discussions with Oracle user industry leaders from around the world. The discussions, insights and perceptions that were discussed every night were in my opinion even more valuable that the technical presentations. The presentations, podcasts and recordings will be available for two weeks following the conference.

It's now on to Java One and Oracle Open World.


Tuesday, May 5, 2009

Collaborate 09 - George Trujillo

Most of my blog entries this week are going to be at http://web.mac.com/george.trujillo since I'm at the Collaborate 09 Users conference.

Friday, May 1, 2009

Oracle Collaborate 09 - Oracle Users Conference

The Oracle User Community's big yearly event, Collaborate 09 in Orlando begins this week. This is going to be a hot conference bringing together the Oracle user communities top technology experts and industry leaders together. During this time of dynamic change it is an invaluable experience to bring get Oracle's top industry leaders for a week of meetings, presentations, discussions and networking events. Include top members from the Oracle product groups and Oracle's largest third party vendors and its a formula for invaluable networking.

My conference starts out this weekend with:

Saturday

  • IOUG Board of Directors meeting
  • Conference committee meeting
  • IOUG, Quest and OAUG board of directors reception
  • Mindy's big birthday bash
Sunday
  • IOUG University events - I'm planning on attending Michael Ault's Tuning university.
  • SELECT Magazine Expert Panel
  • IOUG Volunteer Reception
  • IOUG SIG Reception









Second Life Presentation on Sun Solaris Campus

This virtual world presentation will be delivered at the Sun Microsystems Islands on the Solaris Campus. Up above, I'm chilling at the Solaris Campus on one of the Sun virtual islands.

Second Life Presentation on Sun Solaris Campus

The MySQL open source database continues to increase in popularity with usage estimated at 12 million database servers worldwide. This presentation will help Unix/Linux and storage management administrators understand the reasons for the growing popularity of MySQL. Topics will include:

  • Positioning MySQL in the database market.
  • How is MySQL different than
  • Understanding the MySQL architecture.
  • Strengths and weaknesses of MySQL.
  • Key features of MySQL.
  • Understanding the storage engine story in MySQL.
  • MySQL strategic directions.

The focus of this presentation is on helping attendees understand the philosophy, features, benefits and popularity of MySQL.

Demystifying MySQL for Solaris Administrators: Sun Solaris Campus Event










Second Life Presentation on Sun Solaris Campus May 5, 2009 9:00 am PDT

This virtual world presentation will be delivered at the Sun Microsystems Islands on the Solaris Campus. Up above, I'm visiting the Solaris Certification Center.

Demystifying MySQL for Solaris Administrators

George Trujillo (Ty Valdez) and Steve Jones will be presenting a Demystifying MySQL for Solaris Administrators the week of May 4, 2009 in Second Life. This presentation is a very detailed technical presentation designed for experienced Unix/Linux administrators to understand installation, configuration and management of MySQL on Solaris platforms. MySQL best practices along with Solaris features that can leverage database management such as ZFS will be included. Topics will include:

  • Strategies for laying out database servers on Solaris (Unix/Linux) platforms.
  • Understanding the MySQL architecture.
  • Installing MySQL using MOCA (GOCA) for best practices.
  • Starting and stopping the MySQL database server.
  • Top ten things to configure after creating a MySQL database server.
  • Using MySQL with ZFS.
  • Managing MySQL on a Solaris platform.



Monday, April 27, 2009

Installing MySQL 5.1 on Solaris using MOCA

Introduction

The following instructions will lay out an installation of MySQL on Solaris using the MySQL Optimal Configuration Architecture (MOCA) for someone knowledgeable in MySQL/Solaris administration. MOCA is a set of best practices I put together to lay out a guidelines for installing and configuring a MySQL database server. MOCA is designed for someone with experience with MySQL, it is not for someone brand new to MySQL.

If you are new to MySQL or to Solaris, I recommend using the default package install for MySQL. The MySQL default install is recommended for someone new to MySQL or the operating system platform. If the default package install makes more sense for you, then you can stop reading. This install is for MySQL 5.1.33 but it would be the same steps for any 5.1.xx installation.

Why Perform a Manual Install

The default install with MySQL is great for users new to MySQL. It is simple, requires a few point and clicks and you are up and running. The problem with a default install is that it is designed to be a very simple install and take minimum resources. The default install also puts MySQL files in different locations on the filesystem dependent on the OS release and platform. The default install is not how an experienced DBA would want to set up a production database environment. It is much better to be able to control the layout and configuration of the database software for production database environments and for platforms where multiple MySQL servers may be installed in the future.

    This install assumes you have a fundamental understanding of Solaris and have an understanding of MySQL database administration fundamentals. Oracle DBAs will find this installation very similar to the concepts of the Optimal Flexible Architecture (OFA).
    For experienced MySQL DBAs a manual install is much better. For this purpose I created a best practices configuration and white paper called MOCA (MySQL Optimal Configuration Architecture). This is based on DBA best practices and should be very similar to Oracle, DB2 and SQL Server production DBAs. There are certain fundamental truths about how database servers should be installed, configured and managed. My MOCA whitepaper addresses these fundamental truths. This manual install will follow MOCA standards and conventions.

Why MOCA?

Visit mysql-dba-journey.blogspot.com to get the details of the reasons behind MOCA and why it is based on best practices. There is also an example of installing MySQL on Mac OS that is very similar to a Linux install. In summary, it focuses on:
  1. Separating database software from other software.
  2. Separating data and index files, log files for recovery, administration and backup files.
  3. Developing standard naming conventions.
  4. Defines a flexible configuration that can support multiple database servers on same platform.
  5. A consistent configuration for multiple servers and versions of MySQL database software.

Installation Summary

This installation looks more complex than it is. I use this configuration for all beginning MySQL DBA classes.

  1. Remove old versions of MySQL if they exist. Setup up operating system user mysql and mysql user environment.
  2. Set up directories and directory permissions for all MySQL data files.
  3. Setup MySQL software and install MySQL software as mysql operating system user (not as root). Configure the my.cnf configuration file.
  4. Create the mysql database (mysql_install_db) and setup the security environment (mysql_secure_installation). Start the mysql database server.
  5. Test the shutdown and startup of the database server.

Installation Environment

The environment for this installation is below: Mac OS 10(Leopard) running VM Fusion with Solaris 10 - Downloaded DVD iso image from www.sun.com website. I installed the Solaris 10 05/08 x86/x64 image for this demo (sol-10-u5-ga-x86-dvd.iso). I also used MySQL 5.1 - Downloaded from dev.mysql.com.

    Before installing MySQL on my platform, make sure there are no previous versions of MySQL preinstalled. Unless you want the older version of MySQL, your life will be much easier if you remove any previous releases.
    Read through this installation a few times before starting.

Look for existing MySQL software

This install uses 5.1.33, these installation procedures can be used for any 5.1.x installation. Dependent on the version of Solaris, different packages may need to be installed or removed (old MySQL installations).

Check to see if you see MySQL on your current system.
# grep mysql /etc/passwd #
# find /usr/local -name '*mysql*' - print # look here for MacOS, Unix/Linux
# find /var -name '*mysql*' - print # good place to start with Solaris
# find / -name "*mysql*' - print # look everywhere for MySQL installations

VM Fusion Choices for Installing Solaris 10

My choices for installing Solaris 10 in a VM Fusion environment. During the installation you will be asked to hit F2 to continue. On a MAC that will be EscapeKey-2 or FN-F2. Solaris Interactive US-English Networked - DHCP IPv6 - No You may need to specify the amount of disk space to use. I allocated 10228 MB.

    You should now be able to log in as root. With Solaris choose the Java Desktop Environment or the Common Desktop Environment (CDE), this is a personal preference.

Removing older versions of MySQL

Check for MySQL packages installed and remove them.
# pkginfo | grep mysql
The following packages SUNWmysqlr, SUNWmysqlt, SUNWmysqlu were found and removed.
# pkgrm SUNWmysqlr
# pkgrm SUNWmysqlt
# pkgrm SUNWmysqlu

Remove old MySQL files from common directories.
# sudo rm /usr/local/mysql
# sudo rm -rf /Library/StartupItems/MySQLCOM/

Set up the mysql user. Start by checking to see if there is an existing MySQL user.
# grep mysql /etc/passwd

Setup new mysql user if one does not exist. If a mysql user does exist, set up a password, default shell, default directory, etc.
No mysql user was found so I added one. Add the mysql group, mysql user, password and home directory.
# groupadd -g 300 mysql
# useradd -u 300 -g 300 -d /export/home/mysql -s /usr/bin/bash -c "MySQL DBA" mysql
# passwd mysql
# mkdir /export/home/mysql
# chown -R mysql:mysql /export/home/mysql

Login and verify the mysql user setup

# exec login mysql (or su - mysql)

Then define a default profile file using your favorite text editor.

--- .bash_profile file ------
PS1='$PWD: '
MYSQL_BASE=/opt/mysql
MYSQL_HOME=/opt/mysql/5.1.33
export MYSQL_BASE MYSQL_HOME
PATH=$PATH:$MYSQL_HOME/bin
--- end of .bash_profile file -------

Set your environment by sourcing your profile file.

$ cd $MYSQL_HOME
$ . ./.bash_profile

Downloading MySQL

Go to http://dev.mysql.com and go to downloads. Find the distributions and choose the install release you want. I chose 5.1.33. I prefer a manual install so I choose the Solaris Tar Packages the Solaris 10 64-bit install. Select a mirror. On the Select a Mirror page, I choose "No thanks, just take me to the downloads"!

MySQL Directory Organization

Organize how MySQL files and software will be located:
/opt/mysql/5.1.33 - Symbolic link to software directory location
/db01/mysql/mysql01/data - data directory
/db02/mysql/mysql01/binlogs - location of binary log files
/db03/mysql/mysql01/admin - main administration directory
/db04/mysql/mysql01/backups - location of backup files

I created the following directories to download the MySQL software in /opt/mysql/5.1.33.
# mkdir -p /opt/mysql/5.1.33
# export MYSQL_NAME=mysql01

Setup data directory structure
# mkdir -p /db01/mysql/$MYSQL_NAME/data

Setup mysql administration directory structure
# mkdir -p /db03/mysql/$MYSQL_NAME
# mkdir /db03/mysql/$MYSQL_NAME/logs
# mkdir /db03/mysql/$MYSQL_NAME/errors
# mkdir /db03/mysql/$MYSQL_NAME/sql

# mkdir /db03/mysql/$MYSQL_NAME/startup

# mkdir /db03/mysql/$MYSQL_NAME/run


Setup binary log structure
# mkdir -p /db02/mysql/$MYSQL_NAME/binlogs

Setup backup directory structure for backups and exports.
# mkdir -p /db04/mysql/$MYSQL_NAME
# mkdir /db04/mysql/$MYSQL_NAME/backups
# mkdir /db04/mysql/$MYSQL_NAME/exports

Set permissions and ownership for MySQL file directories.
# chmod -R 750 /db*/mysql/* /opt/mysql/*
# chown -R mysql:mysql /db*/mysql/* /opt/mysql/*

Before going further

Double (triple) check all directory paths and permissions. 99.99% of issues with manual installs are typos in the directory paths, typos in the file names or permission issues with directories. Double check all paths. When you try to bring up database server, if it defaults to the default areas its because it can;t find a directory or doesn't have permission for directories specified so it will then try the default locations.

Setup the MySQL software (as the MySQL OS user, not the root OS user)

All following commands are run as the mysql OS user. In the /opt/mysql directory unzip and untar the MySQL software as the mysql OS user.
$ cd /opt/mysql
$ gunzip mysql-5.1.33-solaris10-64bit.tar.gz
$ tar xvf mysql-5.1.33-solaris10-64bit.tar
$ ln -s mysql-5.1.33-solaris10-64bit 5.1.33

Be careful with the my.cnf configuration file.

At the operating system prompt you can type the following command. If you scroll down you will find the default search path. All commands like mysql_secure_installation, mysql, mysql_install_db, etc. all look in a search path for the configuration file. Make sure any program is finding the right configuration file in the search path. Here are commands that will show you the search path:

$ mysql --help | more

$ mysqld --help --verbose | more

Use one of the sample configuration files provided with the distribution to get started.

$ cp $MYSQL_HOME/support-files/my-small-cnf /dbadmin/mysql/mysql/startup/my.cnf

Add the following entries to the my.cnf file to the [mysqld] group. This separates all your dynamic administration files, data files, and binary log files to different locations. A separate port is defined away from the default.

[mysqld]

datadir=/db01/mysql/mysql01/data
basedir=/opt/mysql/5.1.33
log-error=/db03/mysql/mysql01/errors/mysql5.1.33.err
pid-file=/db03/mysql/mysql01/localhost.pid

log-bin=/db02/mysql/mysql01/mysql-bin
#port = 3426
#socket=/dbadmin/mysql/mysql01/run/mysql.sock

Add the following entries to the my.cnf file to the [client] group.
[client]
#port = 3426
#socket = /dbadmin/mysql/mysql01/run/mysql.sock

Build a symbolic link from the default location to the real startup file. The step of adding a symbolic to the actual startup file is something I avoid but some environments like this addition.

$ cd $MYSQL_HOME
$ ln -s /dbadmin/mysql/mysql01/startup/my.cnf my.cnf

Create the mysql database files for the MySQL instance. This will create the default database schemas and database files.

$ cd $MYSQL_HOME
$ scripts/mysql_install_db --datadir=/db01/mysql/mysql01/data --basedir=$MYSQL_HOME

Verify data files and directories have been created in the datadir directory.

$ cd /db01/mysql/mysql01/data
$ ls

mysql test

Start the MySQL database server pointing to the defined locations.
$ cd /opt/mysql/5.1.33
$ bin/mysqld_safe --defaults-file=/dbadmin/mysql/mysql01/startup/my.cnf &

If there are socket errors:

i.e. MySQL client cannot star twith the error "cannot connect to the MySQL server through socket

Solution:

MySQL needs to write to a socket. If you don't specify one, a default one is chosen which may not have the appropriate permissions. You can specify the socket file in the command line as below. Make sure the permissions are set properly (owned by mysql).

$ mysql -uroot -p -socket=/dbadmin/mysql/mysql01/run/mysql.sock

Verify the mysqld background process is running as well as the mysqld_safe monitoring process. The mysqld background process should be up and running.

$ ps -ef |grep mysql

Clean up the database server by adding passwords and getting rid of anonymous users. If there are problems with the mysql_secure_installation script, then set the password manually and get rid of the anonymous accounts and any accounts with no passwords.

$ cd $MYSQL_HOME
$ bin/mysql_secure_installation

Shutdown the MySQL server to verify you can shutdown and startup the MySQL instance. I like switching to the my.cnf in the startup directory versus in the default software directory.

$ mysqladmin --defaults-file=/dbadmin/mysql/mysql01/startup/my.cnf shutdown
$ cd $MYSQL_HOME

$ bin/mysqld_safe --defaults-file= /dbadmin/mysql/mysql01/startup/my.cnf

You're up and running have fun. Once you are confortable with this configuration layout, you can create a Unix shell script that will automate almost the entire process. With a shell script automation the install takes about ten minutes.

George Trujillo at Collaborate 09: Demystifying MySQL for Oracle DBAs and Developers

George Trujillo at Collaborate 09 Week of May 3, 2009

I will be presenting two presentations at the IOUG Collaborate 09 Conference in Orlando FL, the week of May 3rd 2009. Here is one I have been asked to present. This presentation is a very detailed technical presentation designed for experienced Oracle professionals to get an understanding of the MySQL database server under the hood.

Demystifying MySQL for Oracle DBAs and Developers Tues @ 9:45am - 10:45am, 224A

George Trujillo continues his popular demystifying series that started with Demystifying Oracle Fusion Middleware, to now include a MySQL demystifying series. This presentation will use George's MOCA best practices blueprint for installing, configuring and tuning a MySQL database server. This fast paced detailed technical presentation will cover: MySQL Architecture, Installation, Configuration, Key Features, Security, MySQL Tools, Storage Engines, transactional capabilities and strategic directions. This is a can't miss presentation for any Oracle DBA wanting to understand MySQL.

George Trujillo at Collaborate 09: What every Oracle Professional needs to know about MySQL

George Trujillo at Collaborate 09 Week of May 3, 2009

I will be presenting two presentations at the IOUG Collaborate 09 Conference in Orlando FL, the week of May 3rd 2009. Here is one I have been asked to present. This presentation is a non-technical presentation talking about MySQL positioning, strengths and strategic directions.


What every Oracle Professional needs to know about MySQL Mon @ 3:45pm - 4:45pm, 240CD

The MySQL open source database continues to increase in popularity with usage estimated at 11 million database servers worldwide. This presentation will help Oracle professionals who are considering the addition of MySQL or just getting started with MySQL: how the MySQL database can be utilized, MySQL strategy and architecture, and key functionalities. Similarities and differences between database vendors and best practices will be included. The focus of this presentation is on helping attendees understand the philosophy, features, benefits and popularity of MySQL.

Sunday, April 26, 2009

Trends in Database Design and Optimized Application Development

I'm constantly going out to customers and looking at their database environments and the challenges they are facing. Of course, sometimes my perspective can be rather unique because customers never invite me in to show me how fast their databases are or how great things are. Usually by the time I get called in, the environment has gotten very dynamic.

What never ceases to amaze me is how on average database design seems to keep getting worse and applications are more poorly written than ever before. There are a number of industry and organization reasons for this but these reasons constantly stand out at customer sites:
  1. Developers Designing Databases: Developers are usually the ones meeting with the customers and doing a lot of the initial design on smaller projects. The problem with this is there is a big difference between understanding what a primary and foreign key are and designing a database. Or there is a data modeler who can design beautiful databases in theory but has no idea how that database design is going to work in a production environment.
  2. Most Developers Do Not Know How to Write Optimized Queries: Back in the old days there were database developers that had a strong understanding of SQL development. However the current generation of Java, PHP, .NET and Ruby developers are great with their languages and writing APIs but the major percentage of them have no idea how to write optimized queries for their applications.
  3. Take Out Databases: In this generation of fast food take out and web applications, its all about getting something out the door quick. Which usually means minimal time for proper database design and testing. This is definitely the Dilbert world of databases.
I find these problems to be much more severe in MySQL environments versus Oracle environments. If an organization pays a million dollars for an Oracle license they are going to make sure they get some people that know that they are doing to design and manage a database project. When someone downloads a MySQL database for free for a small web application they think they can use anybody and the project will turn out okay.

When teaching performance tuning classes in Oracle and MySQL I always ask the students what percentage of companies do they think do a good job of designing their databases and their index optimization paths. I always get single digits for the response to these questions. Oh well, I guess this is what makes the database world so fun. :)

Top things that I have found that greatly improves database application success includes:
  • Getting the right expertise to design your databases.
  • Training developers on how to write good queries for their database applications.
  • Design patterns.
  • Taking the time to do it write.
  • Use case scenarios.

Tracking the Storage Engine Race

There are a number of new and enhanced storage engines that promise increased scalability, performance and important new online features. All this competition with storage engines is going to create a win for the MySQL community. This competition is what open source is all about. If you do not continue to innovate and improve you cannot expect to be a leader in the open source world. The nice thing about MySQL 5.1 is the plug-in capability that allows the addition and removal of storage engines.

The top ones I have been tracking include:
  • Falcon (still in alpha)
  • Maria (still in alpha)
  • InnoDB (final release candidate stages)
  • XtraDB (unable to verify release, since only a few months old may be alpha or beta)
  • PBXT (beta I believe )
Instead of just comparing benchmarks and features, what I am focusing on is the adoption by my customers. All these storage engines are promising optimized architectures,
increased performance, scalability and improved metadata for diagnostics and tuning. For most of these it is still too early to tell. However, I will be talking to customers every week to gauge their thoughts and adoption rates on these storage engines.

No matter which one I pick, I'll always upset someone. But its no fun staying on the fence, so I would definitely give the early lead to the InnoDB plug-in. The reason is it is on the final release candidate stages and is the first next generation storage engine used with the MySQL 5.4 scalability and performance release. It will be fun to watch the growing popularity of these storage engines. I would love to hear some detailed feedback on your thoughts of these new storage engines. Be aware that most of these storage engines are still in alpha or beta, so I recommend being very conservative before using them in a production environment.

Wednesday, April 22, 2009

Working with MySQL Hints

A number of developers look at SQL tuning hints as a way to performance tune queries. Where the best way to tune SQL code is not to use hints. Hints add high maintenance code and try to influence the Cost Based Optimizer which can have negative impacts in the future. What if the data distributions change or the optimizer adds new features in the future? The best way to tune SQL code is to look at how the database server environment can be optimized so the optimizer can make the best decisions as possible. However, if you cannot get the optimizer to make the right decision a SQL hint may be required but it should usually be the choice of last resort.

Examples:
SELECT * FROM mytab IGNORE INDEX (col1_idx)
WHERE col1='A' AND col2='B';

SELECT * FROM mytab USE INDEX (col1_idx,col2_idx)
WHERE col1='A' AND col2='B' AND col3=3;

Here are some of MySQL SQL Tuning Hints:
  • STRAIGHT_JOIN
  • IGNORE INDEX [FOR JOIN]
  • FORCE INDEX [FOR JOIN]
  • USE INDEX [FOR JOIN]
  • SQL_NO_CACHE
  • SQL_CACHE
  • The HIGH_PRIORITY
  • The LOW_PRIORITY
  • INSERT DELAYED
  • INSERT LOW_PRIORITY
  • INSERT DELAYED (MyISAM, MEMORY, and ARCHIVE tables)
  • SQL_BUFFER_RESULT
  • SQL_BIG_RESULT
  • The SQL_BIG_RESULT
  • SQL_SMALL_RESULT

Monday, April 20, 2009

Oracle to buy Sun!





Today's announcement that Oracle has agreed to buy Sun definitely starts the MySQL conference which begins this morning with a bang. I expect Oracle InnoDB presentations to be standing room only!

  • Oracle announcement: http://www.oracle.com/sun/index.html
  • Sun announcement: http://www.sun.com/third-party/global/oracle/index.jsp

The MySQL conference just got very interesting for the attendees! Popular user conferences bring together industry leaders from around the world, so the activity at the conference has definitely be raised to a very high level.

For me personally this potential acquisition is great news. It brings together the two company's technologies where I have spent most of my career working with.

Friday, April 17, 2009

MySQL Users Conference 2009 - Santa Clara, CA






The MySQL Users Conference is going to be an excellent conference with a lot of change going on in the MySQL world. Some of the key areas to be looking at include:
  • MySQL strategic directions.
  • Increasing vertical scalability of MySQL with upcoming MySQL software releases, new and enhanced storage engines and patches.
  • Enhanced diagnostics with DTrace.
  • New BI software products and tools.
  • Performance tuning, performance tuning, performance tuning.
  • Evolution of next generation of storage engines. Lots of cool stuff here.
  • Advanced DBA tricks and techniques.

Thursday, April 9, 2009

InnoDB Plugin Has Some Very Cool features

Reducing Administration Costs and Down Time
DBAs are always going to be challenged with growth, scalability and performance as long as databases get bigger and more users access them. Reducing administration and down time are always key issues in our 24 x 7 environments. The new InnoDB Plugin has some very important features that DBAs are going to want to take a serious look at. So I thought I would point out some of the highlights.

The InnoDB Plugin
InnoDB is the primary transaction storage engine through the 5.x releases of MySQL. The InnoDB Plugin allows MySQL DBAs to replace (SKIP_INNODB) the default InnoDB storage engine with an enhanced version of InnoDB (plugin) that offers some important enhanced features and performance. This InnoDB plugin is in the final beta (release candidate) releases.

Key Features
The InnoDB Plugin offers some of these important key features:
  • Fast Index Creation - The ability to add and drop secondary indexes without copying table data. Index maintenance is always an issue. The performance benchmarks on this feature are significant.
  • Out of Line Data - BLOB, Text and VARCHAR columns can be stored out of line. By default only 768 bytes are stored inline for large columns. However even this 768 bytes is not good for your table storage. If the INNODB_FILE_FORMAT is set to Barracuda (default is Antelope) and the ROW_FORMAT=DYNAMIC or ROW_FORMAT=COMPRESSED is set, the only a 20 byte pointer is stored in line and the rest of the data is stored out of line. New InnoDB file formats are likely to use additional animal names in alphabetical order.
  • Enhanced Metadata - New metadata in INFORMATION_SCHEMA on locking and compression. Enhanced metadata definitely helps with dealing with locking issues and getting better information on compression. The new INFORMATION_SCHEMA metadata tables include: INNODB_CMP, INNODB_CMP_RESET, INNODB_CMPMEM_RESET, INNODB_CMPMEM, INNODB_TRX, INNODB_LOCKS and INNODB_LOCK_WAITS. DBAs have been waiting a long time to get detailed locking and transaction information. These new metatables are awesome!
  • Data Compression - Data compression support for InnoDB. There is a new attribute called ROW_FORMAT that can be set to compressed. The KEY_BLOCK_SIZE is another new parameter that can compress in 1K, 2K, 4K, 8K or 16K bytes. The LZ77 compression algorithm compresses data and indexes. Compressed data is uncompressed in the InnoDB buffer cach for reading. Compression requires the INNODB_FILE_TABLE option to be set. Example:
CREATE TABLE mycooltable (col1 INT PRIMARY KEY) ENGINE=InnoDB
ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8;
  • Truncate Table - TRUNCATE TABLE command reclaims space in .ibd file.
  • Optimization Statistics - More control over statistics generation. New features allow better statistics generation to determine the key distributions on indexes. This is a very important feature for the cost based optimizer to make good decisions on indexed columns that have uneven distributions of key values.
  • Dynamic Parameters - Dynamic parameter eliminates need to bounce database server when modification is required. It is very important that DBAs minimize downtime so these making the following parameters dynamic is pretty important:
  • innodb_file_per_table
  • innodb_stats_on_metadata
  • innodb_lock_wait_timeout
  • innodb_adaptive_hash_index
InnoDB Plugin Installation
The installation of the InnoDB Plugin can pretty straight forward and only takes a few minutes once the InnoDB Plugin is downloaded into the proper location. Of course every version and platform has its own unique idiosyncrasies (or the devil is in the details).

Wednesday, March 25, 2009

MySQL Scalability and Performance Directions

MySQL is continuing to grow at a rapid pace in the market place. Continued high growth areas for MySQL continue to include the web application, gaming and embedded systems space. For small and medium sized OLTP environments MySQL continues to increase in popularity. MySQL can have incredible scalability as long as it scales horizontally. However with today's hardware adding more CPU and memory capability, it is important that MySQL be able to grow to much larger sizes through vertical scalability.

There are a lot of upcoming changes in the MySQL world that are going to add significant performance, scalability and feature/functionality. The key areas I will be talking about include:
  • MySQL Version 6 - (currently in alpha) will add significant performance, availability, scalability and online features.
  • Falcon - (currently in alpha) New storage engine with MySQL 6 that will add increased scalability, availability and online features to replace the InnoDB storage engine.
  • Maria - (currently in alpha) New crash safe storage engine that will add increased scalability, availability and transaction safe features to replace the MYISAM storage engine.
  • InnoDB Plugin - (close to end of beta) Oracle offers an InnoDB plug-in that provides a number of important performance, scalability, online functionality and reduced maintenance features.
Each storage engine: Falcon, Maria and the InnoDB Plugin provide a lot of important enhancements that customers are going to want. It will be important to watch the maturity of each of these storage engines as they move from alpha and release candidate versions to general availability (production).


Thursday, February 19, 2009

Performance Tuning Knowledge is Important for the New DBA

As MySQL continues to expand in the market place, I am seeing a common mistake new DBAs make. They are waiting too long to develop their performance tuning skills. Whenever I work with new DBAs I always tell them it is important to focus on three areas:
  1. Understand the architecture and how things work. It is not good to guess.
  2. Get very good at back up and recovery if you want to keep your job.
  3. Get very good at performance tuning if you want to enjoy your job.
It is important that in your future as a DBA you plan on taking a Performance Tuning class. The reasons are:
  • Most databases get bigger (add more data) and have more users over time. So performance tuning will always be a factor in managing your systems as your systems grow.
  • It's important to take a performance tuning class sooner than later. Reason is you could be making decisions that will have very negative performance ramifications in the future, if you do not understand performance factors. Future scalability is impacted by performance decisions made today.
  • Understanding performance tuning makes you a better DBA. Being good at performance tuning requires you understand how a database server works in more detail. This understanding will help with diagnostics and trouble shooting for everything you do as a DBA.
Being good at performance tuning helps you be a more proactive DBA versus being a reactive DBA. Reactive DBAs are always putting out fires and fixing things after the fact. If you want to enjoy your work as a DBA focus on being a proactive DBA.

Monday, February 9, 2009

The New Tipping Point: The Sun Open Source Software Stack

Sun's Software Stack - At the right place at the right time
The technical industry has always had tremendous ebbs and flows. Just ask powerful or up and coming companies like Digital Equipment Corporation, Silicon Graphics, Informix, Netscape, Atari, and even Sun. Yet sometimes, "everything is in the timing".

With the current economic environment, it is safe to say that all organizations are taking a very serious look at how to reduce costs. This perspective is likely to continue through at least 2009 and 2010. Sun's software stack is positioned in the right place at the right time. Sun's current software stack will significantly reduce operational costs and provide more flexibility over operational budgets. As costs continue to impact organizations, you will see more traditional organizations follow the direction of start up companies.

Ninety percent of start up companies in the U.S. are using the LAMP stack (Linux/Apache/MySQL/PHP,Perl). The LAMP stack is a proven software stack that offers tremendous cost savings over proprietary licenses. Sun's software stack is a faster, cost effective and enterprise version of the LAMP stack. So we can expect more and more organizations looking at and moving to the Sun software stack. I'm seeing this every week in my training classes. My students are selling me on all the benefits they are receiving from this stack. And a major percentage of my students are from proprietary vendor environments.

Major Areas of the Sun Software Stack
Major areas of the software stack that will dramatically reduce costs versus alternative options:
  • MySQL - The fastest growing database server in the industry. The adoption rate is increasing every month.
  • Glassfish - A powerful, fast enterprise application server that is a fraction of the cost of its competitors. A great solution for customers that are growing out of their Apache and JBoss servers.
  • Open Storage - With the ever increasing volumes of data that organizations need to work with, storage will continue having a significant impact on IT budgets.
  • Identity Management - Fast enterprise identity management. Sun has one of the best LDAP solutions out in the industry.
  • Solaris/OpenSolaris - Offer proven powerful software solutions that can significantly reduce OS costs.
  • Java - Key software for mobile applications and large enterprise business applications.
So as the trend continues, watch as the Sun software stack increases in popularity. There are reports that the government is evaluating increasing its use of open source to reduce license costs. It won't take much more for the Sun software stack to become a major tipping point in the software industry.

Recent MySQL Trends

In working with customers every week I am seeing some trends that I would like to share with everyone.
  1. There is a tremendous upswing in Oracle customers looking at MySQL. A year ago about half the customers were evaluating MySQL, now I am seeing about 90% saying they have been mandated to look at using MySQL for future DBA projects.
  2. Must stronger acceptance by the traditional Oracle DBA as to the features and benefits of MySQL.
  3. A large number of customers new to MySQL are not very knowledgeable about the environments surrounding MySQL database servers. Mention words like Memcache, InfoBright, KickFire and most customers are not sure what they are.
  4. Customers are still using a wide variety of tools to monitor and manage MySQL database servers. However, I do see an upswing on customers looking at the MySQL Enterprise Monitor.
  5. Too many customers underestimate the need to get their administration teams trained on performance tuning and advanced database features. MySQL is a database server that needs to be tuned and managed correctly just like any other database server. Too many customers think all they have to do is install it and they are done.
  6. There is a lot of interest in MySQL version 6 with the new optimizations, storage engines and availability features.
  7. Too many customers are not sure on whether to use Ruby, Groovy, Java, .NET, PHP, etc. I am constantly surprised on how customers are still trying to figure out the best tools to use for development for specific projects. Maybe this will never change.
  8. More and more web developers do not understand how to write good database code. This leads to big performance issues down the road when database usage begins to go up.
  9. Web developers are almost never using database vendor specific features. They want databases that are fast, easily deployable and easy to work with.
  10. From my perspective in working with customers, MySQL is growing at the fastest rate I've seen in the last few years. Is their any area of technology that is growing faster than MySQL?

Friday, January 16, 2009

MySQL and Oracle: Top 10 things I love about them

Today's Competitive Global Economy Environment for DBAs
With today's competitive environments, the more marketable you are the better. Being flexible and being able to support different solutions is very important. Environments surrounding databases are getting more complex, so increasing your expertise in different areas is important. There are a lot of options with application servers, storage, data warehousing, high availability, etc. Another strong skill combination is Oracle and MySQL. Oracle is the largest database vendor and MySQL is the fastest growing so you are going to see both a lot in the future so why not be able to support both.

What I love about MySQL

I've spent the last two years working with open source and MySQL. It's been two of my coolest years in technology. My top ten things I love about MySQL:
  1. In working with MySQL customers their is a lot of innovation and think out of the box mentality.
  2. MySQL environments use a lot of different software and tools. Shell scripting, perl scripts, awk, sed all remind me of the Oracle 7 environments which were a lot of fun to work in. Being very strong with Linux/Unix and the tools makes you a more marketable MySQL DBA.
  3. MySQL surrounding software such as Apache, JBoss, Memcache, Cacti, Nagios, Mon, phpMyAdmin, PHP, etc. makes it a fun environment for the technical junkie.
  4. MySQL's ability to leverage the power of the Internet is incredible to be a part of.
  5. The open software environment that is all about contribuing to the greater good, sharing knowledge so everybody grows and mass collaboration is incredible powerful. Its fun to be part of the wave.
  6. The MySQL mentality has been one of the richest experiences of my life. Everybody being tied in virtually, knowing my peers in Germany, Sweden and all over the world as well as my peers in the US has make it fantastic to be part of a company that really understands how to function as a "team" in a global econony.
  7. The MySQL environment of we are a team, we all work together, share and work to accomplish the goals of the team and the company. MySQL is one of the few companies that can really get everyone working together and motivated towards a common goal.
  8. Learning the power of MySQL and why more new MySQL database servers are being created than all the proprietary vendors combined.
  9. MySQL's empowering of the individual employee to accomplish great things. Where the emphasis is that the individual employee really does matter.
  10. The incredible vision of the leaders of MySQL and the absolutely fantastic people within the MySQL teams.

What I love about Oracle

I've spent spent the last 20 years working with Oracle (RAC, Data Guard, Streams, backup/recovery, performance tuning, Oracle Fusion Middleware (SOA, BPEL, Web Services, XML, Oracle Business Rules, JDeveloper). My top ten things I love about Oracle products:
  1. The database server is an aircraft carrier solution with a task force. :) When you have complex problems to solve Oracle has powerful tools to solve them.
  2. Oracle RAC is a very robust, scalable clustered solution that is great to work with.
  3. Oracle Streams while complex allows you to solve tons of data sharing problems.
  4. Oracle Fusion Middleware offers a great middle-tier enterprise solution for large companies. Working with BPEL, SOA, Web Services, XML, etc. is like being a kid in a candy store.
  5. I've just flat out loved working with Oracle products and Oracle environments.
  6. The fantastic people I've met in Oracle user groups, the list is endless.
  7. The incredibly dedicated people I've met at Oracle that work in all the different product groups.
  8. Oracle's done a great job of integrating organizations they've purchased and achieving their visions.
  9. I've always had fun performance tuning Oracle multi-tiered architectures. Maybe this is an incurable disease (ha ha).
  10. I've always found the more I've learned about Oracle products the more money I've been able to make (always a good thing). My personal investment in Oracle technology has always paid off.

LAMP Skills for the MySQL DBA

MySQL runs on lots of Operating Systems
The MySQL database server runs with more different operating systems than any other database. The reason being is not only does it run on all the popular operating systems but you can download the source, make a few tweaks and run it on any operating system (if you're a bit twiddler). Yet the place where you're likely to see MySQL the most is with LAMP environments. Of course the L in Linux means Red Hat, SUSE, Centos, Fedora, Ubuntu, etc.

Operating Systems I use with MySQL
The operating systems I use the most with MySQL include Red Hat, Windows, Fedora 10, Mac OS, Solaris and OpenSolaris.
  • Red Hat and Solaris - use the most when working with customers.
  • Mac OS, Fedora and OpenSolaris - for trying different things and playing around with MySQL. Although I do see some customers using OpenSolaris and CentOS for their production environments.
  • Windows - use mainly for teaching beginning MySQL courses.
VMs I use with Mac OS
I like using Parallels and VM Fusion with my Mac OS environment. Both of these VMs allow me to install any operating system I want and play with MySQL, Oracle, Oracle Fusion Middleare and LAMP stack components.

Software skills to have around MySQL
For most MySQL DBAs in general, they need to have skill in popular software surrounding MySQL. These software skills include:
  • Linux administration - basic admin skills, cron, shell scripting and UNIX power tools (awk, sed, REs, PERL,...)
  • Application Servers - Get good with an application server. Apache is a great one to start out with. Then you can migrate to JBoss, Glassfish or the Oracle Application Server dependent on your goals. MySQL excels in web environments so it really helps for a MySQL DBA to understand the middle-tier in a lot of companies.
  • Popular software - Around MySQL you usually see a lot of oftware like memcache, Nagios, mon, cacti, SQLyou, Navicat, Toad, phpMyAdmin, etc. MySQL is like the open source Sun and a lot of open source revolves around MySQL.
  • Development - I'm still seeing lots of PHP, Java and Ruby around MySQL. Of course .NET is always a player.
  • Benchmarking software - mysqlslap, sql-bench, jmeter, MyBench, sysbench, Apache Bench and supersmack are all good skills to have.
LAMP Skills for the MySQL DBA
It's important for any MySQL DBA to increase their value to an organization and marketability by having strong LAMP or LAMP equivalent skills. As well as be able to work with the software that is used in lots of companies that surround MySQL.

Friday, December 19, 2008

Evolving trends and directions for DBAs and Developers: How do DBAs stay marketable?

I visit a large number of organizations every year to deliver training, short term consulting as well as business and technology seminars in the area of database servers, security, software development and middle-tier technology. So I thought I would share some of the trends I have seen in 2008 that I believe will continue in 2009.

I'm always being asked the following questions; "How do I stay marketable?" or "What trends are occurring in technology that impact DBAs and Developers?". The trends are pretty obvious, the question is what conclusions do we draw from them.

IT Continuing Trends for 2009

Some of the noticeable trends:
  • Common DBA skills (administration, backup/recovery, tuning) are becoming more of a commodity and easier to outsource every day. A recent quote from a high end recruiter "I don't have any use for technical DBAs. Now applications DBAs are worth their weight in gold".
  • Oracle Application DBAs (Financials, ERP, Hyperion, ...) are becoming more valuable and marketable every day. This is the skillset of the future, not the basic technical DBA skills.
  • Middle-tier applications increase in importance and visibility. The key in most organizations is their middle-tier business applications (vendor and custom). An Oracle database license can be a few million dollars. An Oracle or SAP Financials implementation can cost 100 million dollars. I wonder where an organization will place their emphasis?
  • Oracle Fusion Middleware skills (Oracle Application Server, web services, BPEL, SOA, XML, ...) are becoming more valuable than ever. As Oracle Fusion applications roll out in the future, these middle-tier skills will increase in marketability and demand.
  • Middle-tier architecture skills working with application servers, middle-tier caching, web services, J2EE, PHP, Ruby on Rails are increasing in demand.
  • Architectural skills and high availability expertise across all tiers are needed more than ever. If a system is slow, there has to be people that can do problem resolution and performance tuning across all tiers.
  • Cross platform expertise. DBAs that can support Oracle, MySQL and SQL Server environments are more valuable than a technical DBA that can only support one environment.
  • Virtualization will increase significantly. Oracle's VM and Sun's VM (Containers, Zones) will see an increase in database and application servers running production environments using VMs. Sun Containers are a very powerful way of setting up cloning and failover. So VMs not only provide very cost and environmental effective ways of implementing servers but offer significant advantages in administration and high availability. One of the biggest success stories with Sun Containers was an Oracle Hyperion implementation that the consultants told me that can't now imagine implementing this any other way.
  • Open source will continue to see significant growth in the next year. The reduced cost and flexibility of open source solutions will significantly help organizations be cost effective and competitive.
The best present you can give yourself for the next year is to learn new skills to increase your marketablity and value to organizations.

Wednesday, December 17, 2008

MySQL Profiling: SQL Tuning

MySQL provides a number of different tools for tuning SQL statements. Some of the key SQL tuning tools include:
  • EXPLAIN - Displays execution plans generated by the MySQL Cost Based Optimizer.
  • Status Variables - Contains statistics on SQL run time activity.
  • Profiling - Contains run time statistics on each phase in the execution of individual SQL statements.
Main Phases in Processing a SQL Statement

Profiling allows access to very detailed run time statistics on each phase of processing a SQL command. The main phases include:
  • Parsing the SQL statement.
  • Generating an execution plan.
  • Performing an execution and fetch.
  • Cleaning up resources.
Profiling Benefits

Profiling provides the following benefits:
  • Ability to understand time spent in each phase of processing a SQL command.
  • Displays if the OS performed a disk read because the requested block is not in memory.
  • The number of waits that occurred while the thread processed the SQL command.
  • Disk activity generated by the SQL statement.
  • Resources allocated for processing the SQL command.
  • Approximations of time required to process each step of the SQL statement.
  • Comparision of execution times relative to other SQL statements.
  • If you're into the source code, each C function and line number where it is executed for each step of processing the SQL statement.

Profiling Resources in MySQL

The profiling resources available beginning in MySQL 5.1.28 include:
  • SHOW PROFILE command.
  • SHOW PROFILES command.
  • Information_schema.profiling table.

Turning Profiling ON

Profiling is turned OFF (0) by default. Profiling is turned ON (1) at the session level. The PROFILING_HISTORY_SIZE parameter is used to determine how many statements are kept in the history. The default is 15 statements. The query id will change for each SQL statement executed in a session.

mysql> SET PROFILING=1;


SHOW PROFILES

The SHOW PROFILES command will display the query id, the duration and the SQL command executed.

mysql> SHOW PROFILES;
+----------+------------+------------------------------------------------------
| Query_ID | Duration | Query
+----------+------------+--------------------------------------------------------
| 1 | 0.00073900 | SELECT Co.Name, Ci.Name, Ci.Population FROM CountryList Co, CityList Ci WHERE Co.Code = Ci.CountryCode AND Ci.Population > 8000000 |
| 2 | 0.00086100 | SELECT query_id, seq,state, duration, source_function FROM information_schema.profiling WHERE query_id = 1 |
...


SHOW PROFILE
SHOW PROFILE [type [, type] ... ]
[FOR QUERY n]
[LIMIT row_count [OFFSET offset]]

type:
ALL
| BLOCK IO
| CONTEXT SWITCHES
| CPU
| IPC
| MEMORY
| PAGE FAULTS
| SOURCE
| SWAPS

mysql> SHOW PROFILE FOR QUERY 3;
+--------------------+----------+
| Status | Duration |
+--------------------+----------+
| starting | 0.000091 |
| freeing items | 0.000040 |
| logging slow query | 0.000007 |
| cleaning up | 0.000007 |
+--------------------+----------+
4 rows in set (0.00 sec)

INFORMATION_SCHEMA.PROFILING

There are much more interersting queries you can get out of this database object, but this simple query fits in the blog eaily and provides a hint of detailed information available.

mysql> SELECT query_id, duration, state
FROM information_schema.profiling WHERE query_id = 8;
+----------+----------+--------------------+
| query_id | duration | state |
+----------+----------+--------------------+
| 8 | 0.000100 | starting |
| 8 | 0.000068 | Opening tables |
| 8 | 0.000008 | System lock |
| 8 | 0.000013 | Table lock |
| 8 | 0.000028 | init |
| 8 | 0.000015 | optimizing |
| 8 | 0.000016 | statistics |
| 8 | 0.000017 | preparing |
| 8 | 0.001850 | executing |
| 8 | 0.000162 | Sending data |
| 8 | 0.000008 | end |
| 8 | 0.000006 | query end |
| 8 | 0.000069 | freeing items |
| 8 | 0.000034 | removing tmp table |
| 8 | 0.000007 | closing tables |
| 8 | 0.000005 | logging slow query |
| 8 | 0.000007 | cleaning up |
+----------+----------+--------------------+
17 rows in set (0.01 sec)

Thoughts on Profiling

The profiling feature needs to mature more in MySQL, yet this early implementation does provide information that can be useful for evaluating SQL statements. Combining profile information, thread information and session state information together provides different perspectives into the evaluation of SQL processing.