Managing Data

November 2, 2009

Silent Install Instructions

Filed under: Uncategorized — lstigile @ 10:50 pm
Tags: , ,

This entry provides instructions for “silently” installing MySQL on a Microsoft Windows server.  I use the term “silent install” to describe the method of passing parameters to the Microsoft Installation Package (MSI) to bypass installation screens.  This allows you to deploy MySQL with your application and to install MySQL without end-user intervention.  The developer can define the install location, the appropriate storage engines, the root password and the server configuration.

Most application developers use some sort of software for creating an application installer.  Common tools include Install Shield, Wise Installation, InstallBuilder and Windows Installer.  These installation software tools can also be used to install MySQL by launching the MySQL MSI and the MySQL Instance configuration utility ( MySQLInstanceConfig.exe ) from the command-line.

To get started, download the Windows Essentials from http://dev.mysql.com/downloads.

Then, look for an option in your installation tool to execute the following:

C:\>msiexec /q /log install.txt /i mysql-advanced-5.1.32-win32.msi datadir=”c:\installs\myapp” installdir=”c:\installs\myapp”

The MSI installer will add MySQL to the start menu, add registry values and add MySQL to the add/remove programs option from the control panel.

To configure the server, prepare the my.ini, set the root password and start the service, you’ll need to execute the following:

C:\>MySQLInstanceConfig.exe -i -q “-lC:\mysql_install_log.txt” “-nMySQL Server 5.1.234” -pC:\installs\myapp”   -v5.1.234  “tc:\installs\myapp\my-small.ini” “-cC:\mytest.ini ServerType=DEVELOPMENT DatabaseType=MIXED ConnectionUsage=DSS Port=3311 ServiceName=MySQLCust RootPassword=1234

This installation method has the benefit of adding MySQL to the control panel and the startup menus.  It also installs the default mysql tables.  The only remaining task is to import your application tables into MySQL by running mysqldump ( http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html ) on your development server and then loading the dump files into the target server.

In a future blog entry, I’ll discuss launching the mysqld.exe directly without actually installing MySQL.

June 17, 2009

MySQL Windows Password Reset

Filed under: Uncategorized — lstigile @ 8:14 pm

If you receive a 1045 error, you may need to reset your password.  Here are the official instructions for resetting the password: http://dev.mysql.com/doc/refman/5.1/en/resetting-permissions.html

If you find these instructions confusing, here’s another method.  

1.  Find the my.ini file (my.cnf for linux).  On windows, the my.ini file is typically in your install directory: c:\Program Files\MySQL\MySQL Server 5.1\my.ini   Open the file with notepad (or any text editor) and add the following entry under the [mysqld] section:

skip-grant-tables

Then, save the file and restart the MySQL service.  This has disabled the grant tables so you should be able to log into mysql without any password ( no more 1045 errors ).  Now, we need to reset the password.  

2.  Log into MySQL with the MySQL command-line utility.  This can be found in the MySQL bin directory.  No user or password parameters are required since grant tables have been disabled.  Execute the following statement:

UPDATE mysql.user SET PASSWORD = PASSWORD(”) WHERE user = ‘root’ and host=’localhost’;

This example will replace the root password with an empty string or no password.  Feel free to substitute the appropriate password in the PASSWORD(”) function or leave it blank and change the password later.

3.  Remove “skip-grant-tables’ from the my.ini file and restart the MySQL service.

4.  Log into mysql.  Example: shell>mysql -u root

I hope this helps.

June 15, 2009

MySQL Windows Install Troubleshooting

Filed under: Uncategorized — lstigile @ 9:04 pm
Tags: , , , ,

If you’re having trouble installing MySQL 5.1 on a Windows server, I’d recommend the following process.  Don’t skip the first steps that describe the uninstall and removal of the previous install as this seems to clear up most issues.   For the uninstall, I’d recommend manually deleting data files.  Caution: deleting the data files will delete any data previously entered into MySQL.

1.  Remove previous MySQL installs (and remnants)

a.  Stop the MySQL service ( Start | Control Panel | Administrative Tools | Services)

Picture 2

b.  Remove MySQL using Windows Add or Remove Programs (Start | Control Panel | Add or Remove Programs)

Picture 3
c.  Clean up (delete) the data directory  (assuming that you don’t have data that you need) c:\documents and settings\all users\application data\mysql
 

Picture 4

 

d.  Clean up (delete) any remaining directories under program files c:\program files\MySQL\MySQL Server 5.1 (or whatever version you’re using)

 

————–

That’s it for the removal and clean up.  Now, we’ll begin the install.

 

2.  The MySQL install is a two step process: running the installer (.MSI) and then running the MySQL configuration utility (MySQLInstanceConfig.exe).  Theoretically, the first step should install the data files and program files and the second step should set up the service, set the root password and start the service.  The installer is designed to call the configuration utility automatically.   Instead, I recommend running each process separately.   

 

a.  Launch the installer from a command-line.  Here’s an example shell>msiexec /log c:\install.txt /i mysql-5.1.34-win32.msi 

 

Note: You’ll need to change the name of the .msi file to match the name of the version that you’re installing.  If you’re not used to working at the command-line, you’ll either need to change the directory to the location of the mysql installation file or specify the full path.  For example, you may need to execute:  msiexec /log c:\install.txt /i “c:\Documents and Settings\Administrator\Desktop\mysql-5.1.34-win32.msi”

 

Continue through the wizard until you reach the final screen which says, “Wizard Completed”  There’s a checkbox to “Configure the MySQL Server now”.  Uncheck the box and select Finish.

 

 

Uncheck the Configure box.

Uncheck the Configure box.

 When the install has finished, please check c:\documents and settings\all users\application data\mysql\mysql server 5.1\data .  You should see a mysql and test directory.  

 

Also, verify that files have been installed in c:\program files\MySQL\MySQL Server 5.1 

You should have a bin, Docs and share folder.  

 

b.  Navigate to c:\program files\MySQL\MySQL Server 5.1\bin and launch the MySQLInstanceConfig.exe from a command-line.  Here’s an example.

shell>MySQLInstanceConfig.exe “-lC:\mysql_install_log.txt” “

 

I suggest launching it from a command-line so that you can pass the -l parameter, which will create an install log in case you need to troubleshoot the problem.

 

Step through the wizard.  Make sure to remember the password that you assign in the wizard.

 

The install should complete successfully.

 

TROUBLESHOOTING

What if it didn’t work?

 

1.  Check the service from the control panel.  Is it running?  If so, try to log into MySQL.  I’ve seen instances where an error was received, but it completed successfully.  If it’s not running, try to start it and please post the complete error message.

 

2.  Check the error log.  It’s in C:\Documents and Settings\All Users\Application Data\MySQL\MySQL Server 5.1\data   It has an .err extension and you should be able to open it with notepad.  Ignore any errors about the innodb plugin not loading.  It’s a symptom, not the problem.  Look for an error about missing data files or a mismatch in file sizes.

 

3.  1045 error?  It should indicate Password: No or Password:yes.  If “yes”, it’s not a port issue, not a firewall issue — it’s a password issue.  It’s usually caused by not deleting out the data files from an old install.  The password is kept in the data directory and it’s likely that the password you entered during the previous install doesn’t match what you’re entering now.  You can either re-install after deleting all the files or try to reset your root password.  (resetting root password: http://dev.mysql.com/doc/refman/5.1/en/resetting-permissions.html )

 

4.  Still stuck?   In the instructions above, I specified c:\install.txt and c:\mysql_install_log.txt  Please check those and post the contents.


May 19, 2009

Using libmysqld with Microsoft Visual C++ 2008 Express

Filed under: Uncategorized — lstigile @ 7:46 pm
Tags: ,

[ Note: this post was updated on Dec 2, 2009 to correct dashes. In the original post, dash dash -- was showing up as a single dash. This is noted in the comments and has been corrected in the text below.]

These are very (very) basic instructions for setting up a very simple application with Microsoft Visual C++ 2008 Express to use the MySQL Embedded Library (libmysqld).

The MySQL Embedded Library is a full version of the server that is available as a dynamic or static library.  Developers can run the library inside their application.  It’s extremely fast, easy to distribute and it’s ideal for stand-alone applications.  More information about the library can be found at http://dev.mysql.com/doc/refman/5.1/en/libmysqld.html

Having little experience using Visual C++, I’ve struggled to find instructions for setting up a project to use a dynamic library.  I finally got it to work thanks to Ulf Wendel’s blog about building a windows client using the Connector/C++ ( http://blog.ulf-wendel.de/?p=215 ).  I was able to take his tutorial, substitute the appropriate libmysqld entries and compile the project.  If you get lost following my instructions, refer back to Ulf’s blog.

To develop a libmysqld application, you’ll need data files, the error message file ( errmsg.sys ), the appropriate libraries and header files.  To get these files, I’d recommend downloading and installing the full version of the server.  Also, it’s easier to design and build your database using the standard version of the server.  Once you have some data, stop the MySQL service and then create your application using libmysqld.

Here’s the process that I used.

Download the Windows MSI Installer (x86) from http://dev.mysql.com/downloads.    For this example, I downloaded and installed version 5.1.34.  I selected the option for a standard install.  After the install, you can create tables and design your database as needed.  For this example, I’m going to use the “user” table, which installs with MySQL.

After the install, go to the services control panel ( Start | Control Panel | Administrative Tools | Services ) and stop MySQL.  You might also change the Startup type to “manual” to ensure that it doesn’t start on reboot and cause a conflict with your project.

Locate required directories for setting up the project:

1)  Data Files

On Windows, the default location for data files is C:\Documents and Settings\All Users\Application Data\MySQL\MySQL Server 5.1\data  We’ll be accessing the user.MYD MyISAM table, which is located in the .\mysql subdirectory.  Similarly, If you were to create a database named world with a city table, there would be a corresponding world directory and city.MYD MyISAM table in the data directory.

2) Error Message File

The error message file ( errmsg.sys ) is required to start libmysqld.  By default, it’s located at C:\Program Files\MySQL\MySQL Server 5.1\share\english

3)  Header Files

C:\Program Files\MySQL\MySQL Server 5.1\include

4) Embedded Library

C:\Program Files\MySQL\MySQL Server 5.1\Embedded\DLL\release

Copy the libmysqld.dll from the embedded library path (above) to the c:\windows\system32 directory (or anywhere in your search path).  For deployment, you’ll probably want to include the embedded library with the executable.

Time to setup up Microsoft Visual C++ Express

Launch Microsoft Visual C++ Express.  From the main menu, select File | New | Project

Choose a Win32 Console Application

g1

Select OK and Finish.

Right-click the project name in the Solution Explorer and select Properties.

g2

Expand Configuration Properties, C/C++ Properties and add the following for the Additional Include Directories:

C:\Program Files\MySQL\MySQL Server 5.1\include

g3

Expand Linker, select General and add the following to Additional Library Directories: C:\Program Files\MySQL\ MySQL Server 5.1\ Embedded\ DLL\releaseg4

Select Linker, Input and add the following for Additional Dependencies: libmysqld.libg5

Select OK to save the setting and return to the project.

Copy the following into the source file ( my copy is named sample.cpp )  Overwrite any existing code in the file.

#include “stdafx.h”

#include <stdlib.h>

#include <iostream>

#include <stdio.h>

#include <stdarg.h>

#include <windows.h>

#include <mysql.h>

MYSQL *mysql;

MYSQL_RES *results;

MYSQL_ROW record;

static char *server_options[] = { “mysql_test”, --datadir=C:/Documents and Settings/All Users/Application Data/MySQL/MySQL Server 5.1/data,

--language=C:/Program Files/MySQL/MySQL Server 5.1/share/english",  NULL };

int num_elements = sizeof(server_options)/ sizeof(char *) – 1;

static char *server_groups[] = { “libmysqld_server”, NULL };

int main(int argc, char* argv[])

{

int retval;

retval = mysql_library_init(num_elements, server_options, (char **)             server_groups);

mysql = mysql_init(NULL);

mysql_options(mysql, MYSQL_OPT_USE_EMBEDDED_CONNECTION, NULL);

mysql_real_connect(mysql, NULL,“root”,NULL, “mysql”, 0, NULL, 0);

mysql_query(mysql, “SELECT user, host FROM user”);

results = mysql_store_result(mysql);

while((record = mysql_fetch_row(results))) {

printf(“%s – %s \n”, record[0], record[1]);

}

mysql_free_result(results);

mysql_close(mysql);

return 0;

}

At this point, you should be able to build and run your project.  The output should look like this.

g6

It’s a simple listing of the users in the mysql table.

The following line of code is critical:

static char *server_options[] = { “mysql_test”, --datadir=C:/Documents and Settings/All Users/Application Data/MySQL/MySQL Server 5.1/data,

--language=C:/Program Files/MySQL/MySQL Server 5.1/share/english",  NULL };

You must have a datadir and a language file.  Note the forward slash in the directory paths.  Of course, these paths can be located anywhere.  You’ll probably want to move these folders to a subdirectory under your application install.  If you don’t see an error log in the data directory, that’s also an indicator that these paths are incorrect.

I hope someone finds this useful.

June 30, 2008

The Case for MySQL Cluster Certification

Filed under: Uncategorized — lstigile @ 9:46 pm
Tags: , , ,

Why are the best technical books also the thinnest? To make the case, I highly recommend the MySQL 5.1 Cluster Certification Study Guide In rack units, it’s a 1U; it fits nicely into my laptop bag; and if you’re considering implementing MySQL Cluster, it can save you a world of time.

Of course, if you get the book, you should consider the certification itself. There’s a legitimate debate about the usefulness of certification exams, but the MySQL 5.1 Cluster certification is a little more important than others for a couple of reasons:

1. In case you haven’t heard it by now, Cluster isn’t always the best fit. In many cases, an active/passive failover setup is a much more flexible and cost-effective approach. It’s not always a clear-cut decision. If you’re willing to make application changes and if you understand the internal workings of MySQL Cluster, it can be an amazing solution.

To see if cluster is a good fit, I recommend the following guides:

http://www.mysql.com/why-mysql/white-papers/mysql_cluster_eval_guide.php

http://www.mysql.com/why-mysql/white-papers/mysql_ha_solutions.php

2. There aren’t many consultants and DBA’s that have actually installed and used MySQL Cluster. There’s still a lot of misconceptions. The primary misconception is: “I know MySQL, I’m sure that I can quickly set it up and support a MySQL Cluster.” Outside of MySQL professional services group, there’s limited expertise in the marketplace. If someone has a MySQL Certification, at least you know that they’re tall enough for this ride.

I thought the most important part of the book was Chapter 7 regarding Indexes in NDB tables. Unless specifically suppressed, a hash and ordered (T-tree) index will be created for primary keys. The hash index is stored in the IndexMemory pool. The ordered index is stored in the DataMemory pool.

If you want to create a unique index, that’s a bit more complicated and it takes about a page to explain. Unique indexes have performance implications, which provides even more reason that someone should really understand MySQL internals before implementing a cluster.

To summarize, I’m hesitant to say this in public, but frankly; the study guide is a good read. It won’t make the Oprah book club any time soon, but it has some good discussions about hash indexing vs. t-tree indexes, database isolation levels, ACID properties and high availability concepts.

[sidenote: I usually rip technical books in half to achieve 1U. Usually, you can safely throw the first half away.]

December 27, 2007

Tune Drupal, XOOPS, Joomla!, Alfresco…with MySQL Enterprise Monitor

Filed under: Uncategorized — lstigile @ 10:10 pm

Have you ever tuned the MySQL instance underlying your content management system (CMS)? For most users of Drupal, Joomla!, XOOPS, Alfresco, or any of the other CMS, the answer is no.

However, I’m seeing an increasing number of mission-critical corporate sites running on open source CMS. Take a look at Alfresco’s customer listing, http://www.alfresco.com/customers , which includes H&R Block, Boise Cascade and Harvard Business School Publishing. If your CMS site has become a crucial part of your enterprise, it’s time to take a closer look at the database.

The MySQL Enterprise Monitor is designed to help developers configure, tune and administer their MySQL Servers. After watching a Drupal presentation in a local meetup, I decided to evaluate the Monitor with Drupal.

Test Scenario
1. Install Drupal and MySQL
2. Install the MySQL Enterprise Monitor ( A trial can be downloaded from http://www.mysql.com/ent-trial-reg-2007 )
3. Add a load to the site using DieselTest ( http://www.dieseltest.com )

MySQL Enterprise Monitor
The Monitor has two components: the agent, which collects statistics (600+) from the production server; and the dashboard, a web-based interface, which is installed on a non-production machine within the firewall. The dashboard can be used to determine what gets monitored (70+ options), to configure email notifications and to view 20+ graphs.

Figure 1.Figure 1.

Heat Chart
By default, the Monitor includes 12 rules that comprise the heat chart (figure 1). The heat chart is on the front page of the web-interface and provides a one-glance status report on your MySQL server. It monitors if the server is up or down; the I/O, CPU and RAM usage; database connections; key and query cache; table scans; temporary tables-to-disk ratio and lock contention.

Critical Alert! -Query Cache Has a Sub-Optimal Hit Rate
The Monitor instantly threw a critical alert that the Query Cache Has a Sub-Optimal Hit Rate. The query cache holds not only the SQL statement, but the data as well. Results for identical statements can be instantly retrieved from cache. However, it also means than any updates to the underlying table will force the cache to be invalidated. Since my load test was primarily duplicate reads, I should probably follow the advice of the alert and increase query_cache_size.

Security and Administration
With version 1.2, there are 18 security rules and 16 Administration rules. These rules will provide email notification for issues ranging from the existence of anonymous user accounts to an excessive number of attempted connections (DOS attack?). I actually received 3 administrative alerts and 7 security alerts. I had covered the basics, but it still triggered several warnings including advice to disable symlinks and to disable older (weaker) 4.1 password authentications.

Adding a Load to Drupal
I browsed the internet and selected Dieseltest (http://www.dieseltest.com) for load generation. Dieseltest looked easy, and it was. I simply recorded some blog and forum updates and some browsing on my drupal site. Then, I re-played the script with 50 virtual users for 10 minutes. Of course, this was a fairly rudimentary process, but it did add load to my database server (see figure 2).

Figure 2. Figure 2.

Performance
In all, I received nine performance events. I outlined the two most critical below:

Critical Alert! -Indexes Not Being Used Efficiently
As I suspected, the default database could use some indexing. See figure 3 for rows read via indexes versus rows read with a full table scan. As the alert suggested, I need to enable –log-queries-not-using-indexes.

Figure 3.Figure 3.

Critical Alert! – Excessive Disk Temporary Table Usage Detected
I may need to do more research on this one. I suspect that the temp tables are a result of using a lot of TEXT data types for storing blog entries. Perhaps I can change some data types to varchar so that disk temporary tables aren’t automatically used. The alert provides plenty of detailed information for adjusting parameters and links to more online references.

Conclusion
A web site developer recently told me that he designed his sites without ever changing the default MySQL installation. While this speaks volumes to the ease-of-use of MySQL, if your site becomes mission-critical or has grown to handle more traffic than you anticipated, consider installing the MySQL Enterprise Monitor for configuring, monitoring and tuning your server.

Theme: Rubric. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.