MySQLMove 1.2 Adds PivotWizard…Works with SQL Too..

In addition to the lack of effective database migration components  in the free software management tools for use in MySQL, there was another piece missing that SQL Server developers have had forever. That is the lack of a PIVOT function. Yes with various hacks you can achieve the same idea in your MySQL query but never to the power that SQL gives us. So since MySQLMove was intended to offer some easy to use tools to achieve what is lacking there, I decided to add pivot functionality for data in MySQL to the toolbox here. I also decided to make it available for use with SQL Server. You may ask why I did that since it is an easy matter to use the PIVOT function there or some of the other excellent tools afforded us in SQL Server Management Studio, SSRS and Excel. The reason for that is this. While experienced developers are very happy to use these tools, some less so experienced and the average person wanted a tool that didn’t seem so ….. huge to them. The idea of drag and drop and seeing all these options put out there that have a learning curve to them intimidates and makes them not want to use tools provided there. Plus the fact, not everyone is comfortable with visually based environments. It also does not hurt that as for me, a primarily SQL Server based back end guy, I can use or teach this tool to be used in minutes.

So lets talk about the MySQLMove Pivot Wizard for a bit…..

First you must get the installation media here…. If you have installed a version of MySQLMove previously, you might want to uninstall it first. The Microsoft provided template installer is buggy when it comes to removing previous versions of anyway. But here is the location of the installation file.

After install, double click the icon MySQLMove on your desktop and you will see:

Click the “Pivot Wizard” button.

 

It is pretty straightforward. Keep in mind if you are using non Windows based MySQL Servers the case of these properties are a bigger deal then some Windows based systems. Notice the checkboxes on the bottom to choose your server type. You have to choose “MySQL” or “SQL”.

Also note the “?” button. If you need help anytime during the pivot wizard, you can click on these buttons to get that. I am trying something different here. Basically I am trying out Facebook for use as a tool to get end users support. When you click on these blue buttons, it will take you to a specific post on a Facebook group called “MySQLMove Help”. Ok you don’t have to be in awe of the such original naming lol. You don’t need to join to use for the help buttons to work. In fact, I don’t know why you would. I have disabled comments so it isn’t a discussion forum. A software developer who isn’t overly social. There’s a shock right?

After your login information has been accepted you will see this:

After you enter your statement, you click the “Parse” button. One of three things will happen. It will either tell you

1. Your statement could not be compiled

2. It compiles correctly and move on to the next screen of the wizard.

3. It compiles correctly but there were issues with the data. MySQL can be finicky this way if  you doing a lot of joins and there are duplicate columns or data that has constraints. It will inform you of the issues, show you a report but if it is not fatal it will allow you to continue. I do this because not every situation we encounter in real life has perfect data.

Also know if you are concerned about injection attacks, I have taken measures to ensure that no such thing can occur. I have barred use of several keywords that would be needed to do such. So rest easy on that.

After that is done you encounter the screen that asks you what columns you want to see depicted as rows. You use the arrows in the middle to add or remove columns that will be used as rows. In this example, I have four breakdowns occurring…. the last of the salesman is king seeing it as first. It then breaks it down into customers, categories of products sold and then the actual product.

You can use the arrows to the right to move your selected row items to different positions of priority. Once you are happy with what you have you click the Next button.

The next screen is the data headers button. This is the data that will actually be presented in the report. You have several options available to you to choose how to show that data. They are:

Average
The average of the values.
Count
The number of values (excluding Null and DBNull values).
Max
The largest value.
Min
The smallest value.
StdDev
An estimate of the standard deviation of a population, where the sample is a subset of the entire population.
StdDevp
The standard deviation of a population, where the population is all of the data to be summarized.
Sum
The sum of the values.
Var
An estimate of the variance of a population, where the sample is a subset of the entire population.
Varp
The variance of a population, where the population is all of the data to be summarized.

For this example we counting the number of orders that meet our row criteria. You can have up to eight data fields.

Once you are satisfied with the data field selection click the Next button.

You then have a screen that asks you what columns do you want to appear in your pivot report and how to group that data in relation to your data field of how many orders were submitted. In this example we have asked for order date and we selected from the drop down to group that information by quarter. You can choose as many columns as your heart desires or the size of your paper you are printing on will allow.

Other grouping methods available here are:

Alphabetical
Combines field values into categories according to the character that the values start with.

Date
This option is in effect only for fields that store date/time values.
Field values are grouped by the date part. The time part of the values is ignored.

DateDay
This option is in effect only for fields that store date/time values.
Field values are grouped by the day part. The following groups can be created: 1, 2, 3,…,31.

DateDayOfWeek
This option is in effect only for fields that store date/time values.
Field values are grouped by the days of the week. Examples of such groups: Sunday, Monday, Tuesday (the actual names of the days of the week are determined by the current culture).

DateDayOfYear
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of the day in which they occur in a year. The following groups can be created: 1, 2, 3,…,365 (,366 in a leap year).

DateHour
This option is in effect only for fields that store date/time values.
Field values are grouped by the date part with the hour value. Examples of such groups: 3/4/2012 0:00, 3/4/2012 1:00, 3/4/2012 2:00, …

DateHourMinute
This option is in effect only for fields that store date/time values.
Field values are grouped by the date part with the hour and minute values. Examples of groups: 3/4/2012 0:00, 3/4/2012 0:01, 3/4/2012 0:02, …

DateHourMinuteSecond
This option is in effect only for fields that store date/time values.
Field values are grouped by the date part with the hour, minute and second values. Examples of groups: 3/4/2012 0:00:00, 3/4/2012 0:00:01, 3/4/2012 0:00:02, …

DateMonth
This option is in effect only for fields that store date/time values.
Field values are grouped by the month part. Examples of groups: January, February, March (the actual names of the months are determined by the current culture).

DateMonthYear
This option is in effect only for fields that store date/time values.
Field values are grouped by months and years. Examples of groups: August 2013, September 2014, January 2015, …

DateQuarter
This option is in effect only for fields that store date/time values.
Field values are sorted by the quarterly intervals of the year. The following groups can be created: 1, 2, 3 and 4. Each quarter includes three months.

DateQuarterYear
This option is in effect only for fields that store date/time values.
Field values are grouped by the year and quarter. Examples of groups: Q3 2012, Q4 2012, Q1 2013, Q2 2013, …

DateWeekOfMonth
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of the week in which they occur in a month. The following groups can be created: 1, 2, 3, 4 and 5. The first week is the week containing the 1st day of the month.

DateWeekOfYear
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of the week in a year in which they occur. The following groups can be created: 1, 2, 3,…,52, 53.
Week numbers are calculated based on the following current culture’s settings.

DateYear
This option is in effect only for fields that store date/time values.
Field values are grouped by the year part. Examples of such groups: 2003, 2004, 2005.

DayAge
This option is in effect only for fields that store date/time values. Field values are grouped by the number of full days that have elapsed till the current date.

Default
Groups combine unique field values.

Hour
This option is in effect only for fields that store date/time values.
Field values are grouped by the hour part, regardless of the date to which the current date/time value belongs.

Minute
This option is in effect only for fields that store date/time values.
Field values are grouped by the minute part, regardless of the date to which the current date/time value belongs.

MonthAge
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of full months that have elapsed till the current date.

Numeric
This option is in effect only for fields that store numeric values.
Field values are grouped into intervals.

Second
This option is in effect only for fields that store date/time values.
Field values are grouped by the second part, regardless of the date to which the current date/time value belongs.

WeekAge
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of full weeks that have elapsed till the current date.

YearAge
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of full years that have elapsed till the current date.

You then click the Next button.

You then see a screen that lets you know it is ready to produce your report. You can still at this point go back and change any parameter you have chosen. Click Finish to produce the report.

You then see a preview of your report. You can see the rows broken down as we have selected. What I want you to pay attention to here is the various ways to export your report. Obviously if you can’t take it with you this data is no good to you.

You choose the format you want and a save dialog will appear. It will then ask you if you want to view it. Go ahead and admire your handiwork.

I have saved the example we created here just in case you want to review it.

Future modifications that I would like to make to this report will be to allow you to save a report setting so you don’t have to redo it each time, give you more appearance options, and integrate a chart wizard type interface that will allow you to build charts based on the data you have produced here. I am also considering building a web interface for the Migration and Pivot Wizard functions.

If you find the tool useful and would like to continue to see it developed, please don’t be a cheap bastard like me  and make a donation here using my email address – kellyjmartens@hotmail.com . Also feel free to send feedback, suggestions, critiques and praise to the same email address. All are read.

Once again here is the download link if you are too lazy to scroll back up and find it….Here is the installation file.

That about covers it. Thank you for reading this and have a great day!

MySQLMove – An Easy To Use Tool To Move MySQL Database Objects and Data

Recently, I was asked to work on a project using a MySQL backend after many years of not having worked with the product. I was amazed when I saw that some of the more common tools to work with MySQL, such as MySQL Workbench or Toad, had still not developed an easy way to move objects and/or data between mySQL databases such as we have become accustomed to in SQL Server Management Studio.  Obviously, this greatly complicated my work and caused me to have to work with data in a production environment more often, which obviously I wasn’t comfortable with.

So like most developers, my brain began to think about how to solve this problem. You could of course, go through each object, get the CREATE script, put it in a giant file, and run it that way. That would be time consuming, bulky and difficult to maintain should there be changes to the database objects. And of course, this did nothing for the issue of data migration.

I looked at other products. MySQLDump, a free tool is fine. Of course, it was more complicated then I would have liked. I just want to move the objects and data and not have a high learning curve of a new product. Just get it done already. Others were available that you had to pay for. Being the cheap bastard that I am, that option didn’t appeal to me either.

So I wrote MySQLMove. This started out as a quick and dirty option to get objects and data moved to another database on another server and gradually moved to including things like a report, the option to only migrate objects, and actually put a user interface on top instead of just running a script. The script was fine for me but it may not be for other developers.

Let’s go into some things you need to know or generally be aware of.

First, some MySQL developers might wonder why I did not use INFILE to import data. The reason is for many such developers, they are coding against a shared hosting environment and often the administrators in such environment have disabled this option. Even using the LOCAL keyword presented problems relating to security. However, in future releases (but see below) I am planning on a interface modification that will allow the end user to indicate that INFILE is available for use and it will operate accordingly.

UPDATE:

The engine now uses INFILE and LOCAL keywords. It is also now lightening fast!

Please make sure you have created the database on the destination server. I know that comes from the “duh” department but you would be surprised…. If you are asking yourself the reason why I don’t do it for you, it is because I drop each object on the destination individually as it is being imported. But more importantly, often each database has a specific username and password designated for access. An idea submitted for future versions (but see below) is that we collect all databases under a given username and password and allow multiple databases to be migrated at one time.  That day is not here yet but it intrigues me.

Another “duh” is it is important to make sure the destination database doesn’t have any users writing changes.

Next…. it is important that if you are importing from or to a Linux based environment, that you are absolutely certain of the case of your database, its objects and server name. Windows is much more forgiving then Linux is in this regard. If you are importing from Linux, be prepared for windows to make all objects lower case, regardless of how you created them in Linux based servers. I have also seen some versions of MySQL run on Windows that were also sticklers for such. My best advice to you is to make all objects lower case. Both Linux and Windows based MySQL servers can handle such and if you are often moving between the two systems you will save yourself a lot of trouble.

Next, you will see an option in a checkbox to only import MySQL objects.

If you have the time and the data available to you in text or csv format, I suggest you check this. Why? While this engine can and will move data, it takes longer then a manual import would. The way it is built is all of your insert statements for the data are collected as one statement and inserted collectively. It was about 20 percent faster then inserting a row at a time and also if it should bomb, I can rollback the transaction so that you are not left with a table with partial data. That said, if you are one of those folks who build 300+ column tables, (by the way who hurt you as a child to do that to yourself? jk) it is going to take awhile. But rest assured, it will get there. This was tested against a table with 80000 rows and 30 columns. So it is pretty robust.

UPDATE:

As mentioned above I now use INFILE and LOCAL keyword. The above text is no longer a concern.

Next, it is important you let the code finish. Check that, it is imperative. If you are impatient, switch to decaf and try some breathing exercises.

Finally, MySQLMove will attempt to export database objects as follows: tables, stored procedures, functions and triggers. Not every single one of all those objects can be migrated using this tool. One issue that has commonly arisen is when the code uses a PREPARE statement along with a string for use as the command. Future versions (but see below) will probably rectify this. Should something not be exportable to the destination, at the end of the process a report will appear showing what objects weren’t able to be moved. In my testing, it is a rare occurrence. Also with triggers, remember the “defined user” is copied straight from the trigger to the destination. You will need to make sure that user exists on the destination database as well. Oh and by the way, triggers are migrated last. Otherwise each insert might cause that trigger to fire. And yeah that would not be fun.

Finally, I would love to continue developing this tool. There are so many things I could and would love to do with it. Better reports, table optimization, code optimization, object selection etc. Unfortunately that costs me time, which in translation for those that live under a rock, this means it costs me money. If you find the tool useful and would like to continue to see it developed, please don’t be a cheap bastard like me and make a donation here using my email address – kellyjmartens@hotmail.com .

 

I would be extremely grateful. I know the honor system is putting yourself out there isn’t real effective but it gives those of you who would like to say thank you and keep going with this a chance to do so.

So on to the download…. You have two options….

One if you are a person who already has the MySql.Data dll already installed on your system (make sure it is version 6.9.9.0 and you have at least .NET runtime v4.0.30319) you can download just the bin folder. If you don’t or aren’t sure, get the install package zip file called “MySqlMoveSetup”. If you need the .NET Framework you can download it here.

Both the bin folder zip file and the installer zip file are located here on my OneDrive.

Please do send me an email at kellyjmartens@hotmail.com with any bug reports, suggestions or praise. All are accepted.  

 

Have a great day!