Northwind Database For Oracle

Ah Northwind. The database everybody learns on. Well almost everyone. I realize it is a Microsoft product but you would have thought somebody along the line would have developed the same for Oracle and MySQL.

Why did this come up? In the next rendition of MySQLMove (which is being renamed for the next release – I still haven’t settled on a title for it yet. Something that indicates it is multi environment data collector and reporter. If you have suggestions or ideas please let me know…) I have been telling you about I wanted to add Oracle back ends as an option to get your data from. During the examples on past releases I used Northwind to demonstrate how to use the software. So I looked high and low on the web and didn’t find a single useable version of Northwind for Oracle. So I did it the old fashioned way. I exported my Northwind from SQL, table by table to excel files and imported them into Oracle. There’s an hour of my life I won’t get back. So just in case any one else has to go through this I pulled all the Oracle scripts for creating the tables and inserting the data, zipped them up and have made them available to you for download. All you have to do is download it, unzip it and run the scripts. Why? Because I am a nice guy that way and I had the time.  If I saved anyone some time, it was worth it.

MySQLMove PivotWizard Adds SQL,CSV, MS Access Support and Chart Creation

If you haven’t been following along, MySQLMove is a tool that started out as a way for MySQL developers to do things the free tools to manage MySQL did not manage well. One of these things was the PivotWizard, which originally allowed MySQL developers, who do not have access to a pivot related function, to accomplish so here. As time went on, it became the most popular part of this software. The early versions of the PivotWizard were crude and did just the basics of pivoting data. I first added SQL support. Now I have added Microsoft Access and CSV file support, in addition to MySQL and SQL Server Database support and also have given you the ability to choose from 50+ different chart types to be able to look at your data in different ways. A lot has changed so I am going to go through it again, step by step.

You can download it here. Be sure to remove the old version if you installed it as the default MS install package can be buggy that way about removing it. Below is the directions on how to use the PivotWizard.

When you open MySQLMove you will see the following:

Click on the “Pivot Wizard”.

You will then see the screen to allow you to specify your connection type and parameters. If you choose MS Access you can use mdb or accdb file types and can specify if you use a password for it not. If you choose MS Access or CSV, a file dialog box will appear and ask  you to browse to the path your file is located at. If you choose CSV, it MUST be a comma delimited file and you should have a header row to define the columns. (Incidentally I should note that as long as your CSV files are located inside the same directory (folder) you specified in your connection you can join separate files in a query just as if you were joining two SQL tables like so:

SELECT *FROM [Orders.csv] a
INNER JOIN [Employees.csv] b on b.[ID] = a.[Employee ID]

No need for tools like PowerShell, other applications or confusing syntax.)

If you choose MySQL, be sure to precise on your casing of words. For this example, we will check the box for SQL Server and fill in its connection parameters.

I suggest you click on the Test button and verify all is well. I will check it for you before you leave the following screen but may as well get that out of the way now. If you get any of this information incorrect, you can’t proceed until it is fixed.

Once this is done, click the “Next” button.

You will then be asked to type or copy and paste a query based on this connection into the textbox. If you chose CSV format in the earlier screen this information will be filled in for you. After  you enter the query, you must click the “Parse” button. This will verify your connection is correct and your statement compiles. If it does, you will be told via message box how many columns you have to work with and then you will click “Next” to proceed. If it does not compile correctly, you will be informed and you will have to fix the statement to be able to proceed. (Please don’t email me about those kinds of issues. I really don’t have time for that.). You might also see a prompt letting you know it compiled with errors. As long as the issues aren’t fatal, you will be allowed to proceed and a report will be given to you letting you know what the problems were. Below the query box, you are asked to identify what you want Excel to call your report. Since this is the raw data, I called it “Raw Data”. This field is not mandatory. You might also notice the “?” mark icon. On the previous version, I attempted to have you directed to links to the MySQL Facebook group to answer common questions. That was a miserable failure. Instead in this edition, I have placed a tooltip that when you hover over the icon, it gives an explanation of what is going on.

Once this step is completed click the “Next” button.

Here you will pick what columns you will want to appear as the rows in your pivot report. You can select multiple columns as rows, just be sure they are related somehow. Once selected you are allowed to choose a name for the field. Excel will use that name instead of the field name in your report. Click the “Next” button to continue.

You will then be asked to choose “data” for your report. In this case I have chosen to see how many Order_IDs exist for each salesman (see the last screen). Use the arrows to choose your fields (you can have more then one, and you can reuse the same field if need be.) and select a grouping to be applied. Click the “Next” button to continue.

You will then be asked to choose the columns you want to represent in your pivot report and how you want the data grouped. If  you notice we choose to see Order_Date twice in our columns because we want it grouped by Quarter and inside that quarter, by Day of Week. I add a column to your original query with the same name and data with a “_1” after it. So now we have asked to see how many orders our salesman did, divided into quarter and inside that quarter by Day of Week. Play with it, you will get the hang of it. Click “Next” to continue.

Here you choose the type of chart to represent the data you have requested and in Chart Title have given Excel the information it needs to call your sheet and chart name what you want. Hopefully the images of the charts you see will assist you in making the choice of charts that you want. Click the “Next” button to continue.

Finally, at the end. Now you can give your pivot report a name for Excel to use for your pivoted data (or not if you don’t care) and then click the “Finish” button. Be patient, it takes a few second for Excel to show you your work. Just as a word of caution, I tried to anticipate anything you could do wrong in naming your charts and reports. If something you should happen, you will get a notice of what the problem is but it will allow you to still see what you have produced. It just will not give the worksheet name you tried to give it.

You can see the sample we produced here.

The next version of this software will allow you to save templates of different reports and allow you to create more then just one query. The next version will allow you to store multiple connections, multiple querys and pivot reports and charts specific to each query, each produced into the same Excel Workbook. Keep an eye on here and on the MySQLMove Facebook page for updates.Again you can download the install package for MySQLMove here.

Well that about covers it. Well not quite. I have been asked if I would be willing to do a custom version of this for a specific person or company. The answer is of course, yes. Also if you have bugs you have seen please let me know. Just email me at kellyjmartens@hotmail.com and I would be happy to help.

MySQLMove 1.2 Adds PivotWizard…Works with SQL Too..

In addition to the lack of effective database migration components  in the free software management tools for use in MySQL, there was another piece missing that SQL Server developers have had forever. That is the lack of a PIVOT function. Yes with various hacks you can achieve the same idea in your MySQL query but never to the power that SQL gives us. So since MySQLMove was intended to offer some easy to use tools to achieve what is lacking there, I decided to add pivot functionality for data in MySQL to the toolbox here. I also decided to make it available for use with SQL Server. You may ask why I did that since it is an easy matter to use the PIVOT function there or some of the other excellent tools afforded us in SQL Server Management Studio, SSRS and Excel. The reason for that is this. While experienced developers are very happy to use these tools, some less so experienced and the average person wanted a tool that didn’t seem so ….. huge to them. The idea of drag and drop and seeing all these options put out there that have a learning curve to them intimidates and makes them not want to use tools provided there. Plus the fact, not everyone is comfortable with visually based environments. It also does not hurt that as for me, a primarily SQL Server based back end guy, I can use or teach this tool to be used in minutes.

So lets talk about the MySQLMove Pivot Wizard for a bit…..

First you must get the installation media here…. If you have installed a version of MySQLMove previously, you might want to uninstall it first. The Microsoft provided template installer is buggy when it comes to removing previous versions of anyway. But here is the location of the installation file.

After install, double click the icon MySQLMove on your desktop and you will see:

Click the “Pivot Wizard” button.

 

It is pretty straightforward. Keep in mind if you are using non Windows based MySQL Servers the case of these properties are a bigger deal then some Windows based systems. Notice the checkboxes on the bottom to choose your server type. You have to choose “MySQL” or “SQL”.

Also note the “?” button. If you need help anytime during the pivot wizard, you can click on these buttons to get that. I am trying something different here. Basically I am trying out Facebook for use as a tool to get end users support. When you click on these blue buttons, it will take you to a specific post on a Facebook group called “MySQLMove Help”. Ok you don’t have to be in awe of the such original naming lol. You don’t need to join to use for the help buttons to work. In fact, I don’t know why you would. I have disabled comments so it isn’t a discussion forum. A software developer who isn’t overly social. There’s a shock right?

After your login information has been accepted you will see this:

After you enter your statement, you click the “Parse” button. One of three things will happen. It will either tell you

1. Your statement could not be compiled

2. It compiles correctly and move on to the next screen of the wizard.

3. It compiles correctly but there were issues with the data. MySQL can be finicky this way if  you doing a lot of joins and there are duplicate columns or data that has constraints. It will inform you of the issues, show you a report but if it is not fatal it will allow you to continue. I do this because not every situation we encounter in real life has perfect data.

Also know if you are concerned about injection attacks, I have taken measures to ensure that no such thing can occur. I have barred use of several keywords that would be needed to do such. So rest easy on that.

After that is done you encounter the screen that asks you what columns you want to see depicted as rows. You use the arrows in the middle to add or remove columns that will be used as rows. In this example, I have four breakdowns occurring…. the last of the salesman is king seeing it as first. It then breaks it down into customers, categories of products sold and then the actual product.

You can use the arrows to the right to move your selected row items to different positions of priority. Once you are happy with what you have you click the Next button.

The next screen is the data headers button. This is the data that will actually be presented in the report. You have several options available to you to choose how to show that data. They are:

Average
The average of the values.
Count
The number of values (excluding Null and DBNull values).
Max
The largest value.
Min
The smallest value.
StdDev
An estimate of the standard deviation of a population, where the sample is a subset of the entire population.
StdDevp
The standard deviation of a population, where the population is all of the data to be summarized.
Sum
The sum of the values.
Var
An estimate of the variance of a population, where the sample is a subset of the entire population.
Varp
The variance of a population, where the population is all of the data to be summarized.

For this example we counting the number of orders that meet our row criteria. You can have up to eight data fields.

Once you are satisfied with the data field selection click the Next button.

You then have a screen that asks you what columns do you want to appear in your pivot report and how to group that data in relation to your data field of how many orders were submitted. In this example we have asked for order date and we selected from the drop down to group that information by quarter. You can choose as many columns as your heart desires or the size of your paper you are printing on will allow.

Other grouping methods available here are:

Alphabetical
Combines field values into categories according to the character that the values start with.

Date
This option is in effect only for fields that store date/time values.
Field values are grouped by the date part. The time part of the values is ignored.

DateDay
This option is in effect only for fields that store date/time values.
Field values are grouped by the day part. The following groups can be created: 1, 2, 3,…,31.

DateDayOfWeek
This option is in effect only for fields that store date/time values.
Field values are grouped by the days of the week. Examples of such groups: Sunday, Monday, Tuesday (the actual names of the days of the week are determined by the current culture).

DateDayOfYear
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of the day in which they occur in a year. The following groups can be created: 1, 2, 3,…,365 (,366 in a leap year).

DateHour
This option is in effect only for fields that store date/time values.
Field values are grouped by the date part with the hour value. Examples of such groups: 3/4/2012 0:00, 3/4/2012 1:00, 3/4/2012 2:00, …

DateHourMinute
This option is in effect only for fields that store date/time values.
Field values are grouped by the date part with the hour and minute values. Examples of groups: 3/4/2012 0:00, 3/4/2012 0:01, 3/4/2012 0:02, …

DateHourMinuteSecond
This option is in effect only for fields that store date/time values.
Field values are grouped by the date part with the hour, minute and second values. Examples of groups: 3/4/2012 0:00:00, 3/4/2012 0:00:01, 3/4/2012 0:00:02, …

DateMonth
This option is in effect only for fields that store date/time values.
Field values are grouped by the month part. Examples of groups: January, February, March (the actual names of the months are determined by the current culture).

DateMonthYear
This option is in effect only for fields that store date/time values.
Field values are grouped by months and years. Examples of groups: August 2013, September 2014, January 2015, …

DateQuarter
This option is in effect only for fields that store date/time values.
Field values are sorted by the quarterly intervals of the year. The following groups can be created: 1, 2, 3 and 4. Each quarter includes three months.

DateQuarterYear
This option is in effect only for fields that store date/time values.
Field values are grouped by the year and quarter. Examples of groups: Q3 2012, Q4 2012, Q1 2013, Q2 2013, …

DateWeekOfMonth
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of the week in which they occur in a month. The following groups can be created: 1, 2, 3, 4 and 5. The first week is the week containing the 1st day of the month.

DateWeekOfYear
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of the week in a year in which they occur. The following groups can be created: 1, 2, 3,…,52, 53.
Week numbers are calculated based on the following current culture’s settings.

DateYear
This option is in effect only for fields that store date/time values.
Field values are grouped by the year part. Examples of such groups: 2003, 2004, 2005.

DayAge
This option is in effect only for fields that store date/time values. Field values are grouped by the number of full days that have elapsed till the current date.

Default
Groups combine unique field values.

Hour
This option is in effect only for fields that store date/time values.
Field values are grouped by the hour part, regardless of the date to which the current date/time value belongs.

Minute
This option is in effect only for fields that store date/time values.
Field values are grouped by the minute part, regardless of the date to which the current date/time value belongs.

MonthAge
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of full months that have elapsed till the current date.

Numeric
This option is in effect only for fields that store numeric values.
Field values are grouped into intervals.

Second
This option is in effect only for fields that store date/time values.
Field values are grouped by the second part, regardless of the date to which the current date/time value belongs.

WeekAge
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of full weeks that have elapsed till the current date.

YearAge
This option is in effect only for fields that store date/time values.
Field values are grouped by the number of full years that have elapsed till the current date.

You then click the Next button.

You then see a screen that lets you know it is ready to produce your report. You can still at this point go back and change any parameter you have chosen. Click Finish to produce the report.

You then see a preview of your report. You can see the rows broken down as we have selected. What I want you to pay attention to here is the various ways to export your report. Obviously if you can’t take it with you this data is no good to you.

You choose the format you want and a save dialog will appear. It will then ask you if you want to view it. Go ahead and admire your handiwork.

I have saved the example we created here just in case you want to review it.

Future modifications that I would like to make to this report will be to allow you to save a report setting so you don’t have to redo it each time, give you more appearance options, and integrate a chart wizard type interface that will allow you to build charts based on the data you have produced here. I am also considering building a web interface for the Migration and Pivot Wizard functions.

If you find the tool useful and would like to continue to see it developed, please don’t be a cheap bastard like me  and make a donation here using my email address – kellyjmartens@hotmail.com . Also feel free to send feedback, suggestions, critiques and praise to the same email address. All are read.

Once again here is the download link if you are too lazy to scroll back up and find it….Here is the installation file.

That about covers it. Thank you for reading this and have a great day!

MySQLMove – An Easy To Use Tool To Move MySQL Database Objects and Data

Recently, I was asked to work on a project using a MySQL backend after many years of not having worked with the product. I was amazed when I saw that some of the more common tools to work with MySQL, such as MySQL Workbench or Toad, had still not developed an easy way to move objects and/or data between mySQL databases such as we have become accustomed to in SQL Server Management Studio.  Obviously, this greatly complicated my work and caused me to have to work with data in a production environment more often, which obviously I wasn’t comfortable with.

So like most developers, my brain began to think about how to solve this problem. You could of course, go through each object, get the CREATE script, put it in a giant file, and run it that way. That would be time consuming, bulky and difficult to maintain should there be changes to the database objects. And of course, this did nothing for the issue of data migration.

I looked at other products. MySQLDump, a free tool is fine. Of course, it was more complicated then I would have liked. I just want to move the objects and data and not have a high learning curve of a new product. Just get it done already. Others were available that you had to pay for. Being the cheap bastard that I am, that option didn’t appeal to me either.

So I wrote MySQLMove. This started out as a quick and dirty option to get objects and data moved to another database on another server and gradually moved to including things like a report, the option to only migrate objects, and actually put a user interface on top instead of just running a script. The script was fine for me but it may not be for other developers.

Let’s go into some things you need to know or generally be aware of.

First, some MySQL developers might wonder why I did not use INFILE to import data. The reason is for many such developers, they are coding against a shared hosting environment and often the administrators in such environment have disabled this option. Even using the LOCAL keyword presented problems relating to security. However, in future releases (but see below) I am planning on a interface modification that will allow the end user to indicate that INFILE is available for use and it will operate accordingly.

UPDATE:

The engine now uses INFILE and LOCAL keywords. It is also now lightening fast!

Please make sure you have created the database on the destination server. I know that comes from the “duh” department but you would be surprised…. If you are asking yourself the reason why I don’t do it for you, it is because I drop each object on the destination individually as it is being imported. But more importantly, often each database has a specific username and password designated for access. An idea submitted for future versions (but see below) is that we collect all databases under a given username and password and allow multiple databases to be migrated at one time.  That day is not here yet but it intrigues me.

Another “duh” is it is important to make sure the destination database doesn’t have any users writing changes.

Next…. it is important that if you are importing from or to a Linux based environment, that you are absolutely certain of the case of your database, its objects and server name. Windows is much more forgiving then Linux is in this regard. If you are importing from Linux, be prepared for windows to make all objects lower case, regardless of how you created them in Linux based servers. I have also seen some versions of MySQL run on Windows that were also sticklers for such. My best advice to you is to make all objects lower case. Both Linux and Windows based MySQL servers can handle such and if you are often moving between the two systems you will save yourself a lot of trouble.

Next, you will see an option in a checkbox to only import MySQL objects.

If you have the time and the data available to you in text or csv format, I suggest you check this. Why? While this engine can and will move data, it takes longer then a manual import would. The way it is built is all of your insert statements for the data are collected as one statement and inserted collectively. It was about 20 percent faster then inserting a row at a time and also if it should bomb, I can rollback the transaction so that you are not left with a table with partial data. That said, if you are one of those folks who build 300+ column tables, (by the way who hurt you as a child to do that to yourself? jk) it is going to take awhile. But rest assured, it will get there. This was tested against a table with 80000 rows and 30 columns. So it is pretty robust.

UPDATE:

As mentioned above I now use INFILE and LOCAL keyword. The above text is no longer a concern.

Next, it is important you let the code finish. Check that, it is imperative. If you are impatient, switch to decaf and try some breathing exercises.

Finally, MySQLMove will attempt to export database objects as follows: tables, stored procedures, functions and triggers. Not every single one of all those objects can be migrated using this tool. One issue that has commonly arisen is when the code uses a PREPARE statement along with a string for use as the command. Future versions (but see below) will probably rectify this. Should something not be exportable to the destination, at the end of the process a report will appear showing what objects weren’t able to be moved. In my testing, it is a rare occurrence. Also with triggers, remember the “defined user” is copied straight from the trigger to the destination. You will need to make sure that user exists on the destination database as well. Oh and by the way, triggers are migrated last. Otherwise each insert might cause that trigger to fire. And yeah that would not be fun.

Finally, I would love to continue developing this tool. There are so many things I could and would love to do with it. Better reports, table optimization, code optimization, object selection etc. Unfortunately that costs me time, which in translation for those that live under a rock, this means it costs me money. If you find the tool useful and would like to continue to see it developed, please don’t be a cheap bastard like me and make a donation here using my email address – kellyjmartens@hotmail.com .

 

I would be extremely grateful. I know the honor system is putting yourself out there isn’t real effective but it gives those of you who would like to say thank you and keep going with this a chance to do so.

So on to the download…. You have two options….

One if you are a person who already has the MySql.Data dll already installed on your system (make sure it is version 6.9.9.0 and you have at least .NET runtime v4.0.30319) you can download just the bin folder. If you don’t or aren’t sure, get the install package zip file called “MySqlMoveSetup”. If you need the .NET Framework you can download it here.

Both the bin folder zip file and the installer zip file are located here on my OneDrive.

Please do send me an email at kellyjmartens@hotmail.com with any bug reports, suggestions or praise. All are accepted.  

 

Have a great day!