Posts Tagged migration

MySQLMove – An Easy To Use Tool To Move MySQL Database Objects and Data

Recently, I was asked to work on a project using a MySQL backend after many years of not having worked with the product. I was amazed when I saw that some of the more common tools to work with MySQL, such as MySQL Workbench or Toad, had still not developed an easy way to move objects and/or data between mySQL databases such as we have become accustomed to in SQL Server Management Studio.  Obviously, this greatly complicated my work and caused me to have to work with data in a production environment more often, which obviously I wasn’t comfortable with.

So like most developers, my brain began to think about how to solve this problem. You could of course, go through each object, get the CREATE script, put it in a giant file, and run it that way. That would be time consuming, bulky and difficult to maintain should there be changes to the database objects. And of course, this did nothing for the issue of data migration.

I looked at other products. MySQLDump, a free tool is fine. Of course, it was more complicated then I would have liked. I just want to move the objects and data and not have a high learning curve of a new product. Just get it done already. Others were available that you had to pay for. Being the cheap bastard that I am, that option didn’t appeal to me either.

So I wrote MySQLMove. This started out as a quick and dirty option to get objects and data moved to another database on another server and gradually moved to including things like a report, the option to only migrate objects, and actually put a user interface on top instead of just running a script. The script was fine for me but it may not be for other developers.

Let’s go into some things you need to know or generally be aware of.

First, some MySQL developers might wonder why I did not use INFILE to import data. The reason is for many such developers, they are coding against a shared hosting environment and often the administrators in such environment have disabled this option. Even using the LOCAL keyword presented problems relating to security. However, in future releases (but see below) I am planning on a interface modification that will allow the end user to indicate that INFILE is available for use and it will operate accordingly.

UPDATE:

The engine now uses INFILE and LOCAL keywords. It is also now lightening fast!

Please make sure you have created the database on the destination server. I know that comes from the “duh” department but you would be surprised…. If you are asking yourself the reason why I don’t do it for you, it is because I drop each object on the destination individually as it is being imported. But more importantly, often each database has a specific username and password designated for access. An idea submitted for future versions (but see below) is that we collect all databases under a given username and password and allow multiple databases to be migrated at one time.  That day is not here yet but it intrigues me.

Another “duh” is it is important to make sure the destination database doesn’t have any users writing changes.

Next…. it is important that if you are importing from or to a Linux based environment, that you are absolutely certain of the case of your database, its objects and server name. Windows is much more forgiving then Linux is in this regard. If you are importing from Linux, be prepared for windows to make all objects lower case, regardless of how you created them in Linux based servers. I have also seen some versions of MySQL run on Windows that were also sticklers for such. My best advice to you is to make all objects lower case. Both Linux and Windows based MySQL servers can handle such and if you are often moving between the two systems you will save yourself a lot of trouble.

Next, you will see an option in a checkbox to only import MySQL objects.

image

If you have the time and the data available to you in text or csv format, I suggest you check this. Why? While this engine can and will move data, it takes longer then a manual import would. The way it is built is all of your insert statements for the data are collected as one statement and inserted collectively. It was about 20 percent faster then inserting a row at a time and also if it should bomb, I can rollback the transaction so that you are not left with a table with partial data. That said, if you are one of those folks who build 300+ column tables, (by the way who hurt you as a child to do that to yourself? Smile jk) it is going to take awhile. But rest assured, it will get there. This was tested against a table with 80000 rows and 30 columns. So it is pretty robust.

UPDATE:

As mentioned above I now use INFILE and LOCAL keyword. The above text is no longer a concern.

Next, it is important you let the code finish. Check that, it is imperative. If you are impatient, switch to decaf and try some breathing exercises.

Finally, MySQLMove will attempt to export database objects as follows: tables, stored procedures, functions and triggers. Not every single one of all those objects can be migrated using this tool. One issue that has commonly arisen is when the code uses a PREPARE statement along with a string for use as the command. Future versions (but see below) will probably rectify this. Should something not be exportable to the destination, at the end of the process a report will appear showing what objects weren’t able to be moved. In my testing, it is a rare occurrence. Also with triggers, remember the “defined user” is copied straight from the trigger to the destination. You will need to make sure that user exists on the destination database as well. Oh and by the way, triggers are migrated last. Otherwise each insert might cause that trigger to fire. And yeah that would not be fun.

Finally, I would love to continue developing this tool. There are so many things I could and would love to do with it. Better reports, table optimization, code optimization, object selection etc. Unfortunately that costs me time, which in translation for those that live under a rock, this means it costs me money. If you find the tool useful and would like to continue to see it developed, please don’t be a cheap bastard like me Smile and make a donation here using my email address – kellyjmartens@hotmail.com .

 

paypal-app

I would be extremely grateful. I know the honor system is putting yourself out there isn’t real effective but it gives those of you who would like to say thank you and keep going with this a chance to do so.

So on to the download…. You have two options….

One if you are a person who already has the MySql.Data dll already installed on your system (make sure it is version 6.9.9.0 and you have at least .NET runtime v4.0.30319) you can download just the bin folder. If you don’t or aren’t sure, get the install package zip file called “MySqlMoveSetup”. If you need the .NET Framework you can download it here.

Both the bin folder zip file and the installer zip file are located here on my OneDrive.

Please do send me an email at kellyjmartens@hotmail.com with any bug reports, suggestions or praise. All are accepted. Smile 

 

Have a great day!

, , , ,

2 Comments

C# Application linked to SharePoint List Web Service

This weekend found myself knee deep in a crisis that my friend who had migrated from SharePoint 2003 to 2007 (there were reasons he couldn’t go to 2010). Simply put the migration from SharePoint 2003 to 2007 had broken his application (tracking program that submitted data to a 2003 SharePoint List) because in SharePoint 2007  you can’t do this while not on the actual server if you have “Web Page Security Validation” enabled. So for the code below you have to have SPWeb.AllowUnsafeUpdates = true; . Obviously not the ideal solution for my friend but we didn’t have time to screw around. Here is what I did….

 

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Collections;
using System.Xml;
using System.ServiceModel;
using System.Net;
 
namespace Trigger_Tracker
{
    public partial class Form1 : Form
    {//form move on click and drag
        bool FormMoving;
        Point initialPoint;
        TriggerTrackerPictureBox frmPicture;
 
        public Form1()
        {//form move on click and drag
            InitializeComponent();
            comboBox1.SelectedIndex = 0;
            FormMoving = false;
 
            frmPicture = new TriggerTrackerPictureBox();
            frmPicture.localForm = this;
            frmPicture.Owner = this;
            frmPicture.Show();
            frmPicture.Width = 68;
            frmPicture.Height = 65;
            SetPositionOfPictureForm();
        }
 
        private void SetPositionOfPictureForm()
        {
            frmPicture.Top = this.Top + 26;
            frmPicture.Left = this.Left + 87;
        }
 
       
 
        private void TrackerButton(object sender, EventArgs e)
        {
            string listGUID = “A93E1A7E-67D0-4D7D-A4ED-803D7DFE684B”;
            string viewGUID = “6B2F3EF2-4B0C-41E1-B87E-0C3185B587DD”;
            //string viewGUID2 = “6B2F3EF2-4B0C-41E1-B87E-0C3185B587DD”;
 
            int ItemCounter = 1;
            ServiceList.Lists listService = new ServiceList.Lists();
           // RetentionLists.ListsSoapClient listService = new RetentionLists.ListsSoapClient();
 
            //////
           listService.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
           //listService.ChannelFactory.Credentials.Windows.ClientCredential   = System.Net.CredentialCache.DefaultNetworkCredentials;
 
            XmlNode activeItemData = listService.GetListItems(listGUID, viewGUID, null, null, “100”, null);
            XmlDocument xDoc = new XmlDocument();
            string tmpString = activeItemData.InnerXml.Replace(“\r\r”, “”);
            xDoc.LoadXml(tmpString);
            XmlNamespaceManager nsManager = new XmlNamespaceManager(xDoc.NameTable);
            nsManager.AddNamespace(“z”, “#RowsetSchema”);
            nsManager.AddNamespace(“rs”, “urn:schemas-microsoft-com:rowset”);
 
            XmlNodeList xNode = xDoc.SelectNodes(“/rs:data/z:row”, nsManager);
 
            foreach (XmlNode tmpNode in xNode)
                ItemCounter++;
 
            StringBuilder strBuilder = new StringBuilder();
            strBuilder.Append(“<Method ID='” + ItemCounter + “‘ Cmd=’New’>”);
            strBuilder.Append(“<Field Name=’Attachments’>” + “0” + “</Field>”);
            strBuilder.Append(“<Field Name=’Title’>” + PolicyNumber.Text + “</Field>”);
            strBuilder.Append(“<Field Name=’Reason’>” + comboBox1.Text + “</Field>”);
            strBuilder.Append(“</Method>”);
 
            string strBatch = strBuilder.ToString();
 
            XmlDocument newDoc = new XmlDocument();
            XmlElement newElement = newDoc.CreateElement(“Batch”);
            newElement.SetAttribute(“OnError”, “Continue”);
            newElement.SetAttribute(“ViewName”, viewGUID);
            newElement.InnerXml = strBatch;
           
            XmlNode returnNode = listService.UpdateListItems(listGUID, newElement);
 
            this.comboBox1.Text = “Please Select….”;
            this.PolicyNumber.Text = “”;
            this.PolicyNumber.Mask = “0000000000”;
 
            comboBox1.Focus();
        }
 
       
 
 
        public void Form1_MouseUp(object sender, MouseEventArgs e)
        {//form move on click and drag
            FormMoving = false;
        }
 
        public void Form1_MouseMove(object sender, MouseEventArgs e)
        {//form move on click and drag
            if (FormMoving)
            {
                if ((Left + e.X – initialPoint.X) <= 0)
                    Left = 0;
                else if ((Right + e.X – initialPoint.X) >= Screen.PrimaryScreen.Bounds.Right)
                    Left = Screen.PrimaryScreen.Bounds.Right – Width;
                else
                    Left = Left + e.X – initialPoint.X;
                if ((Top + e.Y – initialPoint.Y) <= 0)
                    Top = 0;
                else if ((Bottom + e.Y – initialPoint.Y) >= Screen.PrimaryScreen.Bounds.Bottom)
                    Top = Screen.PrimaryScreen.Bounds.Bottom – Height;
                else
                    Top = Top + e.Y – initialPoint.Y;
            }
            SetPositionOfPictureForm();
        }
 
        public void Form1_MouseDown(object sender, MouseEventArgs e)
        {//form move on click and drag
            FormMoving = true;
            initialPoint = new Point(e.X, e.Y);
        }
 
        public void pictureBox1_MouseDown(object sender, MouseEventArgs e)
        {//form move on click and drag
            FormMoving = true;
            initialPoint = new Point(e.X, e.Y);
        }
 
        private void pictureBox1_MouseMove(object sender, MouseEventArgs e)
        {
        }
 
        private void pictureBox1_MouseHover(object sender, EventArgs e)
        {
        }
 
        public void label1_MouseHover(object sender, EventArgs e)
        {
            FormMoving = false;
        }
 
        public void label1_MouseUp(object sender, MouseEventArgs e)
        {
            FormMoving = false;
        }
 
       
 
        public void Form1_MouseHover(object sender, EventArgs e)
        {
            this.Opacity = 1;
        }
 
        public void Form1_MouseLeave(object sender, EventArgs e)
        {
            if(!PolicyNumber.Focused)
                this.Opacity = .25;
        }
 
        public void pictureBox1_MouseLeave(object sender, EventArgs e)
        {
            this.Opacity = 1;
        }
 
        public void pictureBox1_MouseHover_1(object sender, EventArgs e)
        {
            this.Opacity = 1;
        }
 
        public void Form1_Click(object sender, EventArgs e)
        {
            comboBox1.Focus();
        }
    }
}

, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Leave a comment