Taking Advantage of Cloud Based Database Deployment
As many of you will know, one of my passions is semantic data integration. I have been developing the ShowcaseBio system to integrate some really interesting life science company, technology and people data. ShowcaseBio is now pretty large (over 40GB) for a local server bound system and as we are routinely doing more interesting text analytics and larger scale data mining, we are starting to need a bit more compute power than we have to hand. As I was toying with how to drop the bombshell to Dirk and Rowan that I wanted to spend some serious money on kit, I found what looks to me like the ideal solution.
Amazon has recently put into beta a scalable MySQL 5.1 implementation on their AWS cloud called AWS RDS. You can store 5GB-1TB data in a sliding scale of database instance classes ranging from 1-26 ECUs and 1.7-68GB RAM. You get charged for the up-time and data transfers in and out, but they look after the backups, snapshots and maintenance/updating. If you find that you are CPU or memory bound, you can simply jump up to the next instance class and pay a bit more. Sounds perfect I thought, so I took a peek at it. And once I’d got through the slightly difficult documentation and longwinded nomenclature, I must admit that I’m pretty impressed.
Here’s what I found and a practical guide to getting started with programmatic access to AWS RDS using C#.
How To Create, Write To and Query an AWS RDS Database (in C#)
The very first thing you will need to do is to sign up for the service. You will get in return a public and private key pair which you will need later for both the command line and C# tools. You will also need to install the AWS SDK and of course the MySQL dll if you haven’t already got this.
The first thing to say is that the example code provided on the AWS RDS site (at least for C#) is out of date and uses an SDK version that has been deprecated. It also deals mainly with examining the metadata around the service rather than how touse it in anger (hence this post).
The SDK docs are up-to-date though and installation of the AWS SDK in Visual Studio (VS2008) is a doddle. The next thing to stress is that it is very helpful to install the command line tools that Amazon provide. They will help you monitor what is going on and make doubly sure you don’t leave any test instances running (and accruing charges!). You are going to need two main commands:
rds-describe-db-instances --headers
// This displays the status of all current database instances
rds-delete-db-instance --db-instance-identifier testdbinstance --skip-final-snapshot
// This kills any stray testdbinstances that might be lying around
Once you have the command line up and running, you can build the little test application described below. It’s going to take you through the basics of setting up an RDS client service, setting up access via security groups, creating an AWS RDS database instance, getting the remote MySQL connection string, and then creating a table and writing to/reading from it.
For those who like to skip ahead, the code is here, otherwise here is the app in use:

AWS RDS test app opening screen

Waiting for the ASW RDS instance to become available

Created table on remote database instance

Writing and reading to remote database table on AWS RDS
This is the opening screen before connecting to AWS RDA and creating a database. If you hit the Open DB button, it will create an RDS client service with appropriate security access and create a database instance named ‘testdbinstance’ on the AWS RDS cloud. Given it is setting up a whole MySQL instance, this is not quick – it can take 3-4 minutes for the instance to be available and about the same for it to be deleted. As well as monitoring progress on the command line, you can track when the instance becomes available on the status bar of the app.
Once the remote instance is available you will see its name, remote server and port number in the status bar. You can then hit the Create Table button, which will create a test table on the remote database. You will see that a single row has appeared in the DataGridView. This is the first empty row that you get back from a SELECT * FROM table query when there is no data in it.
You can then have fun hitting the Insert New Row button which will INSERT INTO the test table to create new row entries in the test table on the remote database. Finally you can hit the Close DB button to delete the remote database instance (without saving a snapshot).
When you use it you will find that once you have the MySQL connection string, the system (unsurprisingly I suppose) behaves just like any other MySql implementation, so no code/schema changes and many of the headaches of managing performance are allevaited a little. You’ll find that it plays nicely with EC2 apps as well as an added bonus…
Have fun!
Important Reminder
At this stage it’s good to remember that AWS RDS is a commercial service and Amazon will of course charge you for any time that your db instance is up, so if you’re just trying it out, be sure to close it down either with the app or the command line interface before you finish!
C# Code
You can download a (partial) VS 2008 project here. You will still have to install the MySQL and AWS SDK components and supply your AWS account information.
The C# code is shown below. Apologies that the formatting has gone a bit awry, but if you stick it in VS2008 it will reformat properly. You will need 4 things to make it work (beyond the usual VS2008/.Net 3.5 environment), namely the usual MySql dll, your public and private keys for the AWS RDS service and one or more IP ranges (in CIDR-IP format) for which you wish to enable access to your test database. How to get the keys is covered in the AWS RDS documentation.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading;
using System.Windows.Forms;
using MySql.Data.MySqlClient;
using Amazon.RDS;
namespace AWS_RDS_Test
{
public partial class Form1 : Form
{
public string RDSConnection = "";
public string myPublicKey = ""; // INSERT YOUR PUBLIC KEY HERE //
public string myPrivateKey = ""; // INSERT YOUR PRIVATE KEY HERE //
public Amazon.RDS.Model.IPRange ip1 = new Amazon.RDS.Model.IPRange();
public Amazon.RDS.Model.IPRange ip2 = new Amazon.RDS.Model.IPRange();
public Amazon.RDS.AmazonRDSClient RDS;
public Amazon.RDS.Model.CreateDBInstanceResponse response;
public Amazon.RDS.Model.CreateDBInstanceResult result;
public Amazon.RDS.Model.DBInstance db;
public MySqlConnection connection;
public Form1()
{
InitializeComponent();
}
private void openDB_Click(object sender, EventArgs e)
{ // Insert IP addresses to be given access
ip1.Status = "authorized"; // using CIDR-IP definitions
ip2.Status = "authorized"; // of the form xxx.xxx.xxx.xxx/yy e.g. "192.168.0.1/30" (but external IP)
ip1.CIDRIP = ""; // INSERT YOUR IP ADDDRESS HERE //
ip2.CIDRIP = ""; // INSERT YOUR IP ADDRESS HERE //
createRDS(); // Create the RDS service using the public/private keys in the public variables
Amazon.RDS.Model.CreateDBInstanceRequest request = new Amazon.RDS.Model.CreateDBInstanceRequest();
request.DBName = "testdb"; // Name of the database to be created e.g. testdb
request.AllocatedStorage = 5; // Amount (GB) of storage required - INT between 5-1024
request.BackupRetentionPeriod = 0; // How long to keep backups (days), 0 = no backups
request.DBInstanceClass = "db.m1.small"; // How much CPU & memory to allocate - Valid values: db.m1.small | db.m1.large | db.m1.xlarge | db.m2.2xlarge | db.m2.4xlarge
request.DBInstanceIdentifier = "testdbinstance"; // The name of the db instance to be created
request.Engine = "MySQL5.1"; // The name of the database engine to be used - MySQL5.1 is only valid value
request.MasterUsername = "testadmin"; // The admin user name
request.MasterUserPassword = "testpassword"; // The password for the admin user (alphanumeric only)
createRDSDB(request); // Create the database instance on the RDS service
}
private void closeDB_Click(object sender, EventArgs e)
{
if (db != null) // Make sure there is actually something to delete
{
if (db.IsSetDBInstanceIdentifier())
{
deleteRDSDB(db, false);
RDS.Dispose();
}
}
}
private void createTable_Click(object sender, EventArgs e)
{
createRDSTable(db.DBName, "TestTable", "TestCol");
}
private void insertRow_Click(object sender, EventArgs e)
{
insertDBRow(db.DBName, "TestTable", "TestCol", "The time is " + DateTime.Now.ToLongTimeString());
}
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{ // Delete the testdb instance to avoid being charged when closing down
if (db != null) // Make sure there is actually something to delete
{
if (db.IsSetDBInstanceIdentifier())
{
deleteRDSDB(db, false);
RDS.Dispose();
}
}
}
private void createRDS()
{
Amazon.RDS.Model.DescribeDBSecurityGroupsResult secGroupsResult = new Amazon.RDS.Model.DescribeDBSecurityGroupsResult();
List<Amazon.RDS.Model.IPRange> ipRange = new List<Amazon.RDS.Model.IPRange>(); // Create the collection of allowable IP ranges
// Create a new RDS service client with appropriate IP address based security group so we can access it via code later
RDS = new Amazon.RDS.AmazonRDSClient(myPublicKey, myPrivateKey);
secGroupsResult = findSecurityGroups(myPublicKey + "_default"); // Delete any existing security groups
deleteSecurityGroups(secGroupsResult);
ipRange.Add(ip1); // Add the IP addresses to the IPRange List
ipRange.Add(ip2);
addSecurityGroup(myPublicKey + "_default", "Default access group", ipRange); // Add in the new security groups
// with a likely unique name
}
private Amazon.RDS.Model.DescribeDBSecurityGroupsResult findSecurityGroups(string groupName)
{
Amazon.RDS.Model.DescribeDBSecurityGroupsRequest secGroupsRequest = new Amazon.RDS.Model.DescribeDBSecurityGroupsRequest();
Amazon.RDS.Model.DescribeDBSecurityGroupsResponse secGroupsResponse = new Amazon.RDS.Model.DescribeDBSecurityGroupsResponse();
secGroupsRequest.DBSecurityGroupName = groupName; // Find all matching DBSecurity groups
try { secGroupsResponse = RDS.DescribeDBSecurityGroups(secGroupsRequest); }
catch (AmazonRDSException ex)
{
if (ex.ErrorCode != "DBSecurityGroupNotFound")
MessageBox.Show("Caught Exception: " + ex.Message + "\r\nResponse Status Code: " + ex.StatusCode + "\r\nError Code: " + ex.ErrorCode + "\r\nError Type: " + ex.ErrorType + "\r\nRequest ID: " + ex.RequestId + "\r\nXML: " + ex.XML);
}
if (secGroupsResponse.IsSetDescribeDBSecurityGroupsResult())
return (secGroupsResponse.DescribeDBSecurityGroupsResult); // and return them
else
return (null);
}
private void deleteSecurityGroups(Amazon.RDS.Model.DescribeDBSecurityGroupsResult groups)
{
if (groups != null)
{
Amazon.RDS.Model.DeleteDBSecurityGroupRequest secDeleteRequest = new Amazon.RDS.Model.DeleteDBSecurityGroupRequest();
Amazon.RDS.Model.DeleteDBSecurityGroupResponse secDeleteResponse = new Amazon.RDS.Model.DeleteDBSecurityGroupResponse();
foreach (Amazon.RDS.Model.DBSecurityGroup sg in groups.DBSecurityGroup) // Iterate over groups
{
secDeleteRequest.DBSecurityGroupName = sg.DBSecurityGroupName;
try { secDeleteResponse = RDS.DeleteDBSecurityGroup(secDeleteRequest); } // and delete them all
catch (AmazonRDSException ex)
{ MessageBox.Show("Caught Exception: " + ex.Message + "\r\nResponse Status Code: " + ex.StatusCode + "\r\nError Code: " + ex.ErrorCode + "\r\nError Type: " + ex.ErrorType + "\r\nRequest ID: " + ex.RequestId + "\r\nXML: " + ex.XML); }
}
}
}
private void addSecurityGroup(string groupname, string groupdescription, List<Amazon.RDS.Model.IPRange> ipRange)
{
Amazon.RDS.Model.CreateDBSecurityGroupRequest secRequest = new Amazon.RDS.Model.CreateDBSecurityGroupRequest();
Amazon.RDS.Model.CreateDBSecurityGroupResponse secResponse = new Amazon.RDS.Model.CreateDBSecurityGroupResponse();
Amazon.RDS.Model.CreateDBSecurityGroupResult secResult = new Amazon.RDS.Model.CreateDBSecurityGroupResult();
Amazon.RDS.Model.AuthorizeDBSecurityGroupIngressRequest authIngressRequest = new Amazon.RDS.Model.AuthorizeDBSecurityGroupIngressRequest();
Amazon.RDS.Model.AuthorizeDBSecurityGroupIngressResponse authIngressResponse = new Amazon.RDS.Model.AuthorizeDBSecurityGroupIngressResponse();
Amazon.RDS.Model.AuthorizeDBSecurityGroupIngressResult authIngressResult = new Amazon.RDS.Model.AuthorizeDBSecurityGroupIngressResult();
secRequest.DBSecurityGroupName = groupname; // Name for security group
secRequest.DBSecurityGroupDescription = groupdescription; // Simple description of security group
try { secResponse = RDS.CreateDBSecurityGroup(secRequest); } // Check we can create the security group
catch (AmazonRDSException ex)
{ MessageBox.Show("Caught Exception: " + ex.Message + "\r\nResponse Status Code: " + ex.StatusCode + "\r\nError Code: " + ex.ErrorCode + "\r\nError Type: " + ex.ErrorType + "\r\nRequest ID: " + ex.RequestId + "\r\nXML: " + ex.XML); }
if (secResponse.IsSetCreateDBSecurityGroupResult())
{ // Get the security group results
secResult = secResponse.CreateDBSecurityGroupResult;
secResult.DBSecurityGroup.IPRange.AddRange(ipRange); // Add the IP ranges to the security group properties
authIngressRequest.DBSecurityGroupName = secResult.DBSecurityGroup.DBSecurityGroupName;
foreach (Amazon.RDS.Model.IPRange ip in ipRange) // Allow access from each IP range
{
authIngressRequest.CIDRIP = ip.CIDRIP;
try { authIngressResponse = RDS.AuthorizeDBSecurityGroupIngress(authIngressRequest); }
catch (AmazonRDSException ex)
{ MessageBox.Show("Caught Exception: " + ex.Message + "\r\nResponse Status Code: " + ex.StatusCode + "\r\nError Code: " + ex.ErrorCode + "\r\nError Type: " + ex.ErrorType + "\r\nRequest ID: " + ex.RequestId + "\r\nXML: " + ex.XML); }
if (authIngressResponse.IsSetAuthorizeDBSecurityGroupIngressResult())
authIngressResult = authIngressResponse.AuthorizeDBSecurityGroupIngressResult;
}
}
}
private void createRDSDB(Amazon.RDS.Model.CreateDBInstanceRequest request)
{
bool makenew_dbinst = true;
Amazon.RDS.Model.DescribeDBInstancesRequest instRequest = new Amazon.RDS.Model.DescribeDBInstancesRequest();
Amazon.RDS.Model.DescribeDBInstancesResponse instResponse = new Amazon.RDS.Model.DescribeDBInstancesResponse();
Amazon.RDS.Model.DescribeDBInstancesResult instResult = new Amazon.RDS.Model.DescribeDBInstancesResult();
Amazon.RDS.Model.Endpoint e = new Amazon.RDS.Model.Endpoint();
instResponse = RDS.DescribeDBInstances(instRequest); // Get the response to the DescribeDBInstances
if (instResponse.IsSetDescribeDBInstancesResult()) // Check if we have descriptions of the DBInstances on RDS
{ // If we do, check to see if our instance already exists
instResult = instResponse.DescribeDBInstancesResult;
foreach (Amazon.RDS.Model.DBInstance i in instResult.DBInstance)
if (i.DBName.Equals(request.DBName)) // If it does, we don't need to recreate it
{ // So we can resue the existing instance by
db = i; // setting db to point at the existing DB instance
e = i.Endpoint; // Set e to the existing instance's endpoint
toolStripStatusLabel2.Text = "DB: " + db.DBName + " Server: " + e.Address + " Port: " + e.Port.ToString();
statusStrip1.Refresh();
makenew_dbinst = false; // Set the flag to avoid trying to recreate the existing instance
}
}
if (makenew_dbinst)
{
try
{
response = RDS.CreateDBInstance(request); // If it doesn't already exist we need to create it
if (response.IsSetCreateDBInstanceResult()) // Check that we are OK to create the instance
{
result = response.CreateDBInstanceResult; // Then create it
db = result.DBInstance; // and set db to be the DB instance
// It takes 3-4 mins for the instance to be available, so we wait
// until the endpoint information (address & port) is set
toolStripStatusLabel1.Text = "Please wait - creating DB " + db.DBName + " ";
toolStripProgressBar1.Visible = true;
do
{
toolStripProgressBar1.Value = (toolStripProgressBar1.Value + 9) % 100;
statusStrip1.Refresh();
Application.DoEvents();
Thread.Sleep(3000); // Check every 3 seconds to avoid throttling exception
e = IsAvailableRDS(instRequest, request); // Read the DBInstance data to pick up new Endpoint information
// This is convoluted as db.Endpoint does not appear to refresh
} while (e.Address == null);
db.Endpoint = e;
toolStripProgressBar1.Visible = false;
toolStripStatusLabel1.Text = "Database " + db.DBName + " created";
toolStripStatusLabel2.Text = "DB: " + db.DBName + " Server: " + e.Address + " Port: " + e.Port.ToString();
statusStrip1.Refresh();
}
}
catch (AmazonRDSException ex)
{ MessageBox.Show("Caught Exception: " + ex.Message + "\r\nResponse Status Code: " + ex.StatusCode + "\r\nError Code: " + ex.ErrorCode + "\r\nError Type: " + ex.ErrorType + "\r\nRequest ID: " + ex.RequestId + "\r\nXML: " + ex.XML); }
}
// Set the all important MySql connection string for the RDS DB instance
RDSConnection = "SERVER=" + e.Address + ";DATABASE=" + request.DBName + ";Connect Timeout=300;Port=" + e.Port.ToString() +
";UID=" + db.MasterUsername + ";PASSWORD=" + request.MasterUserPassword;
}
private Amazon.RDS.Model.Endpoint IsAvailableRDS(Amazon.RDS.Model.DescribeDBInstancesRequest instRequest, Amazon.RDS.Model.CreateDBInstanceRequest request)
{
Amazon.RDS.Model.Endpoint e = new Amazon.RDS.Model.Endpoint(); // Check whether the requested instance is available
Amazon.RDS.Model.DescribeDBInstancesResponse instResponse = RDS.DescribeDBInstances(instRequest);
if (instResponse.IsSetDescribeDBInstancesResult()) // Find the DBInstance and check if it is available
{
Amazon.RDS.Model.DescribeDBInstancesResult instResult = instResponse.DescribeDBInstancesResult;
foreach (Amazon.RDS.Model.DBInstance i in instResult.DBInstance) // Iterate over the instances
if (i.DBName.Equals(request.DBName)) // If we have the right one
if (i.DBInstanceStatus == "available") // and it's available
{ // return its details
e.Address = i.Endpoint.Address;
e.Port = i.Endpoint.Port;
}
}
return (e);
}
private void deleteRDSDB(Amazon.RDS.Model.DBInstance db, bool snapshot)
{
Amazon.RDS.Model.DeleteDBInstanceRequest delRequest = new Amazon.RDS.Model.DeleteDBInstanceRequest();
Amazon.RDS.Model.DeleteDBInstanceResponse delResponse = new Amazon.RDS.Model.DeleteDBInstanceResponse();
delRequest.DBInstanceIdentifier = db.DBInstanceIdentifier; // Find the right db instance to delete
delRequest.SkipFinalSnapshot = !snapshot; // Decide whether to write snapshot or not
if (snapshot)
delRequest.FinalDBSnapshotIdentifier = db.DBName;
delResponse = RDS.DeleteDBInstance(delRequest);
if (delResponse.IsSetDeleteDBInstanceResult()) // See whether it is OK to delete instance
try // If so delete it
{
Amazon.RDS.Model.DeleteDBInstanceResult delResult = delResponse.DeleteDBInstanceResult;
toolStripStatusLabel2.Text = "DB: - Server: - Port: -";
statusStrip1.Refresh();
}
catch (AmazonRDSException ex)
{ MessageBox.Show("Caught Exception: " + ex.Message + "\r\nResponse Status Code: " + ex.StatusCode + "\r\nError Code: " + ex.ErrorCode + "\r\nError Type: " + ex.ErrorType + "\r\nRequest ID: " + ex.RequestId + "\r\nXML: " + ex.XML); }
}
private void createRDSTable(string database, string tablename, string column)
{
openRDSDB();
MySqlCommand command = connection.CreateCommand();
command.CommandText = "CREATE TABLE " + database + "." + tablename + " (" + column + " varchar(255) NOT NULL);";
try { command.ExecuteNonQuery(); }
catch { MessageBox.Show("Table already exists (or other error): " + command.CommandText); }
closeRDSDB();
updateDataGrid(database, tablename);
}
private void insertDBRow(string database, string tablename, string column, string value)
{
openRDSDB();
MySqlCommand command = connection.CreateCommand();
command.CommandText = "INSERT INTO " + database + "." + tablename + " VALUES ('" + value + "')";
try { command.ExecuteNonQuery(); }
catch { MessageBox.Show("Cannot INSERT: " + command.CommandText); }
closeRDSDB();
updateDataGrid(database, tablename);
}
private void updateDataGrid(string database, string tablename)
{
openRDSDB();
// Populate datagrid with data from the RDS database
MySqlDataAdapter MySqlDA = new MySqlDataAdapter("SELECT * FROM " + database + "." + tablename, RDSConnection);
DataTable table = new DataTable(); // Populate a new data table from the db
table.Locale = System.Globalization.CultureInfo.InvariantCulture;
MySqlDA.Fill(table);
dataGridView1.DataSource = table; // and bind the DataGrid to it
dataGridView1.Columns[0].HeaderText = "Value";
dataGridView1.Columns[0].AutoSizeMode = DataGridViewAutoSizeColumnMode.Fill;
closeRDSDB();
}
private void openRDSDB()
{
connection = new MySqlConnection(RDSConnection);
try { connection.Open(); }
catch
{
MessageBox.Show("Cannot open connection to: " + RDSConnection);
Close();
}
}
private void closeRDSDB()
{
if (connection != null)
connection.Close();
}
}
}
// Define some allowable IP ranges