Migrating existing live data into Crypteron

NOTE: Most users should now follow the documentation here to migrate existing data into Crypteron. The older post below documents a longer, cautious process suitable for certain high-risk scenarios (e.g. nuclear plant).

Today we’ll cover how to migrating existing data from a live application so that Crypteron now protects it. We’ll assume a SQL database here, so we’ll talk in terms of columns but if you’re using an object store, just think “property” when you read “column” below. The general idea is as follows:

1. Your authorized application always has access to it’s data (encrypted or not)
2. “Migration” merely involved your app writing data to the encrypted column. Behind the scenes, Crypteron handles the encryption and key management as necessary.

A simple example

Lets look at an example where a user’s credit card number must now to be encrypted. The logic is as simple as

1. Create a new column for the encrypted credit card number. If you had CreditCard previously, add another column Secure_CreditCard. This new column will have all NULLs to begin with.
2. When writing, write the credit card number to the encrypted column and set the clear-text column to NULL. You’ve just migrated this particular record.
3. When reading, read from the encrypted column. If the encrypted column is still NULL, your data hasn’t migrated yet from step #2 above, so use the clear-text column value.

That’s it! This is visually depicted as follows

This ensures that your application continues to stay online and work while seamlessly migrating over your most frequently used data first. There is no need to take your database or application offline just for migrating. The best part is that your database’s natural access patterns are preserved i.e. you won’t suddenly overload your database with a data-intensive migration job.

Bulk migration as a background job

You can accelerate the above “just-in-time” migration above by running a small job that explicitly migrates data. In C# with CipherDB for Entity Framework, it looks something like below. In Java, CipherDB for Hibernate is equivalently similar.

// Read => Write sample code
using (var secDbCtx = new CipherDbContext()) 
{
    int count = 0;
    foreach (var user in secDbCtx.Users.Where(u => u.Secure_CreditCard == null))
    {
        // migrate user table, sensitive fields
        user.Secure_CreditCard = user.CreditCard;
        user.CreditCard = null;
        // save in chunks for performance
        if ((++count % 100) == 0)
           secDbCtx.SaveChanges();
    }
    secDbCtx.SaveChanges(); // leftovers from last loop
}

Crypteron CipherDB automatically encrypts-on-writes and decrypts-on-reads, so it’s really that simple. If you’re using CipherObject, the only addition is that you’ll have to invoke Seal() before the writes and Unseal() after the reads – but the overall concept is the same.

If you’re migrating a very large number of records, it’s wise to limit work to batches by using something like secDbCtx.Users.Where(u => u.Secure_CreditCard == null).Take(5000) and adding an outer loop to march through your data set in such batches. For large data sets, we suggest performing the migration over the weekend or adding some sleep or throttling above to limit the impact of the migration job on your live production workload.

Post migration

If you’ve followed the above pattern, all clear-text values in that column will be NULL at the end. After you’ve confirmed this, you can safely remove the older clear-text column to clean up your database schema.

If you have any questions, please don’t hesitate to contact us through support.