How my C# Object.Extend implementation works

I’ve had a few people ask me to explain how my C# Object.Extend implementation works – so here I am, going into more detail than I expected people would be interested in.

Making Object.Extend work seemed straight-forward at first. You can read my initial code in my original ObjectExtend post. The problem started when I tried to chain two calls together: I immediately got a RuntimeBinderException. Unfortunately, for a really good reason (the difficulty of discovering the right namespaces to search), the code responsible for runtime method binding doesn’t look for extension methods. That means that when the first call to .Extend returns something marked as dynamic (regardless of what actual type it returns), you can’t use any extension methods on the result. Attempting to call either .Extend again or .Dump (another common extension method in LINQPad) results in the dreaded RuntimeBinderException.

The usual usage of the dynamic keyword is to let us return something like the ExpandoObject type, like this:

dynamic ReturnSomething() {
	return new ExpandoObject();
}

Fortunately, the dynamic keyword in C# essentially just tells the compiler to engage duck-typing mode – it doesn’t actually tell the run-time anything. So there’s nothing stopping us from doing this:

dynamic ReturnSomething() {
	return new List<string>();
}

The compiler (and your IDE) will allow you to try to call anything at all on the return value of ReturnSomething, and hand everything off to the runtime binder. This means that you can’t use extension methods on anything returned by a method marked dynamic, even if you actually return a regular non-dynamic type.

dynamic ReturnSomething() {
	return new object();
}
ReturnSomething().Dump();

Nope. Instant RuntimeBinderException.

There’s one situation which will give us exactly what we want – if we mark the .Extend function as dynamic, but the type we return has our Extend method built right in to it. This means our IDE (LINQPad, in my case) will allow us to access any of the dynamic properties we build up on our types over successive calls to Extend, and the runtime binder will be able to find the methods because they’re right on the type we return – we’re not relying on extension methods!

I initially thought I could create a subclass of ExpandoObject with calls for Extend and Dump, but it turns out that not only is ExpandoObject a sealed class, it’s also a bit special in other ways – so that was a no-go.

So now the only problem we have is that we need to create types on the fly which contain functions for Extend (and Dump, for convenience), and also whatever properties we want to dynamically add – the union of all of the properties on the original object and the objects we want to extend it with. I looked into a few alternatives, and a good compromise was the Microsoft.CSharp.CSharpCodeProvider – it allows me to dynamically assemble a syntax tree and get the framework to generate in-memory assemblies on-the-fly. The details are a little tedious, but it’s very possible to use this to create types on the fly – containing both the method calls we want for Dump and Extend as well as all of the properties we need. We can then instantiate our dynamically-created type and copy all of the values onto it. Our IDE will let us access our runtime-created properties and methods without compile-time errors, because our function is marked as dynamic – and the runtime binder can find all of the properties, as well as our .Extend and .Dump methods, because they’re actually on the type the runtime binder is looking at.

The minimum viable code to do something useful with CSharpCodeProvider looks something like this (note that this requires a couple of helpers you can find in the class linked below):

var compileUnit = new CodeCompileUnit();
var nameSpace = new CodeNamespace("MyDynamicNamespace");
var classType = new CodeTypeDeclaration("MyNewType");
classType.Attributes = MemberAttributes.Public;
compileUnit.Namespaces.Add(nameSpace);
nameSpace.Types.Add(classType);
var returnThree = new CodeMemberMethod { 
	Name = "ReturnThree", 
	Attributes = MemberAttributes.Public,
	ReturnType = new CodeTypeReference(typeof(int))
};
returnThree.Statements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(3)));
classType.Members.Add(returnThree);
var result = _cSharpCodeProvider.Value.CompileAssemblyFromDom(_compilerParameters.Value, compileUnit);
var compiledType = result.CompiledAssembly.GetTypes().Single();
LINQPad.Extensions.Dump(((dynamic)Activator.CreateInstance(compiledType)).ReturnThree());

This is a very round-about way to output the number 3, but it works!
PrintThree

I won’t reproduce the code here, but you can find all the details of the Object.Extend dynamic type creation in the CreateTypeUncached function in ObjectExtend.cs.

You might notice the word ‘Uncached’ there. When I first tried this approach, it was horrifyingly slow – I was using Object.Extend in a Select statement against a large IEnumerable, and generating many identical types. Throwing a quick cache into the mix based on the name and type of all properties of the type we need vastly reduces the number of calls to the compiler service and brings performance up to a tolerable level.

While I have glossed over some details, hopefully this explanation will give readers some background information to aid in reading the code. Please feel free to reach out to me on Twitter and let me know if parts of my explanation are hard to follow.

Object.Extend in C# for exploratory coding

LINQPad is great for exploratory coding. I use it all the time while I’m poking at APIs, and it’s completely replaced other scripting languages for me. I often find myself gradually building up result sets as I grope my way towards the result I’m looking for – and then I go back and re-factor it into something more presentable.

Unfortunately, building up these result sets can mean copying all the members of an old dynamic object into a new dynamic object.

var sites = siteData.Select(s => new { SiteName = s[0], SiteLink = s[1] });
var siteInfo = sites.Select(s => new {
    SiteName = s.SiteName, SiteLink = s.SiteLink, SiteInfo = SiteInformation[s.SiteName]
});
var siteContent = siteInfo.Select(s => {
    var details = GenerateDetailsSomehow(s);
    return new {
        SiteName = s.SiteName,
        SiteInfo = s.SiteInfo,
        SiteLink = s.SiteLink,
        SiteDetails = details
    }
});
// ... more of the same

That gets tedious fast. Wouldn’t it be great if C# had something similar to JavaScript’s Object.Extend? Well, maybe it can. I jumped into the “My Extensions” file in LINQPad and put together the following extension method:

public static dynamic Extend(this object firstObj, params object[] objs) {
    var result = new ExpandoObject();
    foreach (var o in new[] { firstObj }.Union(objs)) {
        foreach (var p in o.GetType().GetProperties().Select(p => new { Name = p.Name, Value = p.GetValue(o) })) {
            var props = ((IDictionary<string, object>)result);
            if (props.ContainsKey(p.Name)) props[p.Name] = p.Value;
            else props.Add(p.Name, p.Value);
        }
    }
    return result;
}

Now you can just call .Extend(...) on any object! So instead of having to create new objects all the time, you can do this:

var sites = siteData.Select(s => new { SiteName = s[0], SiteLink = s[1] });
var siteInfo = sites.Select(s => s.Extend(new {SiteInfo = SiteInformation[s.SiteName]}));
var siteContent = siteInfo.Select(s =>s.Extend(new { SiteDetails = GenerateDetailsSomehow(s) }));

That’s much easier to read (and quicker to write) than the first snippet! Unfortunately, it doesn’t work – the first call to our object.Extend(...) extension method is just fine, but the second call fails. Sadly, the way the runtime binder works means that our extension method won’t be available on the dynamics we create, so we can’t chain multiple calls using this approach.
RuntimeBinderException

I have solved that (and a number of other minor annoyances) and put it all together in a Nuget package called ObjectExtend. Sadly, if you don’t have a license for LINQPad you may have to download the package and reference it manually, but if you do have a license you can use the Nuget client built right in to LINQPad.

After adding ObjectExtend to our script, the chained script above works as expected:
ObjectExtendScreenshot

There you have it! Object.Extend in C#.

Please note this is a package focused on tinkering in LINQPad. It’s not the kind of thing you should be using while building maintainable, production-quality software.

Update: I had a few requests for source code, and it’s up on github now, but rather than making people who want to understand it dig through code I wrote up an explanation.

Measure (At Least) Twice

20180218_143538 - SmallBear with me here. I promise this is about software – but first I need to talk about building.

I have several uncles and uncles-in-law who are builders, and when I was younger I ended up helping out with minor jobs here and there. One of the most recurrent mantras was this one:

“Measure Twice, Cut Once.”

The idea, if it’s not immediately obvious, is that you double-check your measurements before committing to an action which is costly or hard to undo. Once you’ve cut that beam or board, you can’t un-cut it – the board will stay cut despite your best efforts to turn back time. If you cut it too small, you can no longer use it for what you planned to. If you cut it too big, you have to try again. Measuring is cheap – cutting is expensive. Measure twice, cut once.

I consistently see software engineers fail to even measure once.

I once reviewed a pull request from a developer who had decided to optimise part of our code base which performed intensive weather modelling. He had spent a week on it – and he had spent that time re-writing a large number of string plus operations to use StringBuilders. In some cases this can provide some performance benefit – but the re-written code was in the setup and teardown phases of the modelling. Setup and teardown took only seconds, while the modelling itself could take minutes or hours. Worse, this developer had introduced several bugs into previously-working code; but that would make this an anecdote for a blog post about the importance of testing and safe refactoring, and this is about measuring, so let’s focus on a different “worse”. Not only had he optimised the wrong part of the code base, but because he hadn’t measured performance, he couldn’t demonstrate that he’d actually made anything faster (aside from some vague assertions that StringBuilder is faster than string-plus, which isn’t always true).

I didn’t even need to measure to know that his change wasn’t going to make things significantly faster – the code wasn’t in a loop, and I knew that the overall modelling time was long. Shaving off microseconds at a time in a scenario like that is a losing game. The other reason I knew for certain that his change wasn’t going to pay dividends was that I had already spent quite some time improving the efficiency of that setup process. We had benchmark tests around parts of that code, and we had made significant wins. Most of them were had by fixing the .GetHashCode and .Equals methods of a custom coordinate class we were using. We were wrangling a fairly large amount of data into data structures needed by the modelling code, and set operations like .Contains were taking far too long. It turns out both .GetHashCode and .Equals were creating additional instances of the class to perform certain comparisons, and so we were churning the garbage collector as well as performing far too many operations. Some careful tweaks to those methods – operations performed many times within nested loops – took our setup time down from minutes to seconds. We knew that we’d made things better because we were measuring – but the benefits of measurement go much deeper than just knowing how you did.

All developers fall somewhere on the spectrum between, roughly, terrible and genius. I’m not sure where on that spectrum I fall (I suspect it varies depending on what sort of programming I’m doing), but I am pretty sure that I wasn’t good enough to read through thousands of lines of C# across many different files and realise, through pure mental analysis, that we had a performance problem in our coordinate comparison code. Even if I had noticed that it wasn’t efficient, I wouldn’t have been sure how much the problem was contributing to our run time (as compared to, say, our inefficient string concatenation). Instead, I found out by measuring.

In this case, I used one of the simplest of all code profiling techniques: the pause button in my debugger. Simply run your code, and sometime during the long-running process, hit pause. The line you’ve broken on is extremely likely to be part of the problem. Typically, it will fall into one of two categories:
1. A line which is executing many times. In this situation, something further up the stack trace is calling the code you’re looking at inside a large (or nested) loop.
2. A line which is taking a long time to execute. Perhaps it’s hitting disk or network and waiting a long time before continuing.
There is a third (statisically less-likely) option: you’ve just happened to break execution somewhere that isn’t part of the performance problem. That’s why you measure (at least) twice. Break your long-running operation a few times, and compare the stack traces. If your stack dump consistently looks the same, you’ve found your performance hot-spot.

That’s one of the simplest ways of measuring your code, and it may suffice for tracking down hot-spots, but there is a lot more to measurement than just hitting pause in your debugger. Do you suspect you have a database problem? Don’t go hunting through code looking for your favourite anti-patterns. Instead, measure the number of database round-trips your operation makes. Use your database tools to identify expensive queries – and even once you’ve found them, don’t necessarily just start staring at code. Are you missing an index? There are tools which can tell you. Even if the query is bad, looking at the query plan will tell you exactly which parts of the query are causing the problem, so you don’t need to guess.

Maybe you are hitting multiple services and combining the results? If you’ve just learned about the async keyword, you might be tempted to dive in and start making everything async – but if almost all of the delay is coming from a single call, it might not help much (you’d be better trying to find out why that service is so slow). If you really are dealing with multiple similar-cost calls which you combine after they’re all complete, async may be a good investment of your time – but you won’t know until you’ve measured.

For one weather-forecasting algorithm I measured, I didn’t turn up anything obvious. I measured and measured some more, and eventually I discovered we were utilising the CPU cache poorly – the bulk of our algorithm time was spent fetching data from main memory after cache misses. Updating the algorithm was a very costly measure, but investing in hardware with better memory bandwidth gave us some short-term wins. There are very few developers who can look at an algorithm and spot a cache utilisation problem, but measurement can find it for you.

I think I understand why some developers prefer to dive right into code, instead of measuring the problem. We like to think we’re smart. We know stuff about coding, and we’re good at it, and so we can jump in and spot what the original developer didn’t. We can use the sheer power of our minds to solve the problem. It’s a little like the builder who drives in nails with his forehead. It might be impressive, but it’s not very effective, and you’ll end up with a headache.

The opposite problem happens too: we make the mistake of not valuing our time. “What can it hurt to spend a week re-writing those string-plus operations as StringBuilder operations,” we ask ourselves. It makes our code better (by one measure, I suppose), and it won’t hurt performance. But if we spend a week of our time, was it really worth the investment? Measure up front, so you can make that decision based on facts instead of guesses.

Perhaps some developers just feel like they have great intuition. One of my uncles was actually pretty good at estimation: he’d cut timber a shade too big, and trim it down as he was placing it. He only did that occasionally, though. A lifetime of building told him when it made sense to trust his estimation skills, and when to measure twice, cut once.

So we’re back to builders, and here’s the deep lesson. If a builder will measure twice to avoid wasting a bit of cheap wood, how much more should we measure to avoid wasting precious time? I’ve seen weeks wasted on the wrong problem, simply because someone didn’t bother to measure.

Please, my fellow developers: measure (at least) twice.

Hooking (Hacking?) the ASP.Net FileChangeNotifier

I’m going to tell you a little bit about a package I’ve just put together which lets you exclude certain files from the ASP.Net change notifier (the thing which restarts your IIS app pool whenever you change certain files). On the way, we’re going to dig into some internal framework code, and I’m going to horrify you with some code that should definitely never get run in production! Let’s start with the inspiration for this work.

A Bad Dev Experience

The dev experience on one of our legacy projects had been annoying me for a while: whenever I triggered a Gulp build, my IIS app pool would restart. That sometimes had (annoying) flow-on effects, and we ended up with various work-arounds depending on what we were doing. It was never a priority to fix, because Gulp builds only happened on dev and CI machines, never in deployed environments – but one week it really got in the way of some work I was doing, so I decided to get to the bottom of the problem.

You may know that ASP.Net will restart your application whenever there are changes to your web.config or any files in your /bin folder. What you might not know (I didn’t) is that it can also monitor various other files and folders in your application, and what it monitors doesn’t seem to be well-documented. Our Gulp build dumps the output into a /dist folder, and ASP.Net was restarting our app every time any file in that folder changed. After a little Googling, I discovered a StackOverflow answer which gave a slightly-hacky way to disable the ASP.Net file monitoring, which I did – and it worked. No more app restarts!

Not So Fast…

After a few hours of front-end work, I needed to make some back-end changes. I finished wiring up my API endpoints, ran the project, and expected success! No luck. After some more tinkering with the front-end, I suspected my C# code – but after some checking, it was definitely correct: I had the unit tests to prove it. Eventually I fired up the debugger, and found my problem: the debugger was reporting that the running code didn’t match my file. Disabling file change notifications had also disabled notifications on the /bin folder, so my rebuilt DLL files weren’t being loaded. I wasn’t prepared to accept rebuilds not triggering reloads, so I needed to revisit my solution.

Before I went too far, I wanted to be really sure that I was looking at the right problem. It turns out someone going by the name Shazwazza has already done the hard work to build a list of files and folders ASP.Net is helpfully monitoring for changes, and yes – there is my Gulp output folder.
FileChangeNotifierFileList

What I really want is to be able to preserve all of the existing functionality – but skip monitoring for certain paths. I investigated a few different switches and options, but none of them gave me exactly what I wanted. I was going to have to get my hands dirty.

Diving Into Framework Code

Having read Shazwazza’s code, I had already learned that the file monitor is stored on the HttpRuntime singleton, so I started my investigation there. I was hoping it just stored a reference to an interface, so I could just replace it with my own wrapper over the base implementation.
FileChangeNotifierReference

Well, no interface – but I suppose I can just subclass FileChangesMonitor and use reflection to swap the base implementation out for my own.
FileChangeNotifierSealed
Oh. It’s a sealed class – so I can’t even subclass it. It was about this point that I stopped trying to solve the problem on work time: this had become a personal mission, and I wasn’t going to give up until I’d made this thing behave the way I wanted to.

First, I needed to get a solid understanding of how the FileChangesMonitor class worked. After a lot of reading, it boils down to this: other classes ask the FileChangesMonitor to monitor a particular path, and it maintains a hash of paths to DirectoryMonitor instances which it creates. In turn, DirectoryMonitor has a hash of filenames to FileMonitor objects it has created, and FileMonitor has a HybridDictionary which stores callbacks. FileChangesMonitor, DirectoryMonitor, and FileMonitor are all sealed classes, so it looks like I’m out of luck – but I refused to be beaten!

The core classes I’m dealing with are all sealed, so there’s definitely no option to change behaviour there – but at each step of the object hierarchy, a parent object is storing instances of these sealed classes in plain old collection types: Hashtables and HybridDictionaries. Those classes aren’t sealed, so that’s going to have to be the seam where I stick my metaphorical crowbar.

A Word of Warning

If the idea of messing with private-member Hashtables makes you nervous, good. I’m about to go messing with the internals of framework classes in a very unsupported, highly-fragile way. This is not production-quality code. This is nasty, hacky code which is probably going to break in the future. If you ever use this code, or this extension technique, please do it in a way which will never, ever, ever run on a production machine.

I’m trying to fix the dev experience here, and so I decided to accept the hacky nature – but I have put plenty of guards around it to keep it clear of our production machines. I’ve even made sure it stays clear of our test environments. It’s a local-dev-environment only thing!

A Note on Reflection

If you’re familiar with the .Net reflection API, you’ll know it can be a little clunky. I’m using the following extension methods to make my code look a little cleaner.

public static class ReflectionHelpers
{
  public static FieldInfo Field(this Type type, string name) => type.GetField(name, BindingFlags.NonPublic | BindingFlags.Instance);
  public static FieldInfo Field<T>(string name) => typeof(T).GetField(name, BindingFlags.NonPublic | BindingFlags.Instance);
  public static FieldInfo StaticField<T>(string name) => typeof(T).GetField(name, BindingFlags.NonPublic | BindingFlags.Static);

  public static ConstructorInfo Constructor(this Type type, params Type[] parameters)
    => type.GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, parameters, null);

  public static MethodInfo StaticMethod(this Type type, string name, params Type[] parameters)
    => type.GetMethod(name, BindingFlags.Static | BindingFlags.NonPublic, null, parameters, null);

  public static Type WebType(string name) => typeof(HttpRuntime).Assembly.GetType(name);
  public static Object Make(this ConstructorInfo constructor, params Object[] parameters) => constructor.Invoke(parameters);
  public static Object Call(this MethodInfo method, params Object[] parameters) => method.Invoke(null, parameters);
}

How I Hooked (Hacked) FileChangesMonitor

What I really want to do is to prevent FileMonitor from setting up file watches for specific paths, and my only extension point is the HybridDictionary of watches. When FileMonitor is asked to set up a watch, it checks its dictionary to see if there’s already one there for that specific callback. If it doesn’t already have one, it sets one up. So what I need is a HybridDictionary which always has an entry for any given callback!

Sadly, HybridDictionary doesn’t declare its methods as virtual, so we can’t just go overriding them. Fortunately, it does just use a Hashtable under the covers – and we can use some reflection trickery in the constructor to replace the Hashtable it creates with one of our own design!

/// <summary>
/// Intercepting hybrid dictionary. It always responds to any key request with an actual value
///  - if it didn't exist, it creates it using a FileMonitor factory.
/// HybridDictionary always uses a hashtable if one has been constructed, so we just rely on InterceptingHashTable.
/// This relies on an implementation detail of HybridDictionary but we're way past worrying about that kind of thing at this stage.
/// </summary>
public class MonitorInterceptingHybridDictionary : HybridDictionary
{
  static Type EventHandlerType = WebType("System.Web.FileChangeEventHandler");
  static ConstructorInfo FileMonitorConstructor = WebType("System.Web.FileMonitorTarget").Constructor(EventHandlerType, typeof(string));
  public MonitorInterceptingHybridDictionary()
  {
    var fakeTable = new InterceptingHashTable(o =>
      {
        try
        {
          return FileMonitorConstructor.Make(null, "test");
        }
        catch (Exception ex)
        {
          Log.Error(ex, "Unable to create a file monitor target pointing at {Target}", o);
          return null;
        }
      });
    Field<HybridDictionary>("hashtable").SetValue(this, fakeTable);
  }
}

Our new HybridDictionary is using another new class I’m going to re-use shortly – the InterceptingHashTable. Here, we finally catch a break: Hashtable declares a lot of its methods as virtual, so we can override functionality. What we’re going to do here is to create a new subtype of Hashtable which always has the key you’re looking for, even if it hasn’t been added. We pass a factory function in to the constructor, so whenever something asks for the value for a given key, we can construct it on the fly if it hasn’t already been added.

/// <summary>
/// Intercepting hash table. It always responds to any key request with an actual value
///  - if it didn't exist, it creates it with the provided factory.
/// This currently only implements this[key] because that's all the file monitor stuff uses.
/// To be generally useful it should override more methods.
/// </summary>
public class InterceptingHashTable : Hashtable
{
  private readonly Func<object, object> _factory;
  public InterceptingHashTable(Func<object, object> factory)
  {
    _factory = factory;
  }
  public override Object this[Object key]
  {
    get
    {
      var maybe = base[key];
      if (maybe == null)
        maybe = CreateMonitor(key);
      return maybe;
    }
  }

  public Object CreateMonitor(Object key)
  {
    return _factory(key);
  }
}

If you take a look back at the MonitorInterceptingHybridDictionary code above, you’ll see that it’s going to pretend it’s already got a FileMonitorTarget anytime something looks – and that means the calling FileMonitor will think it’s already got a watch on the file, and skip creating it. Bingo! The next step is to replace the HybridDictionary on FileMonitor with an instance of our new class.
FileMonitorTargets

Of course, it’s a private member, and we’ve currently got no way to hook the creation of FileMonitor instances. What does create FileMonitor instances? The DirectoryMonitor class – and fortunately for us, it keeps a Hashtable of all of the FileMonitors it’s created. That means we can swap out the Hashtable for another of our InterceptingHashTables, and every time the DirectoryMonitor looks to see if it already has a matching FileMonitor, we create it if it doesn’t already exist.

// Create a dodgy hashtable which never has a blank entry - it always responds with a value, and uses the factory to build hacked FileMonitors.
var fakeFiles = new InterceptingHashTable(f =>
  {
    try
    {
      // Build a FileMonitor.
      var path = Path.Combine(o as string, f as string);
      var ffdParams = new object[] {path, null};
      ffd.Invoke(null, ffdParams);
      var ffdData = ffdParams[1];
      object mon;
      if (ffdData != null)
      {
        var fLong = ffdData.GetType().Field("_fileNameLong").GetValue(ffdData);
        var fShort = ffdData.GetType().Field("_fileNameShort").GetValue(ffdData);
        var fad = ffdData.GetType().Field("_fileAttributesData").GetValue(ffdData);
        var dacl = secMethod.Call(path);
        mon = fileMonConstructor.Make(dirMon, fLong, fShort, true, fad, dacl);
      }
      else
      {
        // The file doesn't exist. This preserves the behaviour of the framework code.
        mon = fileMonConstructor.Make(dirMon, f as string, null, false, null, null);
      }
      // Break the FileMonitor by replacing its HybridDictionary with one which pretends it always has a value for any key.
      // When the FileMonitor sees it already has this key, it doesn't bother setting up a new file watch - bingo!
      var fakeHybridDictionary = new MonitorInterceptingHybridDictionary();
      mon.GetType().Field("_targets").SetValue(mon, fakeHybridDictionary);
      // Take this warning away later
      Log.Debug("Provided fake monitoring target for {Filepath}", path);
      return mon;
    }
    catch (Exception ex)
    {
      Log.Error(ex, "Unable to create a file monitor for {Filepath}", f as string);
      return null;
    }
  });

Now, we’re faced with another problem: we have no control over the creation of DirectoryMonitor instances, but we need to replace the private Hashtable it keeps with the fake one we’ve just made.
DirectoryMonitorFilemons

Luckily for us, the FileChangesMonitor class keeps a Hashtable of DirectoryMonitors!
FileChangesMonitorDirs

You might be able to guess where this is going: we need another fake Hashtable which always creates DirectoryMonitor objects on the fly. Stay with me here, we’re approaching the end of this thread we’ve been pulling on.

// This is a factory which produces hacked DirectoryMonitors. DirectoryMonitor is also sealed, so we can't just subclass that, either.
Func<object, object> factory = o => {
  try
  {
    if (!(o as string).Contains(@"\EDS\dist"))
    {
      Log.Debug("Allowing file change monitoring for {Filepath}", o as string);
      return null;
    }
    var dirMon = dirMonCon.Make(o as string, false, (uint) (0x1 | 0x2 | 0x40 | 0x8 | 0x10 | 0x100), fcnMode);
    // Create a dodgy hashtable which never has a blank entry - it always responds with a value,
    // and uses the factory to build hacked FileMonitors.
    // ... the previous code block goes here ...
    // Swap out the hashtable of file monitors with the one which lets us always create them ourselves.
    dirMon.GetType().Field("_fileMons").SetValue(dirMon, fakeFiles);
    return dirMon;
  }
  catch (Exception ex)
  {
    Log.Error(ex, "Unable to create a directory monitor for {Filepath}", o);
    return null;
  }
};

Finally, the HttpRuntime class holds a reference to a single copy of FileChangesMonitor, and HttpRuntime itself is a singleton!
HttpRuntimeFcm

HttpRuntimeSingleton

With a little more reflection, we can therefore swap out the Hashtable in FileChangesMonitor with our hacked Hashtable of DirectoryMonitors, and now we have control of all the instantiation of DirectoryMonitors and FileMonitors, and we can intercept any filenames we don’t want tracked and provide dummy FileMonitorTarget responses to prevent the watches from being set up.

/*
* I am so, so sorry if you ever have to maintain this.
* ASP.Net monitors a pile of folders for changes, and restarts the app pool when anything changes.
* Gulp often triggers this - restarting the app pool unnecessarily, annoying developers. Annoyed developers write code like this.
*/
var httpRuntime = StaticField<HttpRuntime>("_theRuntime").GetValue(null); // Dig out the singleton
// Grab the FileChangesMonitor. It's a sealed class, so we can't just replace it with our own subclass.
var fcm = Field<HttpRuntime>("_fcm").GetValue(httpRuntime);

// Grab a bunch of internal types, methods, and constructors which we're not meant to have access to.
var dirMonType = fcm.GetType().Field("_dirMonSubdirs").FieldType;
var dirMonCon = dirMonType.Constructor(typeof(string), typeof(bool), typeof(uint), typeof(int));
var fileMonType = dirMonType.Field("_anyFileMon").FieldType;
var fadType = fileMonType.Field("_fad").FieldType;
var fileMonConstructor = fileMonType.Constructor(dirMonType, typeof(string), typeof(string), typeof(bool), fadType, typeof(byte[]));
var ffdType = WebType("System.Web.Util.FindFileData");
var ffd = ffdType.StaticMethod("FindFile", typeof(string), ffdType.MakeByRefType());
var secMethod = WebType("System.Web.FileSecurity").StaticMethod("GetDacl", typeof(string));

// ... insert all the above code to create the factory ...

// Swap out the hashtable of directory monitors with the one which lets us always create them ourselves.
var table = new InterceptingHashTable(factory);
fcm.GetType().Field("_dirs").SetValue(fcm, table);

So what’s the up-shot of all of this? Well, firstly, I’ve written some of the nastiest, hackiest code of my career. Secondly, I’ve wrapped it all up into a nuget package so you can easily run this nasty, hacky code (it’s not up quite yet – I’ll share another post when it’s ready to go). And thirdly, I need to warn you again: Do not put this near production! I don’t care how you accomplish it, but please put plenty of gates around the entry point to make sure you don’t trigger this thing in a deployed environment. It really is a dev-only device.

Surely There’s a Better Way?

There is absolutely a better way. I encourage you to find other ways to avoid dev changes constantly churning your app pool. I strongly suspect our problem was serving our Gulp build output via a secondary bundling step: the Asp.Net bundler seems to be one of the things which might register files with the FileChangesMonitor. For various reasons (mostly, that the project was a big, complex, legacy project which is hard to test thoroughly), I strongly preferred dev-only changes to making production-affecting changes. And so here we are.

If you can find a way to avoid using this technique, I strongly recommend you do.

What I talk about when I talk about DevOps

What does ‘DevOps’ mean, anyway?

devops-photo-mWe’re a DevOps consultancy, and we’re hiring DevOps engineers. But the software industry has a crisis right now – and that crisis is all about the word ‘DevOps’ and what it really means. I’m starting to wonder whether putting ‘DevOps’ in our job ads might actually be counter-productive – and at YOW! this week, I finally understood why.

Elabor8 (where I work) had a booth at YOW!, and I gave a couple of lightning talks there. Probably the biggest crowd-pleaser was my talk on resiliency patterns in distributed systems. I covered some difficult topics like the importance of having idempotent rollback steps in compensating transactions and how lessons learned from the ship-building industry help us craft better distributed systems, all presented in 10 minutes in a crowded event space.

Hang on, you may well ask. If I’m a DevOps consultant, why am I talking about atomicity and consistency in distributed systems? Shouldn’t I be talking about cool PowerShell tips and how to set up Jenkins?

As is so common with rhetorical questions, the answer is a resolute ‘No’.

When I talk about DevOps, I talk about Software Engineering.

When I do DevOps work, I’m doing software engineering. When I hire for DevOps roles, I hire software engineers. But I don’t hire just any software engineers: I want the ones who care about the delivery process. They know how to build software, and also how to put it in front of the user – and how to keep on putting that software in front of users, sprint after sprint, story after story, rapidly, efficiently, and without breaking things.

‘DevOps Engineer’ is the next ‘Full-Stack Developer’ – and not because it’s the next hype-cycle in tech hiring. In the same way full-stack developers expanded their scope to cover both the UI and the back-end, DevOps engineers have expanded their scope beyond writing software – and into the realm of how we get that software in front of users, in the fastest and most reliable way possible.

No longer are we content to just build back-end systems and UI layers on top of them: as a profession, we’re coming to understand that software engineering is bigger than just churning out vertically-sliced user stories. Software engineering is about building the right thing and keeping it running – and a DevOps engineer is a software engineer who cares about both steps. You don’t want a team full of DevOps engineers – but you definitely need at least one.

When I talk about DevOps, I talk about Agile.

You start to see the real benefits of automation when you’re deploying to production regularly. For example, if you’re only shipping a few times a year, the overhead doesn’t hurt you that much. If you have a 6-week QA pipeline and a 2-month UAT window, you don’t need DevOps (yet). Once you start trying to deploy regularly – getting your cycle time down, keeping your WIP low, delivering value to the user faster and more frequently – that’s when the overhead starts to hurt.

Once you introduce agility to your process, that’s when you need to pay attention to the DevOps movement. That’s when you need some software engineers who care about automation. Please note though that I’m not saying you need “some DevOps” – there’s no such thing as “some DevOps”, and anyone who tries to sell you “some DevOps” is doing you no more favours than someone who tries to sell you “some Agile”. What you need is some smart software engineers who care about DevOps – and you need to give them the time and resources they need to do their job.

When I talk about DevOps, I talk about teams.

DevOps engineers are software engineers, but that doesn’t mean you should fill your software teams up with DevOps engineers. DevOps engineers tend to be passionate about a bunch of really interesting stuff: resilience patterns, testing, automation, source control, and release management. The great thing about multi-disciplinary, cross-functional teams, however, is that you get a bunch of people with different passions together, and that breadth gives you the ability to do great things. Don’t try to hire DevOps engineers (or worse, to build a DevOps team). Hire software engineers, and when you find ones who are great at DevOps, keep them.

Having cross-functional teams also gives your team members the chance to cross-skill while they work with other team members, which is why…

When I talk about DevOps, I talk about teaching.

Great DevOps engineers have a genuine enthusiasm for quality software engineering and release management, and their enthusiasm is infectious. Great DevOps engineers don’t hoard their knowledge, but help their fellow software engineers to learn more about the DevOps mindset by sharing what they know. They also learn from their colleagues who specialise in other software engineering fields, becoming more well-rounded themselves as they help others do the same.

Finally: When I talk about DevOps, I talk about people.

DevOps is a branch of software engineering – and whatever you might hear, software engineering is all about people. It’s about the people who use our software, and the people who build it. DevOps is the intersection of those two groups: users and developers. Our users are not just end-users, who enjoy higher-quality software, but also our fellow engineers, who rely on our automation to work more efficiently. They’re our managers, who rely on the insights we give them into the release process. Our users are the junior software engineers who may one day specialise in DevOps engineering – or who might use what they can learn from us to be better at some other branch of software engineering.

And as always, we have a bunch of work on, and we’re hiring.

So, I have two things to ask. Firstly, if you have feedback on our job ad, please reach out! I’m always trying to improve these things.

Secondly, if you’re a software engineer who cares about the difference between GitFlow and GithubFlow; who has made calls to the Octopus API; or who loves showing other developers how to write a better test or add more context to a log message; please talk to me about joining Elabor8. We’re a bunch of software engineers who care about DevOps – and our users aren’t just one team, but a range of teams across some really great clients. If you want to take the engineering you care about and have a broader impact on more people, you want to join us. Get in touch.

Anatomy of a Job Ad

Wanted: Senior DBA!

  • Minimum 10 years experience managing SQL Server 2015 or newer

It’s a classic joke – the job ad which wants you to have experience in a specific technology for longer than it’s existed. Sadly, it’s funny because there’s a grain of truth in it: job ads are terrible.

pexels-photo-52608 (Small)Wanted: Junior Developer

  • 1-2 years experience writing software
  • Strong knowledge of SQL, git, Python, Haskell, and Perl
  • Good UI skills, excellent API design skills, knowledge of ESBs and other distributed software patterns
  • Excellent grounding in the principles of good software design and Agile project delivery
  • Experience with [you usually find a list of specific platforms and libraries here]
  • Go-getter, always-be-learning attitude
  • Highly regarded: strong maths and stats skills, C++ experience
  • Highly regarded: exposure to data science, experience with Big Data platforms such as Hadoop
  • Excellent communication skills are a must!

I hate seeing job ads like this one. You want someone with 1-2 years’ experience to know all of that – and be confident enough to tell a recruiter that, and back it up in an interview? You’re not selecting for capability here. You’re selecting for over-confidence, and the coincidence of having a first job which involved just the right mix of technologies.

Technology Leadership Position!

Are you a dynamic leader with top-notch management skills and brilliant technical ability? Do you have world-class knowledge of big data platforms and machine learning? Are you just as comfortable writing a PhD dissertation as you are selling a group of executives on a new company strategy? Do you have a passion for creating a dynamic company culture and mentoring a team of brilliant engineers, all while maintaining an unwavering focus on creating an unmatched user experience? Do we have a job for you!

If you’re posting a job ad like this one, you’d better have a remuneration package to match; but you probably don’t. The technology industry is full of people with imposter syndrome – and if the perfect applicant (who would have been great for your team) didn’t have it before they read this job ad, they will afterward. If the candidate who does meet that brief exists, they’re definitely not reading job ads on Seek, anyway.

These are, of course, caricatures, but they’re not so different from real job ads I’ve seen. How about a real example from my own life?

Checkbox Syndrome

Many years ago, I landed a job in the United States (I’m based in Australia, so this would have been a big move for me). The recruiter explained that the job had been open for nearly two years, and they were thrilled to have found me! I was their first suitable candidate, and they were keen to rush me through the hiring process.

This puzzled me, because I’m not that special. How was I the first suitable candidate in two years?

Well, it turns out that they’d written a very specific candidate description, and someone in HR had been handed the job of finding someone that matched. They were after someone who listed optimising high-throughput transactional systems on their LinkedIn profile, and who had completed at least two years of tertiary study in Latin or Ancient Greek.

They were building some language-processing software, and someone had decided that an engineer with a background in linguistics would be good to have on the team. That was somehow translated to requiring some tertiary education in a classical language – and bam! Two years of turning down candidates who probably would have been highly successful in the role. All they actually wanted me to do was to come up with a way to load-test the system, and then find hot-spots in the code and optimise them.

I never did end up moving to Houston: my visa application fell through (thanks, Global Financial Crisis). It was probably for the best: I wouldn’t have been a person to this company, or even an engineer; just a series of checkboxes.

What is a job ad for, anyway?

When you’re writing a job ad, please don’t forget what the ad is for. Your goal is to attract suitable applicants to apply, and discourage unsuitable applicants. Anything which doesn’t accomplish one of those two things is wasteful. Anything which discourages suitable applicants is a net loss.

As a specific example of this, please leave “Good Communication Skills” out of your ad. You might think that poor communication skills will disqualify a candidate – but putting that in the ad isn’t going to stop unsuitable candidates from applying, and it’s not going to encourage suitable candidates to apply. It’s noise. Leave it out.

Sell the job – genuinely

It all starts with empathy. Try to imagine yourself as your target engineer, tester, analyst, or whoever you’re trying to hire. Your goal is to sell them on the job, but also keep enough important criteria in there to discourage unsuitable applicants. The selling part is important: if there aren’t many great candidates out there, you need a job ad which will attract them. Candidates are about to invest a significant amount of their own, personal, outside-work time in applying for and interviewing at your company, and then you’re going to ask them to resign from their current position, where they have friends and valued colleagues and know the system, to join your team. Give them good reasons.

Please don’t give them a sales pitch. Don’t spend most of the ad spruiking the company. Talking about how great you are is for your marketing material, or your annual shareholders’ report.

Just talk honestly about your culture, values, and benefits.

Anatomy of a Good job ad

Here it is. This is what your job ad should look like.

  • Talk about the role.
    Tell prospective candidates a little bit about the company and the role. This should be short. What does your company do? Who are you looking for? How will this position further the company’s mission?
  • Say who you’re looking for.
    Really cut it down. Don’t have 10 bullet points. Keep it to 3 or 4. Try to focus on higher-order skillsets, rather than specific technologies, unless that technology really is core to the role.
    If “Good Communication Skills” is making it to your short-list of 3 or 4 key skillsets, I hope you’re hiring a radio operator or air traffic controller. Otherwise, please, just leave it off.
  • Tell them what’s in it for them.
    Describe your benefits. Tell them about the great company culture. Talk about training and conference budgets and career opportunities. Once again, keep it short and to the point.
    If you can’t think of anything to put here, you might have bigger problems.
  • Tell them how to apply.

That’s it. Really. No nice-to-haves – those just give suitable candidates a reason not to apply. Don’t do that. If you get two suitable candidates, and one of them happens to have one of your nice-to-haves, you might still offer the role to the other based on the bigger picture. If it’s not necessarily a differentiator even at the end of the recruitment process, it definitely has no place at the beginning.

But I want to list a bunch of stuff!

Resist the temptation. Are you hiring a team lead? Just say you’re hiring a team lead. Don’t list out all the stuff that team leads need to do.

  • Lead a team of software experts to deliver innovative products.
  • Help to work with product owners to deliver real business value!
  • Mentor senior engineers, and help them to mentor juniors.
  • Engage with the business about technical challenges.
  • Foster a strong team culture.

Don’t do this! Team leads already know the day-to-day detail of the role, and they won’t be going back to the job ad to work out how to spend their time. Just say that you’re looking for an experienced team lead or a senior engineer looking to step up to a team lead role – and get on to covering the more important points! What kind of team is it? What is the team mission? Are you looking for someone to drive a major change, to keep an already-high-performing team pointed in the right direction, or to build a whole new team? That stuff is much more useful than saying things like “Ensuring the team is aligned with business priorities”.

Where do I put things like “Strong work ethic” and “Ability to work in a collaborative team environment”?

In the same place you put “Good Communication Skills”. People who don’t have a strong work ethic or the ability to work in teams are going to apply anyway, so all you’re doing is making the job ad longer and more boring.

This sounds like you’re a really cool person and I’d like to work with you.

Readify is always looking for great people.

Wait, was this secretly a job ad?

No. This doesn’t match my “Anatomy of a Good job ad” at all! But recruitment is bigger than just job ads, and this was, secretly, a bit of guerrilla recruitment. That’s another idea I’m hoping you’ll take away from this post: recruitment is about a lot more than just writing a good job ad. It’s about being a place people will want to work, and making sure the right people know it.

Measure What Matters

What’s your average API response time? Do you know? Is it important to your business? What about the 90th percentile? Do response times suffer during peak demand?

Do you think about those questions? How about these ones:

How long does it take to get a software change reviewed? Do you know? Is it important to your business? Is it a bottleneck? Do reviews get skipped during busy periods?

If you care about code reviews, you should measure them. Put them on your system dashboard. They’re as much an indicator of the health of your software environment as your API response times. Minimising Work In Progress and Mean-Time-To-Release are important parts of your QA process, and making sure your pull requests are reviewed and merged in a timely fashion is a great way to improve those numbers.

What existing products are there out there to do this? Depending on the tools you use, you can probably pull out a few relevant reports. Jira is popular, and I’ve seen PMs produce some great graphs to include in their monthly management update. The problem is, the numbers you get out of these tools don’t give you direct, real-time feedback. Their very nature as longer-term averages mean they can’t represent a call to action.

Enter TeamLab

As a software shop, if the tools I’m using don’t do what I want, I have an option: build something. This is a dangerous option to have, and countless business hours have been wasted solving the wrong problems, but I really needed a nice visual prompt of how we’re doing at our code reviews in-the-moment. I also wanted a side-project for the team to tinker with new ideas for writing web applications – so even if the project didn’t turn out to be useful, the experiment would teach us something.

I had a specific technology I wanted to try out: React Storybook. This is a really nice way to visualise your React components in various different states, and I wanted something relevant to use as a demo for the team. It was very quick and easy to get up and running with a create-react-app project including Storybook, and I hacked together a quick picture of what my PR display should look like:
Storybook10PRs
On the right, you can see my quick mock-up of a board displaying ten pull requests, and the left is the Storybook control panel.

I decided it would be useful to colour-code the pull requests, and display any reviewers and approvers on the PR cards. A new PR is yellow, and an approved one is green. A PR with reviewers turns blue, and most importantly, any PR which is older than 48 hours turns red.
StorybookPRs

This was a nice little mock-up, but there was no real data behind it at this stage. Fortunately, the Git server we use has a fairly straightforward API, and so it didn’t take long to get some real data behind this component.
TeamLabPRs

It’s really easy to see when we have PRs which are starting to get stale, and need attention. Quick – at a glance, how many PRs here have been hanging around too long and need attention?
MorePRs

This has become the go-to way of seeing our outstanding PRs at a glance, and has since gone up on a big screen on the wall in our dev team office. I soon got requests for a few other widgets to go on the same dashboard, and our little side project has become a key part of our DevOps toolkit.
TVDashboard

Has It Worked?

Having those cards up where we can see them during the day has been good – but the biggest signal is during stand-up each morning. A quick glance at the TeamLab PR board has become part of the ritual, and if those cards start to build up – especially if they start to turn red – the team has a really strong signal that we’re getting behind on our code reviews.

I don’t currently have a report which tells me the Mean-Time-To-Merge for our PRs – but I don’t think I need it. Mean-Time-To-Merge isn’t as strong or immediate a signal as a pile of glaring red PR cards looming over our morning stand-up, nor does it provide the immediate sense of relief when we clear the board.

What Next?

I’m not sure what will go on the dashboard next, but I have some idea what kinds of things I’m looking for.

I need things I can measure – things I can pull straight out of an API. Things which can directly influence numbers like Mean-Time-To-Release – but I don’t want to display averages like that. I’m going to give people a dial they can turn directly. I’ll pick an angry colour like red for things which are outside targets, and nice friendly colours like blue and green for things which are on track. Once something is off the list, I’ll make it go away.

In short, I want to find things which I can measure, which team members can directly influence, and which will improve our overall quality – and I want to put them up where everyone can see them.

Announcing NSchemer 1

If you already know all about NSchemer, you can jump straight to the Version 1 release notes.

What is NSchemer?

Database schema management has been an interest of mine for a very long time. I’ve seen all sorts of approaches tried: folders full of .sql files, schema version tracking in Excel, and of course the tried-and-true1 manual approach using schema diffing tools. I’m a keen proponent of automated schema management. Automated deployment is all the rage these days, and if you can’t automate your schema updates, you can’t automate your releases.

I prefer to go one step further: I like to aim towards single code path schema management. Any database, whether a brand-new one to support a new installation, or an ancient database restored from a backup for a returning client, should get to the current version using the same code path – or at least, as close as possible.

When I started pushing this idea – that developers should write their own SQL migrations as they went, rather than leaving it to the designated DBA to do in the lead-up to a release – I got some push-back. Some of my team didn’t want to write SQL. Thus, NSchemer was born2. The framework languished in alpha status for many years, despite being actively used in a number of production systems. Recently, I finally decided to tidy up the API, add a few new features I’d been meaning to for a while, and bump it up to version 1.

Why Automated Schema Management?

The number one reason for automating your schema management is testing and reliability. Assuming, for a moment, that you have test environments, automated schema management means your test environments should go through the same migrations as your production environments will – automatically, with no opportunity for a manual step to get skipped or done incorrectly. This gives you a lot of confidence that when you hit the Big Red Button to go live with a new version, your schema migrations will work: the same automated set of steps which have run against all of your other environments will run against production.

You get a lot of other nice bonuses, as well. When you merge master into your own branch, not only do you get all of the new code; you also get the migrations that update your database schema to match. No more pulling in another branch, only to have to manually update your local database schema to match.

Digging up a backup from a couple of years ago? No worries, NSchemer will bring it up to current without any hassle at all.

Installation

While you can install NSchemer into an existing assembly, I typically create a new assembly just for managing schema transitions (I use a console app, so I can run the transitions from a script during deployment). Once you’ve created YourProjectName.Schema, just

install-package NSchemer

and you’re ready to go.

Show me the code!

NSchemer uses a single class which inherits from SqlClientDatabase to represent a versioned schema. Just create one, implement the Versions collection, and start writing transitions (beginning from 1 – NSchemer uses version 0 internally). If you’re starting with an existing schema, just use your favourite SQL tool to generate a full CREATE script, and drop it in as version 1 (use the embedded resource transition mentioned below).

public class TestSchema : SqlClientDatabase
{
    public TestSchema(string connectionString) : base(connectionString) {}
    public override List<ITransition> Versions
    {
        get
        {
            return new List<ITransition>
            {
                new CodeTransition(1, "Initial Schema", BuildTheWorld),
                new CodeTransition(2, "Add Widget Table", "This script adds a very important table", AddWidgets)
            };
        }
    }
    private bool BuildTheWorld()
    {
        CreateTable("Thing",
            new Column("ThingId", DataType.BIGINT).AsIdentity(1, 1).AsPrimaryKey(),
            new Column("ThingName", DataType.STRING, 50)                
        );
        CreateTable("ThingAnnotation",
            new Column("AnnotationId", DataType.BIGINT).AsIdentity(1, 1).AsPrimaryKey(),
            new Column("Text", DataType.STRING, 50),
            new Column("ThingId", DataType.BIGINT, false).AsForeignKey("Thing", "ThingId")
        );
        return true;
    }
    private bool AddWidgets()
    {
        RunSql(@"CREATE TABLE DBO.Widget (WidgetId [int],WidgetName [nvarchar](50)) ON [PRIMARY]");
        return true;
    }
}

The core of your schema class is the Versions collection, which contains a numbered list of transitions to be run in order. NSchemer will automatically create a table to track which versions have and haven’t been run, and whenever you call Update() on your class, it will work out which transitions haven’t run yet, and apply them. Assuming your transitions all ran without exceptions and returned true, your schema should now be up-to-date, and the version history table updated. If any of your transitions either returned false, or threw an exception, Update() will throw an exception.

Why not just SQL?

You’ll notice, in the sample above, that as well as being able to write SQL transitions, there are helper methods like CreateTable. These are here for four reasons:

  1. Some developers refuse to write SQL.
  2. Some developers write terrible SQL.
  3. They’re actually pretty convenient.
  4. If/when NSchemer officially supports non-MSSQL databases, your transitions should be cross-platform.

If you don’t want to use these convenience methods at all and you’re happy just using SQL, you could also look at some of the SQL-only frameworks which do the same thing as NSchemer, such as DbUp (which has explicit support for a range of other databases as well as Microsoft SQL Server).

If you have larger blocks of SQL to run, don’t put them all into a string like in the example above: NSchemer also supports resource files. You can create a transition like this:

new SqlScriptTransition(3, "Add another table", "NSchemer.SystemTests.EmbeddedFile.sql")

It will look in the same assembly for an embedded resource file with that name. There is also an overload which allows you to specify a different assembly for the resource file.

NSchemer uses a similar format to SQL Server Management Studio: it uses GO on a line by itself as a command separator, allowing you to submit multiple chunks of the file as separate commands.

Configuration

NSchemer has a couple of options you can use to control its behaviour. I’m afraid the API is inconsistent and the options are limited: I plan to address this when I do the overhaul in 2.0 (see below).

  • VersionTable
    Override this property to change the table NSchemer uses. Default: NSCHEMER_VERSION
  • SchemaName
    Set this property to use a schema other than dbo (use with caution: this has limited test coverage).

Can I rely on NSchemer?

You should be testing all of your migrations in testing and staging environments before they make it to production. These tests will also ensure NSchemer is behaving itself in your environment. Should you start using NSchemer in production without ensuring it goes through a testing pipeline? No, but you shouldn’t be running your own code that way either.

If you find any bugs or problems in NSchemer, please report them on the NSchemer GitHub repository. I use NSchemer myself, and I’m keen to fix any bugs you find as soon as possible. I’m also open to pull requests.

Implementation Advice

I like to run NSchemer from two different places: one in development environments, and another in deployed environments (testing, staging, UAT, production, whatever you prefer to call them).

To run in development environments, I throw some guards in (to make really sure it never runs in a deployed environment), and put it somewhere it will run on application start-up. It might look like this:

if (Debugger.IsAttached && _connectionString.Contains(".\sqlexpress")) {
    new MySchemaClass(_connectionString).Update();
}

For deployed environments, I make sure the assembly containing my transitions is a console app, run the transitions from there, and return or output a success/failure message so my deployment tool knows whether to continue or alert me that something went wrong.

Upgrading from 0.x

The only changes you should need are to sprinkle a few using NSchemer.Sql statements at the top of your files.

You should notice some API improvements:

  • When you create columns, there are new options you can provide using a fluent syntax:
    • .AsPrimaryKey()
      Only allowed when creating a new table. Specifies that this column is part of the primary key. Supports composite keys.
    • .AsForeignKey(…)
      Allowed when either creating a table or adding columns to an existing table. Allows you to indicate the referenced table, column, and (optionally) cascade options.
    • .Identity(…)
      Allows you to create the column as an IDENTITY (auto-increment) column. You can specify the initial seed, and increment value.
  • Description is no longer required on transitions.
  • CreateTable(…) now has a params syntax, so you don’t need to create a List every time you use it.
  • You can now specify the nullable status of a column for data types which don’t require a length.

The Future: NSchemer 2.0

As I’ve used NSchemer, I’ve discovered a few short-comings of the existing API. I made some minor breaking changes when I bumped NSchemer to 1.0, mostly just relating to namespaces, but 2.0 is likely to be a more significant overhaul of the API.

I also want to split Microsoft SQL Server support out into a separate package, and provide official support for other database servers. If you have strong knowledge of MySQL, PostgreSQL, Oracle, or another database platform and would like to help maintain support for that platform, please get in touch.

License

I’ve chosen to release NSchemer under the LGPL because I want it to be generally usable, but I want to make sure any improvements are available to everyone who uses it. If you want to use NSchemer but the LGPL is a problem for you, please let me know. I’m sure we can work something out.

Healthcare Software: Past, Present, Future

A guest post by myself from a different blog.

Healthcare-Past-Present-Future

I did this for a slide deck on the past, present, and future of software at Ramsay Health Care. Disclaimer: opinions are my own, and future may or may not represent actual future. Also I think our wards are usually nicer than this!

%d bloggers like this: