Category Archives: Software

How I Learned to Stop Worrying and Love Consulting

A few months ago I wrote about my motivation problems since switching to consulting. I think this was a somewhat natural thing to feel: I’ve built quite a few in-house software teams over the years, and consulting companies were my natural enemy. They hired many of the top people in my local area, and then wanted to sell them back to me at a substantial mark-up. What jerks!

A little over a year ago, I joined the dark side and became a jerk consultant. My former enemies are now my friends, and I’m the one being sold back to in-house teams for profit. I learned to live with that fairly easily: if we weren’t giving our clients value, they’d stop paying us – and we get heaps of repeat business. No, the economics of it weren’t the problem.

The problem was Mission.

I’ve always thought of myself as strongly mission-driven. When I worked in finance, I believed that what we were doing was making life better for our customers. When I worked in environmental engineering, I believed that we were helping our clients get the most environmental bang for their buck (my more cynical side occasionally observed that we were helping them spend the minimum possible to avoid trouble with the regulators). When I worked for a hospital group, it was easy to think of patient care as a worthwhile mission, because it is.

Oddly, what I’ve since learned is that my mission is much less available for rent as a consultant that it ever was before. Protecting the environment wasn’t my core mission before my pay check came from Pacific Environment, and it hasn’t been since I left. Patient care wasn’t my mission before I joined Ramsay Health Care, and it isn’t now.

Are those all worthwhile missions? Absolutely. But in hindsight, the mission I took on was entirely driven by who was paying my salary.

Don’t get me wrong. I’m not unhappy with my career choices, and I think I made a positive impact on the world (and hopefully my local software industry) on my way through. But joining a software consultancy is a neat way to de-couple my mission from my salary. I still have a core mission, but now it’s much more in concert with who I am as a person. Readify’s mission is all about using technology to make the future better for everyone – and that speaks very deeply to me. I’m a technologist at heart. My secondary mission may be for sale, but the core goal of myself, my colleagues and my employer remains the same. That’s not just something I can live with: it’s something I can embrace.

So, that’s the mission aspect sorted – but I’m not the moony-eyed idealist I might like to think I am. You could see all of this talk about “Mission” as an exercise in justifying my current choice – just like I justified all of my previous employment choices. What about the economics of it all? That’s where my opposition to software consulting began, and it’s not so shallow-rooted to be slain by a simple mission statement.

Well, here’s where I needed to shift my thinking a little. I needed to start thinking about industry sustainability.

Before I get to the point, bear with me while I give you a quick tour through some highlights of my career. My first significant production responsibility came a few years in, when I was a Senior Software Engineer at a successful FinTech start-up. I reported directly to the Managing Director (who was from a finance background). Some years on, I found myself a Senior Software Engineer and Team Lead for an environmental firm, where I reported to a variety of people (including the CTO, and occasionally the CEO) – who were mostly from an environmental engineering background. During my time in the hospital industry, I was again in a leadership role – being such a large multi-national, there were a few more layers involved, but my boss’s boss’s boss (and most of his colleagues) were from a doctoring or nursing background.

I’ve seen this pattern repeated across many industries: flatly-structured software teams with little or no opportunity for career advancement, reporting to non-software people. Reporting to non-software people isn’t intrinsically bad: some of my all-time favourite people are non-software people (I even married one!). However, I don’t think it’s a sustainable model for our industry. I’ve seen far too many software developers come to realise that their best opportunity for advancement was to move sideways (out of software) – and I think that’s terrible. It’s almost become a truism that “software is eating the world”, and certainly one of the highest-value-creating things you can do today is to build software. Why would I help perpetuate a model which pushes our best and brightest to go and do other things?

Participating in a software consultancy like Readify is a way to break that mould. I’m in a leadership role, but I still have plenty of room to grow in my career without abandoning my core mission and my experience as a software engineer. If I decide that my current business group (we call it “Managed Services” but it’s really a combination of production engineering and feature development, driven in a Kanban style) doesn’t suit me, we have plenty of other flavours of software development I can switch to. If I really want control of my own destiny, I’m learning the business of software consulting – so I can always go and do software on my own terms. Mid-career, I have software developers reporting to me, and I report to other people from a software background – all the way up. I can move up the chain, or try to disrupt it – and either way, I’ll be participating in an industry model which mentors and builds software careers, from fresh graduate to C-level executive. Perhaps most importantly, I’ll be participating in a sustainable model – which keeps people in software, because it’s a great career with opportunities in line with any other you could choose.

Software consulting isn’t the only way to achieve this. Software-focused product companies offer the same potential sustainability. I can’t even rule out that my career might take me back to working directly in other industries – but if I do, it will be with a new understanding of the importance of a sustainable software industry, and an eye to ensuring that the technologists among us never feel the need to go do something else to advance their career.

Enduring Software Development Myths

I’ve been asked more than once about software development myths, and I’ve run into plenty over the years. Here is a (non-exhaustive) list of some of the myths I’ve seen people believe. I’ve tried to focus on enduring myths, rather than short-term fads (like “Blockchain is a broadly applicable technology”).

Myths Management Believes

adult-businesswoman-company-325924

1. Every manager has read ‘The Mythical Man-Month’ and thinks they understand that adding more developers to an already-late project tends to make it later. However, they will still do it, apparently believing that some combination of proper management, having good people, and the specifics of a project mean it doesn’t apply in their situation.

2. Management also has a tendency to believe that large, complex software systems get ‘finished’ and then become a cash-cow, bringing in an ever-increasing revenue stream for fixed costs. Even if they do account for increasing support and maintenance costs, they rarely consider that continuing innovation and development is required to keep a product competitive.

3. Managers sometimes believes that the primary role of a developer is to translate specification into code. They don’t understand the numerous design decisions involved, the trade-offs being made, the non-functional requirements being considered. I have never once in my career encountered a specification which considered the sort of logging detail required to diagnose the types of system failures which might occur – and yet that is a key component of writing good code. Every developer runs into minor details from day to day which require assumptions or decisions to be made – and a good developer should be tracking those, communicating them, and ensuring the right people are involved in the right decisions, while being shielded from the bulk of less-important ones.

4. I have frequently run into managers who believe that their developers are not politically savvy, or that they all have poor networking skills. This can lead them to interact with other parts of the organisation in a way which under-estimates the contribution of the development team (see 3). When managers do this, their team does find out about it, and it creates resentment.

5. Managers sometimes think that developers within their team are roughly fungible. Managers usually understand that different developers have different areas of expertise and levels of experience, but they often underestimate the impact. They don’t realise that it can have not a 2x effect, but a 5x or greater effect – and that effect is not limited to time, but also to quality (but see myth #2 under “Myths Developers Believe”, below).

6. “All platforms, languages, and third-party systems are roughly equivalent” or “This specific platform, language, or third-party system will fix all of our problems”. I have run into both over the years. The former leads to upper management thinking the selection of a third-party provider such as a cloud host or database platform is purely a business decision and will have no technical impact. The latter can result in management embarking on costly technical projects to no benefit (“we have to re-write our entire codebase into [language]“ or “we need to move all of our data into MongoDB”).

Myths Developers Believe

devops-photo-m1. Developers often believe that management doesn’t have anything to contribute, or that they’re useless or unskilled. Notwithstanding the above myths, managers can make significant contributions and many of them do really know what they’re doing. The good ones shield developers from distraction, ensure they have proper equipment and software, manage outwards to get realistic (and, importantly, low-stress) deadlines, communicate early and mitigate the effects of missed deadlines, justify budgets and pay increases, bring in appropriate consultants to address short-term skill gaps, ensure meetings minimise disruption, and do countless other things to make projects run more smoothly.

2. Developers often believe in 10x or ‘Rockstar’ developers. Some developers can be significantly more productive than others, but this is usually just a matter of experience and knowledge. It is also often specific to particular topics. For example, I used to do all of my team’s low-level network programming – many on the team considered it an impenetrable field and thought they weren’t capable of doing it. A while ago, though, I took one of our juniors aside and spent some time teaching him what was involved. He is now nearly as productive as me in that area, as long as he doesn’t go too long without working in it. Once he’s relearned the skills a few times, they’ll become a permanent part of his skillset. He’ll then be a 10x programmer – but only when working on low-level networking code, and only when compared to a developer who hasn’t spent much time on that specialty.

3. “Most developers are much better than me.” Known as Imposter Syndrome, developers are particularly prone to it, due to developer myth 2. It is especially common in generalist-type roles, where a developer may go long stretches without working on the sorts of things he or she is particularly good at. Instead of treating these times as opportunities to develop new specialties, they can instead look back at periods of much-increased creativity and think that they’re a bad developer who spends most of their time being unproductive. In reality, they are normally-productive for much of their time, and super-productive when the thing which needs doing is an excellent match for their experience and knowledge.

4. I don’t quite know how this one crops up, but many developers think having a high reliance on tools is bad. I have encountered developers who refuse to use automated deployment systems (“I want to understand what’s going on, and I can’t do that if it’s automated”), refactoring tools (“it’s safer to make each change manually”), and even syntax-highlighting and code completion (“I know my language and my libraries. What if one day I’m stuck using vi on a green-screen linux box from the 70s? Where will I be then if I’m used to these fancy tools?”). I call this concrete-pill syndrome, and I suspect it has something to do with wanting to look capable of handling everything yourself. In reality, these tools (and many others) are great levers, allowing developers to increase productivity and quality.

5. “As a developer, it doesn’t matter if I have poor people-skills, because in developer roles I can ignore office politics anyway.” This is a recurring one, and it doesn’t just hurt your long-term career – it hurts your current project, it affects the sorts of projects you get to work on, and it hurts your team. I try to shield my team from the need to play politics, but I also try to let them see just how much of it I have to do. Not playing office politics well enough can leave you stuck with a bad manager (see some of the earlier myths about the contribution a good manager can make), or it can leave you stuck with a really bad external technical decision. It can leave you stuck with a really bad hire.

Earlier in my career, someone from upper management asked me: “Of the candidates for the developer role you rejected, who was the least bad?” I made the mistake of telling him, and I got stuck with that bad candidate on my team right through his probation period. I should have known to respond with “They were all unsuitable, let’s line up some more interviews”.

6. Developers early in their career sometimes think they are much better than experienced developers. I’ve heard it called ‘God’s Gift to Coding’ syndrome. It happens because they believe they are better with new technology, and new technology is more productive, and so they are worth more than someone who is too out-dated in their thinking or is stuck on old platforms and techniques. While some older developers do remain focused on older systems, they have still built up a depth and breadth of experience which is valuable. However, many developers continue to be intrigued by new technology as they get older, and keep their skills as (or more) current than many juniors.

7. This is a two’fer: “Domain experts outside software aren’t really that smart” and “domain experts have magical knowledge of an esoteric field I can never begin to understand”. The reality is that the domain experts you work with – whether they are accountants, scientists, engineers, or sales people – have years of learning, training, and industry experience which is highly valuable to your team. As a developer, you need to respect that – but you also need to be a sponge, learning what you can about the domain. If you don’t understand what you’re building, you will fail to consider important things. If you don’t rely on your experts for their professional knowledge, though, your project will be built by amateurs who happen to be good at software.

(I originally published this list on Quora but wanted to get it on my own blog as well.)

Consulting and Motivation

Caveat: I’m very hesitant to post this publicly. Please read it in the sense I’ve written it. I’m being vulnerable in the hope of opening up discussion on a topic which I can’t see people talking about, but which I think is probably important.

I’ve spent most of my software career as part of a product team. Whether I’ve been writing code (mostly), managing people (when needful, and more often lately), mentoring, or doing any of the many little side-roles which are needed to build a successful product, I’ve always been able to rely on the fact that everyone I’m working with has the same big-picture mission.

I’m not talking about “producing great software” here. I’m talking about the alignment between business, and product, and people – over the years it has ranged from “let people put a $2 coin into a kiosk and use the internet” to “produce weather forecasts to help mines make better decisions” to “provide better patient outcomes in hospitals”. That shared mission has always been fundamental to my own sense of purpose and motivation. It’s what makes me feel worthwhile as a person, and it helps me justify to myself the salary I get paid – I would hate to feel like I’m being paid money but not adding value!

A bit less than a year ago, I moved from product teams to consulting – and the rug has been pulled out from under my feet.

It probably doesn’t help that I’ve moved straight into a leadership role. I don’t have one client for weeks or months at a time – I have somewhere around 20 clients simultaneously, as well as leading a team of consultants and being part of the overall leadership of my state and my national group. My usual source of motivation – that shared mission – hasn’t just been diluted. It’s been shredded, processed, ground up, kneaded into dough, cooked, and handed back to me in a form I just don’t recognise.

I needed to find a new source of motivation.

I went looking online, and there’s not a lot out there. I read listicles about how to stay motivated as a consultant – “Exercise!” “Don’t sweat the small stuff!” “Take time to recharge!” “Just believe in yourself!”. I read (or mostly listened to) books about consulting, and they gave me all sorts of strategies – for being a better consultant. I just haven’t been able to find much (anything?) addressing my motivational problem.

I’ve had a string of successes, mostly great outcomes, and one or two things which haven’t gone well. I’ve watched the various strategies we’ve used to motivate people with lots of interest – hoping I’d find something to get my gears engaged. Positive call-outs, monetary rewards, team-building exercises. They’re good strategies. They make me feel good in the moment. It’s a great team – some of the smartest people I’ve ever worked with. That can be hard to deal with in itself, but I’ve mostly moved on from my imposter syndrome to something a little more complicated, so that’s OK.

None of these things has been a solid substitute for the fundamental sense of purpose I used to get from being part of a team with a mission. A mission beyond just “build great software” – that’s just a means to an end. A mission like “create better hospital outcomes for patients”. That was a good one.

I have clients with missions now. Some of them even believe in their own missions, and occasionally they’re missions I care about too. But I only get to care about my clients’ missions sometimes – and other times my mission is aligned with my own team (whose mission is somewhere between “build great software” and “keep the client happy”), and sometimes I’m worried about the broader group or consultancy. Sometimes I’m aligned with the sales team, and sometimes I’m opposed to them. Sometimes I’m focused on a client whose mission I disagree with. Success, then, means making the world a worse place (from my point of view).

The best advice I’ve found so far has been this: find a sense of pride in the quality of your work, and the reputation of your team. Good advice, I’m sure, and probably very fulfilling for some.

I haven’t had any luck decoupling the quality of my work from the outcomes and the team and the mission. Quality work is just another means to an end: accomplishing the mission.

I normally write blog posts because I feel like I’ve got some knowledge to share, but the more leadership I do, the more that I understand it’s not always about dishing out nuggets of wisdom from on high – it’s about facilitating discussions, and helping create understanding from our shared experiences. So here I am, ending a blog post with the same uncertainty that I started it with.

Please, reply here, or reply to this tweet. Whether you’re a consultant now, or you have been one in the past and struggled with the same thing, or you’re on a product team and get your motivation from another source. Whether you have an answer, or a different perspective, or just more questions – let’s talk about this topic. One of the biggest industries in my city is consulting, and I feel like this is a big topic which just isn’t talked about. Let’s start talking.

How my C# Object.Extend implementation works

I’ve had a few people ask me to explain how my C# Object.Extend implementation works – so here I am, going into more detail than I expected people would be interested in.

Making Object.Extend work seemed straight-forward at first. You can read my initial code in my original ObjectExtend post. The problem started when I tried to chain two calls together: I immediately got a RuntimeBinderException. Unfortunately, for a really good reason (the difficulty of discovering the right namespaces to search), the code responsible for runtime method binding doesn’t look for extension methods. That means that when the first call to .Extend returns something marked as dynamic (regardless of what actual type it returns), you can’t use any extension methods on the result. Attempting to call either .Extend again or .Dump (another common extension method in LINQPad) results in the dreaded RuntimeBinderException.

The usual usage of the dynamic keyword is to let us return something like the ExpandoObject type, like this:

dynamic ReturnSomething() {
	return new ExpandoObject();
}

Fortunately, the dynamic keyword in C# essentially just tells the compiler to engage duck-typing mode – it doesn’t actually tell the run-time anything. So there’s nothing stopping us from doing this:

dynamic ReturnSomething() {
	return new List<string>();
}

The compiler (and your IDE) will allow you to try to call anything at all on the return value of ReturnSomething, and hand everything off to the runtime binder. This means that you can’t use extension methods on anything returned by a method marked dynamic, even if you actually return a regular non-dynamic type.

dynamic ReturnSomething() {
	return new object();
}
ReturnSomething().Dump();

Nope. Instant RuntimeBinderException.

There’s one situation which will give us exactly what we want – if we mark the .Extend function as dynamic, but the type we return has our Extend method built right in to it. This means our IDE (LINQPad, in my case) will allow us to access any of the dynamic properties we build up on our types over successive calls to Extend, and the runtime binder will be able to find the methods because they’re right on the type we return – we’re not relying on extension methods!

I initially thought I could create a subclass of ExpandoObject with calls for Extend and Dump, but it turns out that not only is ExpandoObject a sealed class, it’s also a bit special in other ways – so that was a no-go.

So now the only problem we have is that we need to create types on the fly which contain functions for Extend (and Dump, for convenience), and also whatever properties we want to dynamically add – the union of all of the properties on the original object and the objects we want to extend it with. I looked into a few alternatives, and a good compromise was the Microsoft.CSharp.CSharpCodeProvider – it allows me to dynamically assemble a syntax tree and get the framework to generate in-memory assemblies on-the-fly. The details are a little tedious, but it’s very possible to use this to create types on the fly – containing both the method calls we want for Dump and Extend as well as all of the properties we need. We can then instantiate our dynamically-created type and copy all of the values onto it. Our IDE will let us access our runtime-created properties and methods without compile-time errors, because our function is marked as dynamic – and the runtime binder can find all of the properties, as well as our .Extend and .Dump methods, because they’re actually on the type the runtime binder is looking at.

The minimum viable code to do something useful with CSharpCodeProvider looks something like this (note that this requires a couple of helpers you can find in the class linked below):

var compileUnit = new CodeCompileUnit();
var nameSpace = new CodeNamespace("MyDynamicNamespace");
var classType = new CodeTypeDeclaration("MyNewType");
classType.Attributes = MemberAttributes.Public;
compileUnit.Namespaces.Add(nameSpace);
nameSpace.Types.Add(classType);
var returnThree = new CodeMemberMethod { 
	Name = "ReturnThree", 
	Attributes = MemberAttributes.Public,
	ReturnType = new CodeTypeReference(typeof(int))
};
returnThree.Statements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(3)));
classType.Members.Add(returnThree);
var result = _cSharpCodeProvider.Value.CompileAssemblyFromDom(_compilerParameters.Value, compileUnit);
var compiledType = result.CompiledAssembly.GetTypes().Single();
LINQPad.Extensions.Dump(((dynamic)Activator.CreateInstance(compiledType)).ReturnThree());

This is a very round-about way to output the number 3, but it works!
PrintThree

I won’t reproduce the code here, but you can find all the details of the Object.Extend dynamic type creation in the CreateTypeUncached function in ObjectExtend.cs.

You might notice the word ‘Uncached’ there. When I first tried this approach, it was horrifyingly slow – I was using Object.Extend in a Select statement against a large IEnumerable, and generating many identical types. Throwing a quick cache into the mix based on the name and type of all properties of the type we need vastly reduces the number of calls to the compiler service and brings performance up to a tolerable level.

While I have glossed over some details, hopefully this explanation will give readers some background information to aid in reading the code. Please feel free to reach out to me on Twitter and let me know if parts of my explanation are hard to follow.

Object.Extend in C# for exploratory coding

LINQPad is great for exploratory coding. I use it all the time while I’m poking at APIs, and it’s completely replaced other scripting languages for me. I often find myself gradually building up result sets as I grope my way towards the result I’m looking for – and then I go back and re-factor it into something more presentable.

Unfortunately, building up these result sets can mean copying all the members of an old dynamic object into a new dynamic object.

var sites = siteData.Select(s => new { SiteName = s[0], SiteLink = s[1] });
var siteInfo = sites.Select(s => new {
    SiteName = s.SiteName, SiteLink = s.SiteLink, SiteInfo = SiteInformation[s.SiteName]
});
var siteContent = siteInfo.Select(s => {
    var details = GenerateDetailsSomehow(s);
    return new {
        SiteName = s.SiteName,
        SiteInfo = s.SiteInfo,
        SiteLink = s.SiteLink,
        SiteDetails = details
    }
});
// ... more of the same

That gets tedious fast. Wouldn’t it be great if C# had something similar to JavaScript’s Object.Extend? Well, maybe it can. I jumped into the “My Extensions” file in LINQPad and put together the following extension method:

public static dynamic Extend(this object firstObj, params object[] objs) {
    var result = new ExpandoObject();
    foreach (var o in new[] { firstObj }.Union(objs)) {
        foreach (var p in o.GetType().GetProperties().Select(p => new { Name = p.Name, Value = p.GetValue(o) })) {
            var props = ((IDictionary<string, object>)result);
            if (props.ContainsKey(p.Name)) props[p.Name] = p.Value;
            else props.Add(p.Name, p.Value);
        }
    }
    return result;
}

Now you can just call .Extend(...) on any object! So instead of having to create new objects all the time, you can do this:

var sites = siteData.Select(s => new { SiteName = s[0], SiteLink = s[1] });
var siteInfo = sites.Select(s => s.Extend(new {SiteInfo = SiteInformation[s.SiteName]}));
var siteContent = siteInfo.Select(s =>s.Extend(new { SiteDetails = GenerateDetailsSomehow(s) }));

That’s much easier to read (and quicker to write) than the first snippet! Unfortunately, it doesn’t work – the first call to our object.Extend(...) extension method is just fine, but the second call fails. Sadly, the way the runtime binder works means that our extension method won’t be available on the dynamics we create, so we can’t chain multiple calls using this approach.
RuntimeBinderException

I have solved that (and a number of other minor annoyances) and put it all together in a Nuget package called ObjectExtend. Sadly, if you don’t have a license for LINQPad you may have to download the package and reference it manually, but if you do have a license you can use the Nuget client built right in to LINQPad.

After adding ObjectExtend to our script, the chained script above works as expected:
ObjectExtendScreenshot

There you have it! Object.Extend in C#.

Please note this is a package focused on tinkering in LINQPad. It’s not the kind of thing you should be using while building maintainable, production-quality software.

Update: I had a few requests for source code, and it’s up on github now, but rather than making people who want to understand it dig through code I wrote up an explanation.