All posts by Lionell Pack

Joining TheTradeDesk (BOOTCAMP!)

Three weeks ago, I started my journey with TheTradeDesk. Working for a US-based scale-up has long been one of my dreams – I’ve come really close a few times, including turning down an offer to move to Seattle a few years ago due to a previous long-term relationship. But this time it finally happened – and the experience so far has been great.

My favourite local spot for a bit of (picturesque) remote work

I’ve really been enjoying inner-city living in Brisbane (especially some of the great remote-working spots I’ve found near my apartment), but TTD has proven very open to remote work. Their closest physical office is Sydney, and they’ve promised me trips there as well as to the US offices (COVID-permitting – and it looks like it finally will).

BOOTCAMP!

TheTradeDesk has a great initiative called Bootcamp. New starters don’t go straight to the team which hired them: instead they join their local bootcamp team (for around 8 weeks) to make sure they get a great on-boarding experience. This makes sure new starters get a chance to learn properly, rather than being thrown straight in to delivery-pressure mode.

I’ve been given plenty of time to learn: I’ve spent time, among other things, learning about our employee stock program, our T4 template code generation, our EF Core standards, and our extensive Engineering Meta repository – the source of truth for our extensive and open-to-contribution software engineering standards. I’ve also been exposed to real tickets – bootcamp participants are given a variety of non-time-sensitive tickets from across the organisation to give them real-world opportunities for hands-on learning.

I think this is a great way to onboard people, but I’m also really looking forward to taking on the full role I joined TTD for.

Staff Software Engineer – Front-end

Any readers who’ve known me from earlier in my career will recognise what a big shift it is for me to go to a front-end-focused role. I started my career as a back-end engineer with a well-developed allergy to anything front-end, but over the years I’ve done the needful – which has often been to make sure my team’s front-end engineering is keeping up with the trends.

This is a shift back onto the tools as well. I’m still developing my sense for the right ladder-vs-pendulum approach for my own career, but I’m confident I’ll enjoy getting back to coding for a while. I can’t keep myself away from writing code even when I’m meant to be focusing on being a manager, so this is a very comfortable shift for me.

A new sense of scale

One of the things I wanted most of all with this job change was the chance to learn – and I have that in spades. For a start, I’m faced with the largest code base I’ve ever worked on. More interestingly, the data involved creates genuine engineering challenges: TheTradeDesk deals with true web-scale (and I mean “a large fraction of the web“-scale) loads, dealing with upwards of ten million requests per second (that’s over 800 billion requests per day!).

Along with the engineering challenges, we are a rapidly-growing organisation, planning to significantly increase our software engineering team in the coming year. That creates plenty of opportunities for me to contribute to initiatives that will help grow our team – something I’ve been passionate about for a long time. Everything I’ve seen so far makes me very confident in recommending TheTradeDesk as a great place to work for people at any stage in their career.

If you’re interested in learning (don’t worry if that level of scale sounds intimidating!) maybe TheTradeDesk would be a good next role for you too.

Hiring

If any of that sounds interesting – please get in touch. Particularly if you’ve worked with me before! While TheTradeDesk has no current plan for a Brisbane office, I hope to thwart that lack of a plan by helping to recruit a heap of new engineers in Brisbane. I’ve worked with so many talented developers over the years, and I would love to see some of them join TheTradeDesk. We’re focused on hiring a balanced team, so I’m particularly keen to hear from early and mid-career software engineers looking for a career change (but there are roles for more senior engineers as well).

If you’re interested, start by getting in touch with me – you can find me in all the usual places (LinkedIn, Twitter, or @rophuine in some of the local developer Slack groups). We’re particularly looking for .NET and React developers, but we’re open to people with other skillsets (and we also need account managers, BI analysts, data engineers, and more!).

Make Your Job Suit You

Have you heard the advice “Do what you love, and you’ll never work another day in your life”? Or how about the Japanese concept of Ikigai, where you should find the intersection of what you love, what you’re good at, what the world needs, and what you can get paid for, and do that? (My apologies for what I’m sure is an awful attempt to sum up a philosophy in one sentence.)

I have. I read those advice pieces regularly, wonder if there’s room to refine where I’ve ended up, and then shrug and go back to the job I really like (most of the time) which pays the bills (as long as I’m not too extravagant). I’ve done OK by those measures – perhaps better than most. But I’ve always wondered whether it’s actually useful advice? As a trite throw-away line, I’m pretty sure it isn’t – but if you go a little deeper I believe there’s something useful there.

This article is going to go a little deeper – but first I need to set some context. “Show, don’t tell” doesn’t just apply to fiction. If you really want to skip the rambling and get to the actual point, scroll down to the section titled “The Actual Point”.

Writing My Own Job Description

I’m leaving my job shortly. As I write this, I have one week left. The job I’m leaving is a “management” role (it says so in my job title!) and I was the first person in the role. I didn’t start the team, nor was I the first person to have play a leadership role on the team, but I was the first one to play a purely-overhead, non-developer role (although when I started, there was an intent from both sides that I would be on the tools).

That means I got to write the job description.

To be fair, someone else wrote a job description before I took the role – but they hadn’t done the job, and they didn’t really have a deep understanding of what would be involved. The role started as an informal one: I was a senior consultant, and the job was a team lead one – but for a particular long-term team, rather than our usual temporary project teams. Later, the role was formalised, and I applied for it, interviewed, and officially became “Regional Manager” (to some minor applause and a significant number of The Office jokes). I won’t dig up the original job description, because it’s just not relevant – I don’t think I even read it when I applied. It certainly didn’t guide what I did. I knew what needed to be done: I’d already been doing it for years.

Having recently resigned, and needing to help find the right person to replace me, I sat down to describe what I actually do. I thought about what my one-up had said when I asked what he expected from me early on. “Happy clients, happy team.” Great advice, but a four-word job description probably won’t cut it! Perhaps reasonably, I started reviewing all the things I spend my time on. First at the day-to-day level, then at the monthly level, then the longer-term things. I looked at what changes I’d helped achieve, and thought about how I helped achieve them. If I can find someone who can keep doing the things I’ve been doing for the past few years, I thought, they’ll keep achieving the success I’ve been achieving!

Perhaps you can see the problem with that idea. Hiring someone to repeat the patterns of the past isn’t necessarily the best way to lead the team into the future. That’s not the lesson here, though – although I’ve just added that article to my backlog. I zoomed out a little – I described the goals I had pursued, and (to a lesser extent) the outcomes I’d been hoping for. I looked forward as well: what outcomes do we need in the future? They’ll be the real responsibilities of my replacement. Or perhaps they’ll even decide they’re not actually the right outcomes, and come up with new ones! That’s the real job.

Anyway, after a few more rounds of zooming in and out, reviewing and editing, and seeking input from my colleagues, I had what I thought was a workable job description. Only there was a problem.

There is no way I would have applied for that job.

I like my job. Quite a lot. I’m not sure I’d say I love it, but I work with some amazing people, I get to see a lot of different clients (we’re a consultancy, after all), and I have the autonomy (for the most part) to organise the team as I see fit. I pass a lot of that autonomy down to my team, to organise themselves around the clients and problems they’re facing. That wasn’t the problem. The position description didn’t make it sound like an unlikeable job.

It made it sound like an unapproachable job.

Echoes of Imposter Syndrome

“I’m not qualified for my job!” is a thought I’ve had plenty of times throughout my career, but not recently – and I didn’t quite think that now. I seem to have succeeded in the role. The things people have said (particularly since I resigned) and the metrics of the team I manage suggest I’m doing pretty well.  But I know that if I’d seen that job ad a few years ago, I probably wouldn’t have applied. It was too intimidating. If I saw it today … the same. It’s too much. I’m not ready for a job like that!

I had two adjustments to make. One to my own expectations of myself. I can do a job like that! And while the expectations of the role I’m going in to won’t be quite so lofty, there’s a good chance I’ll turn it into something just as big. I’ll broaden my responsibilities, I’ll fine-tune the role to suit me, and when I leave people will (hopefully) talk about how I have big shoes to fill. That leads me to the second adjustment I needed to make.

I added a sentence to the beginning of the position description. “The right candidate will have the opportunity to shape the role to fit their strengths, and delegate some of the following responsibilities and goals to other members of their team.” I think all leadership roles carry this implication, and we might find more people would be comfortable in stepping in to leadership (and be better at delegating) if we made it more explicit up front.

That’s still not the main point of this article, but I have finally arrived at it.

The Actual Point

Whether or not you’re in leadership, you should find ways to make your role suit you.

The real insight I came to is that the role description only looks the way it does because I’ve shaped the role around my own strengths. I hunted around looking for ways we could be better, and where I found something I would be great at, I added it to the list of things I do. Where I found something that would be highly valuable that I would be bad at (or was too far from my sphere of influence), I tried to get someone else to do it. Had another person been in the role instead of me, and they’d actively shaped it to suit themselves in the same way I did, the role would probably look different – and they’d probably like it just as much as I do.

That’s fine advice, Lionell (you might object), but you’re a manager! Individual contributors (ICs) don’t have the authority to do that!

It’s true that I have more authority than I did as an IC, but I also have a broader variety of larger-scale outcomes I’m on the hook for. I’ve been shaping my role to suit myself for most of my career, and it hasn’t gotten any easier now that I’m in management. Whether you’re a manager or an IC, it’s tough – but it’s also extremely rewarding, and I think everyone should try to do it. You won’t always succeed (I’ve certainly had jobs where I couldn’t make it work), but you won’t know if you don’t try.

It’s Not All About You

It’s not just personally rewarding. It’s likely to be great for your team, as well. In one role where I was a senior developer or a tech lead (to this day I’m not quite sure), I advocated for 20% time – where developers get the freedom to spend 20% of their time on side projects. We didn’t get 20% time – but we did get 10% time, and one of my first 10%-time projects was to build a proof-of-concept user interface for our flagship product based around Google Earth. It was extremely well-received, and while we ended up pivoting to Google Maps instead, that layout became core to the product for many years to come (it might still be that way).

In another IC role, I put my hand up for presales work every single time the sales team needed an engineer. I enjoyed it, I built some great skills, and I helped us close some deals which we might otherwise have missed out on! Or maybe I lost us some deals. At least, I helped make sure engineering had a voice at the table early on. My goal was not to win or lose deals, but to help shape opportunities into successful projects. It meant my engineering work was regularly interspersed with presales work, and I loved the variety.

Back to my current role: that presales experience came in mighty handy. I’ve done presales work on just about every new client my team has picked up in the last three years. I’ve loved that part of my job! But if I’d hated presales, I wouldn’t have done all that presales at a previous job, and I would probably have left it to others. Hopefully I would have found someone on my team interested in it.

Advice for Individual Contributors

If you’re an IC, I can’t tell you how much of your current role will be negotiable. Perhaps it’s only 10%. Perhaps it’s 100%. If you keep scanning your sphere of influence for things which need doing, pick the things you think you’d be good at, and ask to do them, you’ll find out. If you don’t know how to find things which need doing, ask your manager! Tell them that you’re looking for opportunities for expanded responsibilities, and tell them what kind of things you think you’d be interested in or good at. Most managers I’ve met have too much on their plate and will be happy to offload some of what they do – but watch out for them giving you busy work. A great manager will give you opportunities which you can learn from and feel good about. A poor manager will just give you the menial tasks they hate.

One thing to watch out for: if you’re very early in your career, be wary of a trap I’ve seen some people fall into. Don’t let yourself end up as a gopher (“Hey [name], go’fer coffee” / “hey [name], go’fer [other task]”). You need to be learning the fundamentals of your craft, and that means spending plenty of time on the tools. A little extra responsibility is a good thing, but you shouldn’t over-do it and you shouldn’t let yourself be de-valued by getting all the menial tasks. Picking up coffee for the team is the kind of thing that should probably be a shared or rotated responsibility, not a task given to the most junior IC on the team.

Advice for Managers

If you’re a manager, you (hopefully) already have the autonomy to build your role into something you will like (or even love). The more senior you become, the more your responsibilities should be focused on achieving outcomes with your team, rather than doing specific things personally. If you’re not already taking advantage of this – get on it!

There are three things I’d recommend a new manager flexing their autonomy-muscles watch out for:

First, don’t lose sight of the actual goals you’re responsible for. This isn’t a license to just do whatever seems fun and forget your responsibilities! Even more importantly, don’t lose sight of the fact that one responsibility every manager has is to build up their team. Don’t keep all the fun jobs and delegate all the rubbish ones: that would be failing in that responsibility.

Second, be aware that you’re part of a broader organisation, and there are other people around you with their own goals and passions. If you think you’d absolutely love to run a customer-facing product forum, but there’s already a community engagement manager running a customer-facing product forum, this isn’t a license to go start your own competing one! Instead, see if there’s a related gap: perhaps there’s an opportunity to launch a public Slack or Discord group focused on your product. Perhaps there’s just no need, and you’ll need to scratch that particular itch on your own time (maybe by finding an open source project or not-for-profit group in need of a spare-time community manager).

Third, be aware of the time costs of the things you take on. If you’re the CTO, spending half your time moderating a community forum is probably not what you’re being paid for. Keeping an eye on trends in a community forum and making sure you understand how your users see your product might be, though! If there’s no community forum right now and you think there should be, delegate. You get bonus points if you can find someone in your organisation who would love to take on that responsibility for you.

Finally…

My final advice is to take your time over this. You’ll get the best outcomes if you find things to do which not only help you like your job more, but let you learn new things and help the team around you achieve better outcomes. If I had to choose, however, I’d focus on things which make me happier. You might be surprised just how valuable it is to your team for you to be happy at work!

New LINQ Method: ToRandomCollectionType<>()

If you’ve spent time in the C# world, you’ve probably encountered (and possibly asked) the question: “Should I use .ToList() or .ToArray()?”

The answer, of course, is that it’s a false dichotomy: I’ve written before about how you should pick the right data structure for the job. But what if you really don’t care? What if you’d prefer to just roll the dice and let the computer pick for you? Is that possible?

Of course it is. First, we need an extension method:

public static class Extensions {
    public static IEnumerable<T> ToRandomCollectionType<T>(this IEnumerable<T> collection) {
        throw new NotImplementedException();
    }
}

The first thing that’s going to have to do is pick a collection type. Let’s load all of the types we can find which implement IEnumerable:

var types = AppDomain.CurrentDomain.GetAssemblies()
                     .SelectMany(assembly => assembly.GetTypes())
                     .Where(t => typeof(IEnumerable).IsAssignableFrom(t))

We actually want to be pickier than that – because we need to work out how to actually instantiate one of these types. So let’s make sure all of our candidate types have a constructor which accepts a single IEnumerable of the passed-in collection type:

                     .Select(t => { try { return t.MakeGenericType(typeof(T)); } catch { return null; }})
                     .Where(t => t != null && t.GetConstructor(constructorParamTypeArray) != null)

The try/catch block there is to deal with classes like ConcurrentDictionary, which require multiple generic parameters even though they implement IEnumerable. Is there a better way to check? Probably! But we’re not writing production-grade code here, because .ToRandomCollectionType<>() is fundamentally production-unsuitable anyway.

The next step is to randomly select one of our candidate types and grab a reference to the right constructor.

var type = types[_rnd.Next(0, types.Count-1)];
var constructor = type.GetConstructor(constructorParamTypeArray);

Now all that’s left is to invoke the constructor and return the result!

return (IEnumerable<T>)constructor.Invoke(new [] {collection});

Does it work? Absolutely! Most of the time, anyway. Here’s the outcome of feeding { 1, 2, 3 } into it a few times:

ToRandomCollectionType

Much like the FileChangeMonitor hack I blogged about, this is unsuitable for anything production-like. Please treat it as the humourous hack it is and don’t use it for anything important!

Hang on though, we’re not done. You should always test your code. Like, with automated tests.

public class ItShouldUsuallyWork
{
    [Test]
    public void TryingAFewTimes_SometimesWorks() {
        var inputSet = new List<int> {1, 2, 3};
        int successCount = 0;
        for (int i = 0; i < 10; i++) {
            var result = inputSet.ToRandomCollectionType();
            try {
                inputSet.Should().BeEquivalentTo(result);
                Console.WriteLine($"Got a {result.GetType()}");
                successCount++;
            }
            catch {
                Console.WriteLine("Looks like that one failed.");
            }
        }
        successCount.Should().BeGreaterOrEqualTo(5);
    }
}

The code is available on GitHub, and there is a NuGet package so this lovely little extension method is never more than a quick “Install-Package ToRandomCollectionType” away.

If you enjoy a little C# whimsy, you might like to read about my Object.Extend(…) implementation in C#.

How I Learned to Stop Worrying and Love Consulting

A few months ago I wrote about my motivation problems since switching to consulting. I think this was a somewhat natural thing to feel: I’ve built quite a few in-house software teams over the years, and consulting companies were my natural enemy. They hired many of the top people in my local area, and then wanted to sell them back to me at a substantial mark-up. What jerks!

A little over a year ago, I joined the dark side and became a jerk consultant. My former enemies are now my friends, and I’m the one being sold back to in-house teams for profit. I learned to live with that fairly easily: if we weren’t giving our clients value, they’d stop paying us – and we get heaps of repeat business. No, the economics of it weren’t the problem.

The problem was Mission.

I’ve always thought of myself as strongly mission-driven. When I worked in finance, I believed that what we were doing was making life better for our customers. When I worked in environmental engineering, I believed that we were helping our clients get the most environmental bang for their buck (my more cynical side occasionally observed that we were helping them spend the minimum possible to avoid trouble with the regulators). When I worked for a hospital group, it was easy to think of patient care as a worthwhile mission, because it is.

Oddly, what I’ve since learned is that my mission is much less available for rent as a consultant that it ever was before. Protecting the environment wasn’t my core mission before my pay check came from Pacific Environment, and it hasn’t been since I left. Patient care wasn’t my mission before I joined Ramsay Health Care, and it isn’t now.

Are those all worthwhile missions? Absolutely. But in hindsight, the mission I took on was entirely driven by who was paying my salary.

Don’t get me wrong. I’m not unhappy with my career choices, and I think I made a positive impact on the world (and hopefully my local software industry) on my way through. But joining a software consultancy is a neat way to de-couple my mission from my salary. I still have a core mission, but now it’s much more in concert with who I am as a person. Readify’s mission is all about using technology to make the future better for everyone – and that speaks very deeply to me. I’m a technologist at heart. My secondary mission may be for sale, but the core goal of myself, my colleagues and my employer remains the same. That’s not just something I can live with: it’s something I can embrace.

So, that’s the mission aspect sorted – but I’m not the moony-eyed idealist I might like to think I am. You could see all of this talk about “Mission” as an exercise in justifying my current choice – just like I justified all of my previous employment choices. What about the economics of it all? That’s where my opposition to software consulting began, and it’s not so shallow-rooted to be slain by a simple mission statement.

Well, here’s where I needed to shift my thinking a little. I needed to start thinking about industry sustainability.

Before I get to the point, bear with me while I give you a quick tour through some highlights of my career. My first significant production responsibility came a few years in, when I was a Senior Software Engineer at a successful FinTech start-up. I reported directly to the Managing Director (who was from a finance background). Some years on, I found myself a Senior Software Engineer and Team Lead for an environmental firm, where I reported to a variety of people (including the CTO, and occasionally the CEO) – who were mostly from an environmental engineering background. During my time in the hospital industry, I was again in a leadership role – being such a large multi-national, there were a few more layers involved, but my boss’s boss’s boss (and most of his colleagues) were from a doctoring or nursing background.

I’ve seen this pattern repeated across many industries: flatly-structured software teams with little or no opportunity for career advancement, reporting to non-software people. Reporting to non-software people isn’t intrinsically bad: some of my all-time favourite people are non-software people (I even married one!). However, I don’t think it’s a sustainable model for our industry. I’ve seen far too many software developers come to realise that their best opportunity for advancement was to move sideways (out of software) – and I think that’s terrible. It’s almost become a truism that “software is eating the world”, and certainly one of the highest-value-creating things you can do today is to build software. Why would I help perpetuate a model which pushes our best and brightest to go and do other things?

Participating in a software consultancy like Readify is a way to break that mould. I’m in a leadership role, but I still have plenty of room to grow in my career without abandoning my core mission and my experience as a software engineer. If I decide that my current business group (we call it “Managed Services” but it’s really a combination of production engineering and feature development, driven in a Kanban style) doesn’t suit me, we have plenty of other flavours of software development I can switch to. If I really want control of my own destiny, I’m learning the business of software consulting – so I can always go and do software on my own terms. Mid-career, I have software developers reporting to me, and I report to other people from a software background – all the way up. I can move up the chain, or try to disrupt it – and either way, I’ll be participating in an industry model which mentors and builds software careers, from fresh graduate to C-level executive. Perhaps most importantly, I’ll be participating in a sustainable model – which keeps people in software, because it’s a great career with opportunities in line with any other you could choose.

Software consulting isn’t the only way to achieve this. Software-focused product companies offer the same potential sustainability. I can’t even rule out that my career might take me back to working directly in other industries – but if I do, it will be with a new understanding of the importance of a sustainable software industry, and an eye to ensuring that the technologists among us never feel the need to go do something else to advance their career.

Enduring Software Development Myths

I’ve been asked more than once about software development myths, and I’ve run into plenty over the years. Here is a (non-exhaustive) list of some of the myths I’ve seen people believe. I’ve tried to focus on enduring myths, rather than short-term fads (like “Blockchain is a broadly applicable technology”).

Myths Management Believes

adult-businesswoman-company-325924

1. Every manager has read ‘The Mythical Man-Month’ and thinks they understand that adding more developers to an already-late project tends to make it later. However, they will still do it, apparently believing that some combination of proper management, having good people, and the specifics of a project mean it doesn’t apply in their situation.

2. Management also has a tendency to believe that large, complex software systems get ‘finished’ and then become a cash-cow, bringing in an ever-increasing revenue stream for fixed costs. Even if they do account for increasing support and maintenance costs, they rarely consider that continuing innovation and development is required to keep a product competitive.

3. Managers sometimes believes that the primary role of a developer is to translate specification into code. They don’t understand the numerous design decisions involved, the trade-offs being made, the non-functional requirements being considered. I have never once in my career encountered a specification which considered the sort of logging detail required to diagnose the types of system failures which might occur – and yet that is a key component of writing good code. Every developer runs into minor details from day to day which require assumptions or decisions to be made – and a good developer should be tracking those, communicating them, and ensuring the right people are involved in the right decisions, while being shielded from the bulk of less-important ones.

4. I have frequently run into managers who believe that their developers are not politically savvy, or that they all have poor networking skills. This can lead them to interact with other parts of the organisation in a way which under-estimates the contribution of the development team (see 3). When managers do this, their team does find out about it, and it creates resentment.

5. Managers sometimes think that developers within their team are roughly fungible. Managers usually understand that different developers have different areas of expertise and levels of experience, but they often underestimate the impact. They don’t realise that it can have not a 2x effect, but a 5x or greater effect – and that effect is not limited to time, but also to quality (but see myth #2 under “Myths Developers Believe”, below).

6. “All platforms, languages, and third-party systems are roughly equivalent” or “This specific platform, language, or third-party system will fix all of our problems”. I have run into both over the years. The former leads to upper management thinking the selection of a third-party provider such as a cloud host or database platform is purely a business decision and will have no technical impact. The latter can result in management embarking on costly technical projects to no benefit (“we have to re-write our entire codebase into [language]” or “we need to move all of our data into MongoDB”).

Myths Developers Believe

devops-photo-m1. Developers often believe that management doesn’t have anything to contribute, or that they’re useless or unskilled. Notwithstanding the above myths, managers can make significant contributions and many of them do really know what they’re doing. The good ones shield developers from distraction, ensure they have proper equipment and software, manage outwards to get realistic (and, importantly, low-stress) deadlines, communicate early and mitigate the effects of missed deadlines, justify budgets and pay increases, bring in appropriate consultants to address short-term skill gaps, ensure meetings minimise disruption, and do countless other things to make projects run more smoothly.

2. Developers often believe in 10x or ‘Rockstar’ developers. Some developers can be significantly more productive than others, but this is usually just a matter of experience and knowledge. It is also often specific to particular topics. For example, I used to do all of my team’s low-level network programming – many on the team considered it an impenetrable field and thought they weren’t capable of doing it. A while ago, though, I took one of our juniors aside and spent some time teaching him what was involved. He is now nearly as productive as me in that area, as long as he doesn’t go too long without working in it. Once he’s relearned the skills a few times, they’ll become a permanent part of his skillset. He’ll then be a 10x programmer – but only when working on low-level networking code, and only when compared to a developer who hasn’t spent much time on that specialty.

3. “Most developers are much better than me.” Known as Imposter Syndrome, developers are particularly prone to it, due to developer myth 2. It is especially common in generalist-type roles, where a developer may go long stretches without working on the sorts of things he or she is particularly good at. Instead of treating these times as opportunities to develop new specialties, they can instead look back at periods of much-increased creativity and think that they’re a bad developer who spends most of their time being unproductive. In reality, they are normally-productive for much of their time, and super-productive when the thing which needs doing is an excellent match for their experience and knowledge.

4. I don’t quite know how this one crops up, but many developers think having a high reliance on tools is bad. I have encountered developers who refuse to use automated deployment systems (“I want to understand what’s going on, and I can’t do that if it’s automated”), refactoring tools (“it’s safer to make each change manually”), and even syntax-highlighting and code completion (“I know my language and my libraries. What if one day I’m stuck using vi on a green-screen linux box from the 70s? Where will I be then if I’m used to these fancy tools?”). I call this concrete-pill syndrome, and I suspect it has something to do with wanting to look capable of handling everything yourself. In reality, these tools (and many others) are great levers, allowing developers to increase productivity and quality.

5. “As a developer, it doesn’t matter if I have poor people-skills, because in developer roles I can ignore office politics anyway.” This is a recurring one, and it doesn’t just hurt your long-term career – it hurts your current project, it affects the sorts of projects you get to work on, and it hurts your team. I try to shield my team from the need to play politics, but I also try to let them see just how much of it I have to do. Not playing office politics well enough can leave you stuck with a bad manager (see some of the earlier myths about the contribution a good manager can make), or it can leave you stuck with a really bad external technical decision. It can leave you stuck with a really bad hire.

Earlier in my career, someone from upper management asked me: “Of the candidates for the developer role you rejected, who was the least bad?” I made the mistake of telling him, and I got stuck with that bad candidate on my team right through his probation period. I should have known to respond with “They were all unsuitable, let’s line up some more interviews”.

6. Developers early in their career sometimes think they are much better than experienced developers. I’ve heard it called ‘God’s Gift to Coding’ syndrome. It happens because they believe they are better with new technology, and new technology is more productive, and so they are worth more than someone who is too out-dated in their thinking or is stuck on old platforms and techniques. While some older developers do remain focused on older systems, they have still built up a depth and breadth of experience which is valuable. However, many developers continue to be intrigued by new technology as they get older, and keep their skills as (or more) current than many juniors.

7. This is a two’fer: “Domain experts outside software aren’t really that smart” and “domain experts have magical knowledge of an esoteric field I can never begin to understand”. The reality is that the domain experts you work with – whether they are accountants, scientists, engineers, or sales people – have years of learning, training, and industry experience which is highly valuable to your team. As a developer, you need to respect that – but you also need to be a sponge, learning what you can about the domain. If you don’t understand what you’re building, you will fail to consider important things. If you don’t rely on your experts for their professional knowledge, though, your project will be built by amateurs who happen to be good at software.

(I originally published this list on Quora but wanted to get it on my own blog as well.)

Consulting and Motivation

Caveat: I’m very hesitant to post this publicly. Please read it in the sense I’ve written it. I’m being vulnerable in the hope of opening up discussion on a topic which I can’t see people talking about, but which I think is probably important.

I’ve spent most of my software career as part of a product team. Whether I’ve been writing code (mostly), managing people (when needful, and more often lately), mentoring, or doing any of the many little side-roles which are needed to build a successful product, I’ve always been able to rely on the fact that everyone I’m working with has the same big-picture mission.

I’m not talking about “producing great software” here. I’m talking about the alignment between business, and product, and people – over the years it has ranged from “let people put a $2 coin into a kiosk and use the internet” to “produce weather forecasts to help mines make better decisions” to “provide better patient outcomes in hospitals”. That shared mission has always been fundamental to my own sense of purpose and motivation. It’s what makes me feel worthwhile as a person, and it helps me justify to myself the salary I get paid – I would hate to feel like I’m being paid money but not adding value!

A bit less than a year ago, I moved from product teams to consulting – and the rug has been pulled out from under my feet.

It probably doesn’t help that I’ve moved straight into a leadership role. I don’t have one client for weeks or months at a time – I have somewhere around 20 clients simultaneously, as well as leading a team of consultants and being part of the overall leadership of my state and my national group. My usual source of motivation – that shared mission – hasn’t just been diluted. It’s been shredded, processed, ground up, kneaded into dough, cooked, and handed back to me in a form I just don’t recognise.

I needed to find a new source of motivation.

I went looking online, and there’s not a lot out there. I read listicles about how to stay motivated as a consultant – “Exercise!” “Don’t sweat the small stuff!” “Take time to recharge!” “Just believe in yourself!”. I read (or mostly listened to) books about consulting, and they gave me all sorts of strategies – for being a better consultant. I just haven’t been able to find much (anything?) addressing my motivational problem.

I’ve had a string of successes, mostly great outcomes, and one or two things which haven’t gone well. I’ve watched the various strategies we’ve used to motivate people with lots of interest – hoping I’d find something to get my gears engaged. Positive call-outs, monetary rewards, team-building exercises. They’re good strategies. They make me feel good in the moment. It’s a great team – some of the smartest people I’ve ever worked with. That can be hard to deal with in itself, but I’ve mostly moved on from my imposter syndrome to something a little more complicated, so that’s OK.

None of these things has been a solid substitute for the fundamental sense of purpose I used to get from being part of a team with a mission. A mission beyond just “build great software” – that’s just a means to an end. A mission like “create better hospital outcomes for patients”. That was a good one.

I have clients with missions now. Some of them even believe in their own missions, and occasionally they’re missions I care about too. But I only get to care about my clients’ missions sometimes – and other times my mission is aligned with my own team (whose mission is somewhere between “build great software” and “keep the client happy”), and sometimes I’m worried about the broader group or consultancy. Sometimes I’m aligned with the sales team, and sometimes I’m opposed to them. Sometimes I’m focused on a client whose mission I disagree with. Success, then, means making the world a worse place (from my point of view).

The best advice I’ve found so far has been this: find a sense of pride in the quality of your work, and the reputation of your team. Good advice, I’m sure, and probably very fulfilling for some.

I haven’t had any luck decoupling the quality of my work from the outcomes and the team and the mission. Quality work is just another means to an end: accomplishing the mission.

I normally write blog posts because I feel like I’ve got some knowledge to share, but the more leadership I do, the more that I understand it’s not always about dishing out nuggets of wisdom from on high – it’s about facilitating discussions, and helping create understanding from our shared experiences. So here I am, ending a blog post with the same uncertainty that I started it with.

Please, reply here, or reply to this tweet. Whether you’re a consultant now, or you have been one in the past and struggled with the same thing, or you’re on a product team and get your motivation from another source. Whether you have an answer, or a different perspective, or just more questions – let’s talk about this topic. One of the biggest industries in my city is consulting, and I feel like this is a big topic which just isn’t talked about. Let’s start talking.

How my C# Object.Extend implementation works

I’ve had a few people ask me to explain how my C# Object.Extend implementation works – so here I am, going into more detail than I expected people would be interested in.

Making Object.Extend work seemed straight-forward at first. You can read my initial code in my original ObjectExtend post. The problem started when I tried to chain two calls together: I immediately got a RuntimeBinderException. Unfortunately, for a really good reason (the difficulty of discovering the right namespaces to search), the code responsible for runtime method binding doesn’t look for extension methods. That means that when the first call to .Extend returns something marked as dynamic (regardless of what actual type it returns), you can’t use any extension methods on the result. Attempting to call either .Extend again or .Dump (another common extension method in LINQPad) results in the dreaded RuntimeBinderException.

The usual usage of the dynamic keyword is to let us return something like the ExpandoObject type, like this:

dynamic ReturnSomething() {
	return new ExpandoObject();
}

Fortunately, the dynamic keyword in C# essentially just tells the compiler to engage duck-typing mode – it doesn’t actually tell the run-time anything. So there’s nothing stopping us from doing this:

dynamic ReturnSomething() {
	return new List<string>();
}

The compiler (and your IDE) will allow you to try to call anything at all on the return value of ReturnSomething, and hand everything off to the runtime binder. This means that you can’t use extension methods on anything returned by a method marked dynamic, even if you actually return a regular non-dynamic type.

dynamic ReturnSomething() {
	return new object();
}
ReturnSomething().Dump();

Nope. Instant RuntimeBinderException.

There’s one situation which will give us exactly what we want – if we mark the .Extend function as dynamic, but the type we return has our Extend method built right in to it. This means our IDE (LINQPad, in my case) will allow us to access any of the dynamic properties we build up on our types over successive calls to Extend, and the runtime binder will be able to find the methods because they’re right on the type we return – we’re not relying on extension methods!

I initially thought I could create a subclass of ExpandoObject with calls for Extend and Dump, but it turns out that not only is ExpandoObject a sealed class, it’s also a bit special in other ways – so that was a no-go.

So now the only problem we have is that we need to create types on the fly which contain functions for Extend (and Dump, for convenience), and also whatever properties we want to dynamically add – the union of all of the properties on the original object and the objects we want to extend it with. I looked into a few alternatives, and a good compromise was the Microsoft.CSharp.CSharpCodeProvider – it allows me to dynamically assemble a syntax tree and get the framework to generate in-memory assemblies on-the-fly. The details are a little tedious, but it’s very possible to use this to create types on the fly – containing both the method calls we want for Dump and Extend as well as all of the properties we need. We can then instantiate our dynamically-created type and copy all of the values onto it. Our IDE will let us access our runtime-created properties and methods without compile-time errors, because our function is marked as dynamic – and the runtime binder can find all of the properties, as well as our .Extend and .Dump methods, because they’re actually on the type the runtime binder is looking at.

The minimum viable code to do something useful with CSharpCodeProvider looks something like this (note that this requires a couple of helpers you can find in the class linked below):

var compileUnit = new CodeCompileUnit();
var nameSpace = new CodeNamespace("MyDynamicNamespace");
var classType = new CodeTypeDeclaration("MyNewType");
classType.Attributes = MemberAttributes.Public;
compileUnit.Namespaces.Add(nameSpace);
nameSpace.Types.Add(classType);
var returnThree = new CodeMemberMethod {
	Name = "ReturnThree",
	Attributes = MemberAttributes.Public,
	ReturnType = new CodeTypeReference(typeof(int))
};
returnThree.Statements.Add(new CodeMethodReturnStatement(new CodePrimitiveExpression(3)));
classType.Members.Add(returnThree);
var result = _cSharpCodeProvider.Value.CompileAssemblyFromDom(_compilerParameters.Value, compileUnit);
var compiledType = result.CompiledAssembly.GetTypes().Single();
LINQPad.Extensions.Dump(((dynamic)Activator.CreateInstance(compiledType)).ReturnThree());

This is a very round-about way to output the number 3, but it works!
PrintThree

I won’t reproduce the code here, but you can find all the details of the Object.Extend dynamic type creation in the CreateTypeUncached function in ObjectExtend.cs.

You might notice the word ‘Uncached’ there. When I first tried this approach, it was horrifyingly slow – I was using Object.Extend in a Select statement against a large IEnumerable, and generating many identical types. Throwing a quick cache into the mix based on the name and type of all properties of the type we need vastly reduces the number of calls to the compiler service and brings performance up to a tolerable level.

While I have glossed over some details, hopefully this explanation will give readers some background information to aid in reading the code. Please feel free to reach out to me on Twitter and let me know if parts of my explanation are hard to follow.

Object.Extend in C# for exploratory coding

LINQPad is great for exploratory coding. I use it all the time while I’m poking at APIs, and it’s completely replaced other scripting languages for me. I often find myself gradually building up result sets as I grope my way towards the result I’m looking for – and then I go back and re-factor it into something more presentable.

Unfortunately, building up these result sets can mean copying all the members of an old dynamic object into a new dynamic object.

var sites = siteData.Select(s => new { SiteName = s[0], SiteLink = s[1] });
var siteInfo = sites.Select(s => new {
    SiteName = s.SiteName, SiteLink = s.SiteLink, SiteInfo = SiteInformation[s.SiteName]
});
var siteContent = siteInfo.Select(s => {
    var details = GenerateDetailsSomehow(s);
    return new {
        SiteName = s.SiteName,
        SiteInfo = s.SiteInfo,
        SiteLink = s.SiteLink,
        SiteDetails = details
    }
});
// ... more of the same

That gets tedious fast. Wouldn’t it be great if C# had something similar to JavaScript’s Object.Extend? Well, maybe it can. I jumped into the “My Extensions” file in LINQPad and put together the following extension method:

public static dynamic Extend(this object firstObj, params object[] objs) {
    var result = new ExpandoObject();
    foreach (var o in new[] { firstObj }.Union(objs)) {
        foreach (var p in o.GetType().GetProperties().Select(p => new { Name = p.Name, Value = p.GetValue(o) })) {
            var props = ((IDictionary<string, object>)result);
            if (props.ContainsKey(p.Name)) props[p.Name] = p.Value;
            else props.Add(p.Name, p.Value);
        }
    }
    return result;
}

Now you can just call .Extend(...) on any object! So instead of having to create new objects all the time, you can do this:

var sites = siteData.Select(s => new { SiteName = s[0], SiteLink = s[1] });
var siteInfo = sites.Select(s => s.Extend(new {SiteInfo = SiteInformation[s.SiteName]}));
var siteContent = siteInfo.Select(s =>s.Extend(new { SiteDetails = GenerateDetailsSomehow(s) }));

That’s much easier to read (and quicker to write) than the first snippet! Unfortunately, it doesn’t work – the first call to our object.Extend(...) extension method is just fine, but the second call fails. Sadly, the way the runtime binder works means that our extension method won’t be available on the dynamics we create, so we can’t chain multiple calls using this approach.
RuntimeBinderException

I have solved that (and a number of other minor annoyances) and put it all together in a Nuget package called ObjectExtend. Sadly, if you don’t have a license for LINQPad you may have to download the package and reference it manually, but if you do have a license you can use the Nuget client built right in to LINQPad.

After adding ObjectExtend to our script, the chained script above works as expected:
ObjectExtendScreenshot

There you have it! Object.Extend in C#.

Please note this is a package focused on tinkering in LINQPad. It’s not the kind of thing you should be using while building maintainable, production-quality software.

Update: I had a few requests for source code, and it’s up on github now, but rather than making people who want to understand it dig through code I wrote up an explanation.

Measure (At Least) Twice

20180218_143538 - SmallBear with me here. I promise this is about software – but first I need to talk about building.

I have several uncles and uncles-in-law who are builders, and when I was younger I ended up helping out with minor jobs here and there. One of the most recurrent mantras was this one:

“Measure Twice, Cut Once.”

The idea, if it’s not immediately obvious, is that you double-check your measurements before committing to an action which is costly or hard to undo. Once you’ve cut that beam or board, you can’t un-cut it – the board will stay cut despite your best efforts to turn back time. If you cut it too small, you can no longer use it for what you planned to. If you cut it too big, you have to try again. Measuring is cheap – cutting is expensive. Measure twice, cut once.

I consistently see software engineers fail to even measure once.

I once reviewed a pull request from a developer who had decided to optimise part of our code base which performed intensive weather modelling. He had spent a week on it – and he had spent that time re-writing a large number of string plus operations to use StringBuilders. In some cases this can provide some performance benefit – but the re-written code was in the setup and teardown phases of the modelling. Setup and teardown took only seconds, while the modelling itself could take minutes or hours. Worse, this developer had introduced several bugs into previously-working code; but that would make this an anecdote for a blog post about the importance of testing and safe refactoring, and this is about measuring, so let’s focus on a different “worse”. Not only had he optimised the wrong part of the code base, but because he hadn’t measured performance, he couldn’t demonstrate that he’d actually made anything faster (aside from some vague assertions that StringBuilder is faster than string-plus, which isn’t always true).

I didn’t even need to measure to know that his change wasn’t going to make things significantly faster – the code wasn’t in a loop, and I knew that the overall modelling time was long. Shaving off microseconds at a time in a scenario like that is a losing game. The other reason I knew for certain that his change wasn’t going to pay dividends was that I had already spent quite some time improving the efficiency of that setup process. We had benchmark tests around parts of that code, and we had made significant wins. Most of them were had by fixing the .GetHashCode and .Equals methods of a custom coordinate class we were using. We were wrangling a fairly large amount of data into data structures needed by the modelling code, and set operations like .Contains were taking far too long. It turns out both .GetHashCode and .Equals were creating additional instances of the class to perform certain comparisons, and so we were churning the garbage collector as well as performing far too many operations. Some careful tweaks to those methods – operations performed many times within nested loops – took our setup time down from minutes to seconds. We knew that we’d made things better because we were measuring – but the benefits of measurement go much deeper than just knowing how you did.

All developers fall somewhere on the spectrum between, roughly, terrible and genius. I’m not sure where on that spectrum I fall (I suspect it varies depending on what sort of programming I’m doing), but I am pretty sure that I wasn’t good enough to read through thousands of lines of C# across many different files and realise, through pure mental analysis, that we had a performance problem in our coordinate comparison code. Even if I had noticed that it wasn’t efficient, I wouldn’t have been sure how much the problem was contributing to our run time (as compared to, say, our inefficient string concatenation). Instead, I found out by measuring.

In this case, I used one of the simplest of all code profiling techniques: the pause button in my debugger. Simply run your code, and sometime during the long-running process, hit pause. The line you’ve broken on is extremely likely to be part of the problem. Typically, it will fall into one of two categories:
1. A line which is executing many times. In this situation, something further up the stack trace is calling the code you’re looking at inside a large (or nested) loop.
2. A line which is taking a long time to execute. Perhaps it’s hitting disk or network and waiting a long time before continuing.
There is a third (statisically less-likely) option: you’ve just happened to break execution somewhere that isn’t part of the performance problem. That’s why you measure (at least) twice. Break your long-running operation a few times, and compare the stack traces. If your stack dump consistently looks the same, you’ve found your performance hot-spot.

That’s one of the simplest ways of measuring your code, and it may suffice for tracking down hot-spots, but there is a lot more to measurement than just hitting pause in your debugger. Do you suspect you have a database problem? Don’t go hunting through code looking for your favourite anti-patterns. Instead, measure the number of database round-trips your operation makes. Use your database tools to identify expensive queries – and even once you’ve found them, don’t necessarily just start staring at code. Are you missing an index? There are tools which can tell you. Even if the query is bad, looking at the query plan will tell you exactly which parts of the query are causing the problem, so you don’t need to guess.

Maybe you are hitting multiple services and combining the results? If you’ve just learned about the async keyword, you might be tempted to dive in and start making everything async – but if almost all of the delay is coming from a single call, it might not help much (you’d be better trying to find out why that service is so slow). If you really are dealing with multiple similar-cost calls which you combine after they’re all complete, async may be a good investment of your time – but you won’t know until you’ve measured.

For one weather-forecasting algorithm I measured, I didn’t turn up anything obvious. I measured and measured some more, and eventually I discovered we were utilising the CPU cache poorly – the bulk of our algorithm time was spent fetching data from main memory after cache misses. Updating the algorithm was a very costly measure, but investing in hardware with better memory bandwidth gave us some short-term wins. There are very few developers who can look at an algorithm and spot a cache utilisation problem, but measurement can find it for you.

I think I understand why some developers prefer to dive right into code, instead of measuring the problem. We like to think we’re smart. We know stuff about coding, and we’re good at it, and so we can jump in and spot what the original developer didn’t. We can use the sheer power of our minds to solve the problem. It’s a little like the builder who drives in nails with his forehead. It might be impressive, but it’s not very effective, and you’ll end up with a headache.

The opposite problem happens too: we make the mistake of not valuing our time. “What can it hurt to spend a week re-writing those string-plus operations as StringBuilder operations,” we ask ourselves. It makes our code better (by one measure, I suppose), and it won’t hurt performance. But if we spend a week of our time, was it really worth the investment? Measure up front, so you can make that decision based on facts instead of guesses.

Perhaps some developers just feel like they have great intuition. One of my uncles was actually pretty good at estimation: he’d cut timber a shade too big, and trim it down as he was placing it. He only did that occasionally, though. A lifetime of building told him when it made sense to trust his estimation skills, and when to measure twice, cut once.

So we’re back to builders, and here’s the deep lesson. If a builder will measure twice to avoid wasting a bit of cheap wood, how much more should we measure to avoid wasting precious time? I’ve seen weeks wasted on the wrong problem, simply because someone didn’t bother to measure.

Please, my fellow developers: measure (at least) twice.

Hooking (Hacking?) the ASP.Net FileChangeNotifier

I’m going to tell you a little bit about a package I’ve just put together which lets you exclude certain files from the ASP.Net change notifier (the thing which restarts your IIS app pool whenever you change certain files). On the way, we’re going to dig into some internal framework code, and I’m going to horrify you with some code that should definitely never get run in production! Let’s start with the inspiration for this work.

A Bad Dev Experience

The dev experience on one of our legacy projects had been annoying me for a while: whenever I triggered a Gulp build, my IIS app pool would restart. That sometimes had (annoying) flow-on effects, and we ended up with various work-arounds depending on what we were doing. It was never a priority to fix, because Gulp builds only happened on dev and CI machines, never in deployed environments – but one week it really got in the way of some work I was doing, so I decided to get to the bottom of the problem.

You may know that ASP.Net will restart your application whenever there are changes to your web.config or any files in your /bin folder. What you might not know (I didn’t) is that it can also monitor various other files and folders in your application, and what it monitors doesn’t seem to be well-documented. Our Gulp build dumps the output into a /dist folder, and ASP.Net was restarting our app every time any file in that folder changed. After a little Googling, I discovered a StackOverflow answer which gave a slightly-hacky way to disable the ASP.Net file monitoring, which I did – and it worked. No more app restarts!

Not So Fast…

After a few hours of front-end work, I needed to make some back-end changes. I finished wiring up my API endpoints, ran the project, and expected success! No luck. After some more tinkering with the front-end, I suspected my C# code – but after some checking, it was definitely correct: I had the unit tests to prove it. Eventually I fired up the debugger, and found my problem: the debugger was reporting that the running code didn’t match my file. Disabling file change notifications had also disabled notifications on the /bin folder, so my rebuilt DLL files weren’t being loaded. I wasn’t prepared to accept rebuilds not triggering reloads, so I needed to revisit my solution.

Before I went too far, I wanted to be really sure that I was looking at the right problem. It turns out someone going by the name Shazwazza has already done the hard work to build a list of files and folders ASP.Net is helpfully monitoring for changes, and yes – there is my Gulp output folder.
FileChangeNotifierFileList

What I really want is to be able to preserve all of the existing functionality – but skip monitoring for certain paths. I investigated a few different switches and options, but none of them gave me exactly what I wanted. I was going to have to get my hands dirty.

Diving Into Framework Code

Having read Shazwazza’s code, I had already learned that the file monitor is stored on the HttpRuntime singleton, so I started my investigation there. I was hoping it just stored a reference to an interface, so I could just replace it with my own wrapper over the base implementation.
FileChangeNotifierReference

Well, no interface – but I suppose I can just subclass FileChangesMonitor and use reflection to swap the base implementation out for my own.
FileChangeNotifierSealed
Oh. It’s a sealed class – so I can’t even subclass it. It was about this point that I stopped trying to solve the problem on work time: this had become a personal mission, and I wasn’t going to give up until I’d made this thing behave the way I wanted to.

First, I needed to get a solid understanding of how the FileChangesMonitor class worked. After a lot of reading, it boils down to this: other classes ask the FileChangesMonitor to monitor a particular path, and it maintains a hash of paths to DirectoryMonitor instances which it creates. In turn, DirectoryMonitor has a hash of filenames to FileMonitor objects it has created, and FileMonitor has a HybridDictionary which stores callbacks. FileChangesMonitor, DirectoryMonitor, and FileMonitor are all sealed classes, so it looks like I’m out of luck – but I refused to be beaten!

The core classes I’m dealing with are all sealed, so there’s definitely no option to change behaviour there – but at each step of the object hierarchy, a parent object is storing instances of these sealed classes in plain old collection types: Hashtables and HybridDictionaries. Those classes aren’t sealed, so that’s going to have to be the seam where I stick my metaphorical crowbar.

A Word of Warning

If the idea of messing with private-member Hashtables makes you nervous, good. I’m about to go messing with the internals of framework classes in a very unsupported, highly-fragile way. This is not production-quality code. This is nasty, hacky code which is probably going to break in the future. If you ever use this code, or this extension technique, please do it in a way which will never, ever, ever run on a production machine.

If you love the idea of #antiproduction code, you’ll probably enjoy my Object.Extend(…) implementation and my .ToRandomCollectionType<>() LINQ extension.

I’m trying to fix the dev experience here, and so I decided to accept the hacky nature – but I have put plenty of guards around it to keep it clear of our production machines. I’ve even made sure it stays clear of our test environments. It’s a local-dev-environment only thing!

A Note on Reflection

If you’re familiar with the .Net reflection API, you’ll know it can be a little clunky. I’m using the following extension methods to make my code look a little cleaner.

public static class ReflectionHelpers
{
  public static FieldInfo Field(this Type type, string name) => type.GetField(name, BindingFlags.NonPublic | BindingFlags.Instance);
  public static FieldInfo Field<T>(string name) => typeof(T).GetField(name, BindingFlags.NonPublic | BindingFlags.Instance);
  public static FieldInfo StaticField<T>(string name) => typeof(T).GetField(name, BindingFlags.NonPublic | BindingFlags.Static);

  public static ConstructorInfo Constructor(this Type type, params Type[] parameters)
    => type.GetConstructor(BindingFlags.NonPublic | BindingFlags.Instance, null, parameters, null);

  public static MethodInfo StaticMethod(this Type type, string name, params Type[] parameters)
    => type.GetMethod(name, BindingFlags.Static | BindingFlags.NonPublic, null, parameters, null);

  public static Type WebType(string name) => typeof(HttpRuntime).Assembly.GetType(name);
  public static Object Make(this ConstructorInfo constructor, params Object[] parameters) => constructor.Invoke(parameters);
  public static Object Call(this MethodInfo method, params Object[] parameters) => method.Invoke(null, parameters);
}

How I Hooked (Hacked) FileChangesMonitor

What I really want to do is to prevent FileMonitor from setting up file watches for specific paths, and my only extension point is the HybridDictionary of watches. When FileMonitor is asked to set up a watch, it checks its dictionary to see if there’s already one there for that specific callback. If it doesn’t already have one, it sets one up. So what I need is a HybridDictionary which always has an entry for any given callback!

Sadly, HybridDictionary doesn’t declare its methods as virtual, so we can’t just go overriding them. Fortunately, it does just use a Hashtable under the covers – and we can use some reflection trickery in the constructor to replace the Hashtable it creates with one of our own design!

/// <summary>
/// Intercepting hybrid dictionary. It always responds to any key request with an actual value
///  - if it didn't exist, it creates it using a FileMonitor factory.
/// HybridDictionary always uses a hashtable if one has been constructed, so we just rely on InterceptingHashTable.
/// This relies on an implementation detail of HybridDictionary but we're way past worrying about that kind of thing at this stage.
/// </summary>
public class MonitorInterceptingHybridDictionary : HybridDictionary
{
  static Type EventHandlerType = WebType("System.Web.FileChangeEventHandler");
  static ConstructorInfo FileMonitorConstructor = WebType("System.Web.FileMonitorTarget").Constructor(EventHandlerType, typeof(string));
  public MonitorInterceptingHybridDictionary()
  {
    var fakeTable = new InterceptingHashTable(o =>
      {
        try
        {
          return FileMonitorConstructor.Make(null, "test");
        }
        catch (Exception ex)
        {
          Log.Error(ex, "Unable to create a file monitor target pointing at {Target}", o);
          return null;
        }
      });
    Field<HybridDictionary>("hashtable").SetValue(this, fakeTable);
  }
}

Our new HybridDictionary is using another new class I’m going to re-use shortly – the InterceptingHashTable. Here, we finally catch a break: Hashtable declares a lot of its methods as virtual, so we can override functionality. What we’re going to do here is to create a new subtype of Hashtable which always has the key you’re looking for, even if it hasn’t been added. We pass a factory function in to the constructor, so whenever something asks for the value for a given key, we can construct it on the fly if it hasn’t already been added.

/// <summary>
/// Intercepting hash table. It always responds to any key request with an actual value
///  - if it didn't exist, it creates it with the provided factory.
/// This currently only implements this[key] because that's all the file monitor stuff uses.
/// To be generally useful it should override more methods.
/// </summary>
public class InterceptingHashTable : Hashtable
{
  private readonly Func<object, object> _factory;
  public InterceptingHashTable(Func<object, object> factory)
  {
    _factory = factory;
  }
  public override Object this[Object key]
  {
    get
    {
      var maybe = base[key];
      if (maybe == null)
        maybe = CreateMonitor(key);
      return maybe;
    }
  }

  public Object CreateMonitor(Object key)
  {
    return _factory(key);
  }
}

If you take a look back at the MonitorInterceptingHybridDictionary code above, you’ll see that it’s going to pretend it’s already got a FileMonitorTarget anytime something looks – and that means the calling FileMonitor will think it’s already got a watch on the file, and skip creating it. Bingo! The next step is to replace the HybridDictionary on FileMonitor with an instance of our new class.
FileMonitorTargets

Of course, it’s a private member, and we’ve currently got no way to hook the creation of FileMonitor instances. What does create FileMonitor instances? The DirectoryMonitor class – and fortunately for us, it keeps a Hashtable of all of the FileMonitors it’s created. That means we can swap out the Hashtable for another of our InterceptingHashTables, and every time the DirectoryMonitor looks to see if it already has a matching FileMonitor, we create it if it doesn’t already exist.

// Create a dodgy hashtable which never has a blank entry - it always responds with a value, and uses the factory to build hacked FileMonitors.
var fakeFiles = new InterceptingHashTable(f =>
  {
    try
    {
      // Build a FileMonitor.
      var path = Path.Combine(o as string, f as string);
      var ffdParams = new object[] {path, null};
      ffd.Invoke(null, ffdParams);
      var ffdData = ffdParams[1];
      object mon;
      if (ffdData != null)
      {
        var fLong = ffdData.GetType().Field("_fileNameLong").GetValue(ffdData);
        var fShort = ffdData.GetType().Field("_fileNameShort").GetValue(ffdData);
        var fad = ffdData.GetType().Field("_fileAttributesData").GetValue(ffdData);
        var dacl = secMethod.Call(path);
        mon = fileMonConstructor.Make(dirMon, fLong, fShort, true, fad, dacl);
      }
      else
      {
        // The file doesn't exist. This preserves the behaviour of the framework code.
        mon = fileMonConstructor.Make(dirMon, f as string, null, false, null, null);
      }
      // Break the FileMonitor by replacing its HybridDictionary with one which pretends it always has a value for any key.
      // When the FileMonitor sees it already has this key, it doesn't bother setting up a new file watch - bingo!
      var fakeHybridDictionary = new MonitorInterceptingHybridDictionary();
      mon.GetType().Field("_targets").SetValue(mon, fakeHybridDictionary);
      // Take this warning away later
      Log.Debug("Provided fake monitoring target for {Filepath}", path);
      return mon;
    }
    catch (Exception ex)
    {
      Log.Error(ex, "Unable to create a file monitor for {Filepath}", f as string);
      return null;
    }
  });

Now, we’re faced with another problem: we have no control over the creation of DirectoryMonitor instances, but we need to replace the private Hashtable it keeps with the fake one we’ve just made.
DirectoryMonitorFilemons

Luckily for us, the FileChangesMonitor class keeps a Hashtable of DirectoryMonitors!
FileChangesMonitorDirs

You might be able to guess where this is going: we need another fake Hashtable which always creates DirectoryMonitor objects on the fly. Stay with me here, we’re approaching the end of this thread we’ve been pulling on.

// This is a factory which produces hacked DirectoryMonitors. DirectoryMonitor is also sealed, so we can't just subclass that, either.
Func<object, object> factory = o => {
  try
  {
    if (!(o as string).Contains(@"\EDS\dist"))
    {
      Log.Debug("Allowing file change monitoring for {Filepath}", o as string);
      return null;
    }
    var dirMon = dirMonCon.Make(o as string, false, (uint) (0x1 | 0x2 | 0x40 | 0x8 | 0x10 | 0x100), fcnMode);
    // Create a dodgy hashtable which never has a blank entry - it always responds with a value,
    // and uses the factory to build hacked FileMonitors.
    // ... the previous code block goes here ...
    // Swap out the hashtable of file monitors with the one which lets us always create them ourselves.
    dirMon.GetType().Field("_fileMons").SetValue(dirMon, fakeFiles);
    return dirMon;
  }
  catch (Exception ex)
  {
    Log.Error(ex, "Unable to create a directory monitor for {Filepath}", o);
    return null;
  }
};

Finally, the HttpRuntime class holds a reference to a single copy of FileChangesMonitor, and HttpRuntime itself is a singleton!
HttpRuntimeFcm

HttpRuntimeSingleton

With a little more reflection, we can therefore swap out the Hashtable in FileChangesMonitor with our hacked Hashtable of DirectoryMonitors, and now we have control of all the instantiation of DirectoryMonitors and FileMonitors, and we can intercept any filenames we don’t want tracked and provide dummy FileMonitorTarget responses to prevent the watches from being set up.

/*
* I am so, so sorry if you ever have to maintain this.
* ASP.Net monitors a pile of folders for changes, and restarts the app pool when anything changes.
* Gulp often triggers this - restarting the app pool unnecessarily, annoying developers. Annoyed developers write code like this.
*/
var httpRuntime = StaticField<HttpRuntime>("_theRuntime").GetValue(null); // Dig out the singleton
// Grab the FileChangesMonitor. It's a sealed class, so we can't just replace it with our own subclass.
var fcm = Field<HttpRuntime>("_fcm").GetValue(httpRuntime);

// Grab a bunch of internal types, methods, and constructors which we're not meant to have access to.
var dirMonType = fcm.GetType().Field("_dirMonSubdirs").FieldType;
var dirMonCon = dirMonType.Constructor(typeof(string), typeof(bool), typeof(uint), typeof(int));
var fileMonType = dirMonType.Field("_anyFileMon").FieldType;
var fadType = fileMonType.Field("_fad").FieldType;
var fileMonConstructor = fileMonType.Constructor(dirMonType, typeof(string), typeof(string), typeof(bool), fadType, typeof(byte[]));
var ffdType = WebType("System.Web.Util.FindFileData");
var ffd = ffdType.StaticMethod("FindFile", typeof(string), ffdType.MakeByRefType());
var secMethod = WebType("System.Web.FileSecurity").StaticMethod("GetDacl", typeof(string));

// ... insert all the above code to create the factory ...

// Swap out the hashtable of directory monitors with the one which lets us always create them ourselves.
var table = new InterceptingHashTable(factory);
fcm.GetType().Field("_dirs").SetValue(fcm, table);

So what’s the up-shot of all of this? Well, firstly, I’ve written some of the nastiest, hackiest code of my career. Secondly, I’ve wrapped it all up into a nuget package so you can easily run this nasty, hacky code (it’s not up quite yet – I’ll share another post when it’s ready to go). And thirdly, I need to warn you again: Do not put this near production! I don’t care how you accomplish it, but please put plenty of gates around the entry point to make sure you don’t trigger this thing in a deployed environment. It really is a dev-only device.

Surely There’s a Better Way?

There is absolutely a better way. I encourage you to find other ways to avoid dev changes constantly churning your app pool. I strongly suspect our problem was serving our Gulp build output via a secondary bundling step: the Asp.Net bundler seems to be one of the things which might register files with the FileChangesMonitor. For various reasons (mostly, that the project was a big, complex, legacy project which is hard to test thoroughly), I strongly preferred dev-only changes to making production-affecting changes. And so here we are.

If you can find a way to avoid using this technique, I strongly recommend you do.