Hi folks,

This is the weekly digest for my blog, Irrational Exuberance, and these are my posts from the last week. Always grateful to hear your thoughts and suggestions for topics to write about, drop me a tweet or direct message over on Twitter at @lethain.

Read posts on the blog:

- Running systems library as Cloud Function.
- Path to engineering manager of managers.
- An Elegant Puzzle @ Software Engineering Daily

Running systems library as Cloud Function.

I was chatting with my friend Bobby Powers, and he mentioned a systems dynamics app he's been hacking on. That conversation inspired me to want to finish a project I've been neglecting, which is explosing systems as an HTTP endpoint running within a Google Cloud Function.

The rough goals of this exercise were: (1) expose systems via an HTTP API, (2) run it as a function, aka serverless, to minimize maintenance and cost, (3) the development workflow should be very simple since I'll otherwise forget how it works when I periodically forget about it, (4) it should be extensible for me to add a JavaScript GUI that calls into the API at some later point.

These goals have been accomplished, although it was a bit trickier than I expected.

I already had the Google Cloud SDK installed, so I skipped over that bit. Next I moved on to create a new Git repository:

cd sys-app
git init

Initially it was just three files. The first was an HTTP function at sys-app/functions/run-model/main.py:

def hello_get(request):
    return 'Hello World!'

The second was in that same directory at sys-app/functions/run-model/requirements.txt and specified the only Python library dependency:

systems==0.1.0

The final file was a build configuration for Cloud Build:

steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    args: ['functions',
           'deploy',
           'run_model',
           '–trigger-http',
           '–runtime',
           'python37'
          ]
    dir: 'functions/run-model'

Then created a new private repository on Github and sync the local repo up:

git remote add origin git@github.com:lethain/sys-app.git
git push -u origin master

Then I updated Cloud Source Repositories on GCP to mirror over lethain/sys-app. I could have also just connected to the repository directly on Github instead of mirroring it over, but for whatever reason I sort of prefer mirroring over and then getting to operate on it directly within Cloud Source as it's a bit more decoupled.

Started following the steps to do automatic builds. First enabled Cloud Build, then created a build trigger firing on all deploys on sys-app to master branch.

As a caveat, I know it says Github is the source, but the source I selected was the Cloud Source mirror. As best I can tell, this is just a weird UI bug.

The penultimate step was granting Cloud Function deploy permissions to Cloud Builder as its documented here. I initially missed this step, and honestly found it a bit confusing even following the directions word for word, but I eventually got it (I debugged by continuing on with next steps and futzed around until it all worked).

At this point, I committed another change to the sys-app repository and deployed it, and viola, the build trigger fired, the function was created, and it all worked.

In full transparency, I got stuck here for some time, with everything seeming to work, with new versions of the function getting created on each deploy, but the code it ran never getting updated at all. I had created that function through the UI, and there just wasn't much useful context in the logs to say why it wasn't updating, so I went ahead and deleted it and relied on the cloud build configuration to create and update a new function, at which point it worked properly.

I'm able to verify the function through the gcloud CLI:

gcloud functions call run_model \
–data '{"definition": "a > b @ 1"}' \
–project=PROJECT

Alternatively, I can also call it over HTTP in Python:

import requests
definition = """[a] > b @ 5
b > c @ 4"""
resp = requests.post('https://$MY-FUNCTION-URL',
                     json={'definition': definition})
print(resp.content.decode("utf-8"))

I'm now able to modify sys-app, push the new commit, and the function updates automatically behind the scenes, which was pretty much the original goal.

Unlike with my GKE experiments, I was able to get this deployment flow without creating any sort of services to handle scheduling upgrades after builds completed or whatnot, which is quite nice. (Looking at the kubectl cloud builder, I bet I could figure out a way to do continuous deployments to k8s without a service monitoring the build pub-sub queue, I'm not sure if this is new functionality or if I just missed it last time.)

If I were going to be developing with this workflow frequently, my next step would be to figure out how to run tests in a blocking fashion on each build, but it doesn't seem like there is a good way to do that for Python (they're missing pip as a cloud-builder, so I think I'd need to setup a Dockerfile to use for testing, which is totally fine, but not something I'm particularly excited for).

Altogether, there is a great user experience just over the hill from what GCP is offering. Imagine how delightful this user experience would be if after I created a new Cloud Function I could select a checkbox and it would automatically deploy each time I changed it in the Cloud Source repository? Imagine further yet if I could include a test.py that would automatically run before attempting to deploy an update?

You absolutely can build all of this yourself, but why should each user need to add this glue? This is why platforms like netlify are cropping up despite the cloud vendors having all the right components underneath. This evolution reminds me a bit of AWS' attempt to move downstream into the personal VM market with Amazon Lightsail, executing a few years too late on what other hosting providers like Slicehost and now Digital Ocean have realized some users want.

A good reminder for all of us that we should take a few hours each quarter and integrate our products from scratch and see what the rough edges are.

Path to engineering manager of managers.

Not very polished, but I'm giving a short talk about the path to engineering manager of managers roles, and these are my notes.

Once your company gets large enough to roll out a formal career ladder, it becomes the foundation of most discussions about performance and promotion.

All ladders have certain requirements that have to be met in order to advance, but typically the resources required to fulfill those requirements aren't scarce. As you get more senior, scarcity starts to creep in:

• How many "staff"-level engineering projects are available this cycle?
• How many folks can deeply influence organizational direction?
• How many folks can lead the team's project for this quarter?

The boon and curse of most scarce resources is that there is a great deal of room for interpretation – maybe it really was a staff project – but managers often run into a particularly difficult scarce resource: they have to start managing managers to advance in their role.

This is a well-known anti-pattern to ladder writers, so it's an interesting phenomenon that every company is very deliberate to avoid codifying "managing managers" – or even team size – into their ladder requirements, but most folks in most companies end up believing that it's an implicit requirement.

It's pretty easy to understand why: it's not explicit anywhere, but pretty much everyone past a certain point is managing managers. To get there, it feels like you have to follow.

Challenging this assumption a bit, you do not have to become a manager of managers to continue advancing your impact or your career. There are real, senior management and leadership roles that don't require that switch, but they typically (a) exist at either very large or very small companies, (b) tend to rely more on subject-matter expertise in a critical but adjacent topic (e.g. for a product oriented company, that might be managing infrastructure costs or reliability), and finally (c) they're quite a bit less common and might not exist at your company.

If the existing manager of manager roles at your company aren't the right fit, this is a place where you can really flex your career narrative. Find ways to develop yourself and increase your impact, and then partner with your management team to translate that into your ladder. Remember, your ladder was almost certainly carefully crafted with the intention of this being possible, and folks love to set a positive precedent that things which appear impossible are possible.

But maybe you really do want to move into a manager of managers role, and it's not just because you "have" to, it's what you're interested in. How do you get one?

There are only really three paths to your first manager of managers role:

1. Replace an existing role because someone leaves your company.
2. Fill a new role created by your company growing (e.g. participate in the managerial pyramid scheme of hypergrowth).
3. Get hired by a different company into the role.

My decision to join Uber in 2015 was a very deliberate decision to find a company where I could develop from managing a small team to a larger one. I'd found getting hired directly as a manager of managers was quite difficult without the existing experience, and in retrospect I'm very glad I was able to make this particular transition while operating in an environment with enough structure and other folks operating in the same sorts of roles that I could learn from and with.

I've seen folks successfully take all three of these paths, but the second is the path I've seen work the most consistently. As such, the single most effective thing you can do is to join a company that is growing fairly quickly.

Any quick growing company isn't enough though, it's also much easier when you join a company that has a structured process to consider internal candidates. Allowing qualified internal candidates to participate in such searches is something we've done more and more at Stripe as we've grown, and is something I'm quite proud of.

Another side-effect is that you get much better feedback when you interview internally. Internally, you'll generally get thoughtful written feedback for you to learn from. For external interviews, most companies just won't give you very structured feedback out of an abundance of legal caution.

That's not to say that process isn't sometimes quite awkward – to the contrary, it can create some delightfully weird professional situations, but I think it's still the best approach thing.

A really interesting thing happens, though, when you get to a standardized process for considering internal candidates within your interview loops, which is that now there is a single path to getting these sorts of roles – pass an interview loop for a manager of manager role.

That's a more concrete problem to focus on, and the question then becomes "how do interview loops for manager of managers roles work?"

The basics are going to be roughly the same – a phone screen, a couple of onsites, some reference checks – but the skills being evaluated will generally differ quite a bit.

You will, at times, find yourself being interviewed for a manager of managers position through a general engineering manager loop (the same as you might for a non-manager of managers role), and you might even find some very small companies that primarily evaluate you as a senior developer... and I don't have much advice here for those scenarios.

Or perhaps the advice is to ask yourself if you're motivated to do the teaching and growing of the organization that those choice suggest will be a big part of your upcoming work.

A good manager of managers loop is going to focus more on:

1. communication - are you able to communicate a business topic to a technical audience? Communicate a technical constraint to a business audience where they come away understanding the tradeoff? Can you summarize a nuanced, complex topic concretely? Can you explain tradeoffs clearly, showing understanding of all perspectives?

2. leading at scale - many leadership techniques don't work well when you're only able to devote 10-20% of your time to an area, have you developed compensating set of skills to remain effective in such scenarios? Things like coaching, effective goals, strategy and vision become your core toolkit. A good process tests how these components intersect, for example having you coach someone on their goals, as well as writing a strategy along with identifying the short and long-term goals to support that strategy.

3. operating at scale - have you operated an organization? This will focus on topics like scaling the hiring process, succession plannining, evolving and using performance management systems. Are you an effective hiring manager, including for roles that you haven't personally performed (will you be able to hire senior candidates to work on your team)? Are you able to read an organizational health report and action it effectively? Are you familiar with typical organization evolution issues and rough edges like the friction around career levels?

4. good partner - are you a collaborative, effective partner to other teams and functions? Do you have a non-hierarchical world view across functions? Do you understand how your function impacts other functions, and are willing to optimize globally rather than for your function?

5. domain expertise - do you have experience to be the functional leader for the area you're joining? Are you familiar with the domain's evolution and where it's evolving next? Do you have thoughtful opinions and are willing to be challenged on those opinions (without becoming defensive)? This varies a bit by company. with some expecting their senior management to be domain specialists with a history of operating, and others being more comfortable with general management experience.

The most interesting thing is that you can almost certainly practice all of those in your current role. You absolutely do not need to be in a manager of managers role to build these skills, but you do need to be deliberate about selecting the projects that will fill in your personal gaps.

As you build up your skills, periodically apply to different roles – internally if available but otherwise externally – and get more feedback. Take that feedback and focus for the next set of projects.

Not discussed here, but once you do get into these sort of roles, you need to be pretty deliberate about staying in them. Particularly, to ensure that you're scaling yourself inline with the company's growth.

The intended audience for this talk is folks at a rapidly growing company who are likely to have access to manager of manager roles within their company, which means that it doesn't address what can be the most pressing question for folks who aren't in such situations: how do you get access to these roles to begin with?

It's pretty difficult.

An Elegant Puzzle @ Software Engineering Daily

Got an opportunity to speak on Software Engineering Daily about engineering management and on An Elegant Puzzle. This is my favorite podcast I've been on so far, with exceptionally good questions and discussion. Talk about distributing desirable and undesirable work, Stripe's incident program and much more.

Quite a few more podcasts coming over the next couple months!

That's all for now! Hope to hear your thoughts on Twitter at @lethain!