This post is part 2 of a series on how you can take advantage of processes and tooling that large public open-source projects have developed to build a foundation that will help you scale your engineering organization and better take advantage of each engineer’s talents to meet your engineering goals. You can read the entire series here.
You’ve heard the old saying “a rising tide lifts all boats”, right? Some version of the saying stretches back as far as the Qing Dynasty [1] so it’s fair to say most people have heard it at this point. In essence, as a particular group flourishes, its members also flourish. You may be asking, how are tides and fences relevant to my software engineering organization. Simply, remove one and get the other.
Where Traditional Software Engineering Strategy Fails
A few years ago I was pulled into a meeting with my VP and others to discuss a problem that our group within the software engineering organization had identified. Like many companies, we had seen an explosion of web applications in the preceding years. What followed was the creation of engineering teams that built fences around the particular application in question. They didn’t do this maliciously, hence my use of the term “golden fences“. As they were building their assigned apps, each team essentially walled off all knowledge about their application–what technology it was built on, what features it had, how it worked, how they architected it, and even access to the source code. Nearly every aspect of the app was hidden behind these fences. They simply didn’t know any other way to operate.
During this meeting, we talked about feedback we were getting from our users. They were telling us that our apps were causing friction for them when trying to do their jobs. Many of our users interacted with more than one of our apps on a daily basis. The UI/UX wasn’t consistent between apps, similar features in various apps didn’t work the same way, and other observations were made by our users. When my VP tasked me with designing and building a platform where we could more easily build web applications that our end users needed to perform their daily tasks, I knew that the operating the same way as we had before wouldn’t be productive. I decided to take a few pages out of the book(s) that many successful public open-source projects and our “internal open source” or “innersource” philosophy was born.
Using Open Source Methodologies to Scale Development Teams
One of the first decisions we made was that everything would be done “in the open” just like those public open-source projects I mentioned. Look at any large open-source project on GitHub, GitLab or BitBucket and you can see issues filed, code changes submitted by committers, and comments and discussion on the issues and code. All of that is accessible to anyone. This concept was completely foreign to our group. Not only were the engineers not used to working this way, but this was also a major change for product management and engineering leadership. They simply weren’t used to having all decisions, issues and code being so accessible to people who weren’t involved in the project directly.
Every company has their own distinct “culture”. The longer it is in place, the more difficult it is to change. Honestly, we are still working to change our engineering culture from using isolated “component teams” to more collaborative “feature teams” (I’ll talk about the feature team concept in a later post). It takes time, patience and a willingness to provide consistent leadership in order to realize the benefits to doing business this way in your own organization.
Sensible Processes For Orderly Software Development
So, that’s all well and good, but you are still wondering how getting rid of these fences raises boats right? One of the other ideas we brought from the public open-source projects was that anyone that could log into the company’s Git server could have access to our code. We established some guard rails around accepting submissions to keep things orderly:
- All changes have to be made in a branch and a pull request (PR) submitted in order for those changes to be merged
- Only 3 or 4 people can push to the main code branches without a pull request (for emergency situations which thankfully we’ve rarely needed)
- A minimum of 2 other engineers have to approve the pull request
- One of those approvals must be from the list of engineers identified as code owners of a particular part of the repository (we use a monorepo organization for our front-end apps)
- We asked all engineers to spend a few minutes 2-3 times per day reviewing pull requests
I tell all my teams that not one of us has all the good ideas. Every engineer in the organization has something to add or else they wouldn’t be on the team. We have all engineers, regardless of skill level, review PRs every day. One of the benefits of this is that they are all exposed to everyone else’s code and see different approaches to problems that need to be solved. This is invariably good for someone. Either the engineer reviewing the PR learns something or they make a suggestion that leads to someone else learning something. I’ve personally witnessed senior front-end engineers in discussion with more junior folks come away thanking the junior for showing them something they didn’t know. Likewise, junior engineers can benefit from review by more senior folks. Things like “abc causes this ‘bad thing’, try XYZ instead. It’s worked for me in the past” are very common in our pull request comment history.
As an engineering manager, you have to stay connected to this process to make sure that it remains productive for the engineers. You absolutely must have a zero tolerance policy for passive aggressive or non-constructive feedback. It takes time and attention to make sure that engineers of all skill levels and personality types feel safe in posting a comment or question on someone else’s code. You need to be aware if anyone is getting “defensive” over PR comments and remind them that we’re all in the same boat, rowing in the same direction (to mix maritime metaphors a little bit).
Selling Software Development Strategy Improvements to Upper Management
It’s also not all about your engineering staff either. Your higher management needs to understand the benefits of this model. I had the opportunity to talk about our internal open source initiative with our CTO and his direct reports about 18 months after we started the platform. After one slide, my SVP asked “can I see your code” to which I replied “here’s the repo, it’s open to anyone who can log in”. He nodded and about 3 minutes later he elbowed the CTO and said “He was
n’t kidding! They even have a README telling me how to work with their repo. I love this!”
In that meeting and at other times afterward, I was also able to share with my leadership the growth I’d seen in engineers while working within our internal open source arena. Our feature teams are a mix of mostly mid- and senior-level engineers, with some more junior folks mixed in as well. Watching the junior engineers’ skill levels increase as they take part in both the submitter and reviewer side of code reviews and pair program with the other engineers is immensely rewarding. On several occasions, questions or PR submissions by more junior people have resulted in great discussions among the teams resulting in the code getting better and all engineers involved learning something new.
The Results of Software Development Process Improvement with Innersourcing
All of these things have resulted in a massively productive environment where multiple full-stack teams of engineers are able to more quickly build web applications that our business customers need to do their jobs. Leadership loves happy and productive engineering organizations. Engineers love environments where they can focus on solving problems and feel that their work is meaningful. Removing the “Golden Fences” to spur cross-team collaboration and remove friction for our engineers has helped us build that kind of culture within our department. I’m excited to see how it grows inside the larger engineering organization that we’re a part of.
If you are interested in more discussion around this topic, I recently spoke to the “React Wednesday” folks on their Twitch stream. They wrote up a summary blog post and posted a recording to YouTube . Alternatively, reach out directly and I’d be happy to chat more about the innersource concept with you.
[1] A rising tide lifts all boats, Wikipedia