Subversion is not an accounting system

This is the story of how I fixed the accounts payable processes at The Apache Software Foundation and learned to be a proper Treasurer.

I accepted the role of Assistant Treasurer at The Apache Software Foundation in November of 2018 and then accepted the role of Treasurer in October of 2019. I was serving the Apache Fineract community as PMC chair at the time; I had just given a keynote at ApacheCon Montreal on Open Source banking.

I was a new member of The ASF, just voted in March of 2018. Shane Curcuru saw me working on banking software, and thought I’d be a good fit for the Assistant Treasurer job, so he asked me to volunteer. But knowing how to program debits and credits and interest calculations for loans in Java is not the same thing as knowing how to oversee an accountant. When I first started as Assistant Treasurer, my goal was to learn and to maintain the status quo. I watched the bill pay process; I made sure our CDARS were rolled when they matured; I checked over our tax filings; I watched the interactions between Fundraising and our accountant…

When something was asked of me I did it. But I had plenty on my plate. I had started an EMBA in Cologne in September of 2018, I had two kids under 11 at home. Then starting in April of 2019, I had joined the Apache board for a turbulent year. That all kept me busy. I wasn’t planning to make major changes to the Treasury.

But it became increasingly clear that our financial tools and processes were stuck in the past. Previous volunteers hadn’t had time to modernize them. Doing this as a volunteer with software development expertise, but without accounting expertise means you often do the minimum to get by, and operational roles are sometimes undervalued in Open Source. Our accounting firm was similarly just following the existing pattern without driving change. But the processes were no longer serving us. Greg Stein repeatedly bent my ear (for the record, if Greg is complaining about something, you should listen) about how our invoice approvals process was failing us.

“What is an invoice approvals process?”, you might ask. If you’re an Open Source developer like me you may never have had reason to learn this part of business. A business can’t just pay every invoice presented to them. It would be far too easy for a con artist to present a fraudulent invoice. The business needs a way of checking that the invoice is legitimate. You might think at first blush that this is fairly simple, right? You just ask the budget-owner. That person then “approves” the bill. The bill then gets paid out of the business bank accounts. But now you need some way of tracking who approved which invoice, so that you can go back and check that only approved invoices were paid, and also that all approved invoices were paid.

Without a system to record invoices and record approvals, you’re left with only email and word of mouth. If the person who actually orders payments to be made by the bank misunderstands something, or is misled, they could get blamed for making inappropriate payments. This is bad for accountants and very bad for treasurers.

Sometime back in the grey mists of time, someone had introduced an invoice approvals process at Apache that was based on Apache Subversion. For those who don’t know, Subversion is an Open Source version control system developed for software developers to track changes to source code. When it was first introduced, it was a great solution to the problem. Back then, there weren’t SaaS solutions available to track invoice approvals. Source control can represent all of the required pieces of the process. Subversion, of course, requires users to login. And Subversion’s history could track every step of the approval process, by moving bills to a folder called “approved”, and then “paid”. Subversion was doing everything it needed to do. Sort of.

Here’s the thing: not everyone who needed to approve bills had experience developing code. And almost no one involved in the bill pay process (except for myself) had experience developing code. The people doing the most work for the process weren’t comfortable using Subversion. As a result of the mess, another approvals system had also been set up at The Foundation based on folders in GSuite.

Our accountant was able to line up payments based on the invoices. He would set up a batch payment with our bank once a month, and send Treasurer and Assistant Treasurer an email. This is how that looked:

Then I’d log into our bank account and approve the batch. Before approving it, I would check the Subversion folder and the GSuite folder to make sure that every payment had an approval. For the first several years working with this process we had an accountant who was able to use Subversion to move items from “approved” to “paid”. But then she got promoted, and we got a new person. And the ability to use Subversion is not a hiring criteria for accountants. It’s just not. The new guy wasn’t able to move items from “approved” to “paid” in Subversion, and he wasn’t interested in learning. Greg had cleaned up the “approved” folder, but paid invoices accumulated in the “approved” folder. Greg, who is not an accountant and had no way of knowing what had been paid and what hadn’t, wasn’t willing to continue to clean up. At that point Greg raised the alarm to me, and we started looking for alternatives. I told the board in May 2019 at the Chicago face-to-face that this was a problem, but my warning was overshadowed by other events.

It reached the point that I could no longer be certain that all of the approved invoices had been paid. There were too many things in the approved folder. And I couldn’t definitively link an invoice in “paid” to an actual payment either. And neither could the accountant. One of our officers was receiving overdue notices from a vendor. Invoices in Subversion weren’t stored in a way that enabled searching by vendor. The accountant couldn’t tell them whether the overdue notices were valid, or which invoice of the several that we had received from that vendor hadn’t been paid. The accountant couldn’t even tell which invoices we had received from the vendor. At this point, under pressure from the officer, the accountant lined up an “out-of-band” payment with two invoices and presented it to me. I checked, but I could not find an approval for the invoices. The only form of invoice approval I had for this out-of-band payment was a forwarded email from the officer which had not even been sent to an archived mailing list. Neither the officer, nor the accountant, nor I had the tools we would have needed to check if the invoice had already been paid. I was being asked to approve the payment anyways.

I declined.

This was the breaking point. Something had to be done. Greg and I had already identified bill.com as the appropriate tool. So Greg Stein and I spent Easter of 2020 getting a bill.com account, entering The ASF’s vendors’ banking information, and making test payments via the new system to make sure we had everything set up properly. Yes, my kids were mad at me. Rightfully so. I was also in the middle of writing my master’s thesis and finishing up the last of my EMBA classes. And via the pandemic, my kids were schooling at home as well. Pushing that change at that time did not fit my plans.

The accounts payable process at The Apache Software Foundation had to change, and no one else was going to do it.

Greg and I started with the vendors and invoices for one department: Infrastructure. That was the area Greg was responsible for, and it also was one of the most financially complex parts of The Foundation. In addition to being the Infrastructure Administrator for The Apache Software Foundation, Greg is also an Apache Subversion committer, so he also knows his way around the old system. This made it easier to clean up our old system as we transitioned.

With bill.com, the bill approvals process now looks like this:

• First bills come into an inbox. Intake of a bill consists of labelling it with a budget area (or more rarely splitting it across multiple budgets), saying who should approve it, and making sure the vendor is entered so that payment later can succeed. Until January 2020, Greg and I did all of the intake of all of the bills and vendors in bill.com. Since then, our accountant has taken on the task of bill intake, and not a moment too soon. I started a full-time job as Senior Software Engineering Manager at Grafana Labs in December 2019, and I needed to off-load tasks.
• Once bills are assigned to approvers, the approvers receive notification. They then either approve or decline a bill. They also have the opportunity to check that the bill is being applied to the right part of the budget. One thing bill.com does that Subversion can’t do is assign multiple approvers. For example we introduced a process by which very large bills are also approved by the President of The Apache Software Foundation. Another thing bill.com does is synchronize with QuickBooks Online. So invoices are automatically entered into our accounting systems as well.
• Once bills are approved, they land with the Treasurer. I check over the approvers (and now that our accountant is doing intake instead of me, it’s great that we get a two pairs of eyes on the approver list), and I make sure the bill makes sense. Because I’m following several mailing lists for The Foundation, and I know the budget that was approved by the board, I am rarely surprised by a bill. If something confuses me, I can ask. Then I input which account we pay the bills out of, and pay them from within bill.com. Payment is scheduled.
• Bill.com automatically starts the payment with enough time before the due date to be sure that it arrives on time.

By the end of June 2020, we were making all of our bill payments out of bill.com. And for every bill since then, I can tell you exactly when they were received, when they were paid, whether they were on time or late, who approved them, and what the payment method was. It took us a little while to find and cleanup the mistakes we made in the context of the old Subversion process. There weren’t a lot of them. Despite everything the accountant had kept things straight for the most part. But since fall of 2020, our error rate (not counting problems like a vendor forgetting to submit an invoice) has fallen to zero. At the same time, the approvals process is much easier. I’m no longer going through a batch of 15–20 bills all at once trying to match each of them up with a list of approved bills under time pressure to complete it in 1–3 days. And even better, I don’t have to be the “bad guy” in this process anymore. Responsibility is placed where it belongs: with the officers doing the approvals.

Now bill payment approval is a continuous slow drip of easy little tasks, rather than a once a month big bang of mind-numbing high-stakes detail-checking.

So. much. nicer.

So what did I learn here about change management:

• Developers will try to apply developer tools to solve every problem. Sometimes that’s not appropriate. : o)
• Lots of talk can sometimes block change. A bunch of Open Source software developers looked at this particular problem, opined that it would be relatively easy to program a solution, then didn’t program a solution and left the problem in place. The conversation drifted to another topic. That happened a couple of years before I stepped in. Yes, as we always say at Apache, the decision-making process needs to be documented on the list. But execution, by its nature, happens off the list. Having the conversation isn’t enough. In fact, if the opinions expressed in the conversation on the list are “loud” enough, that conversation can have the effect of disempowering people who were otherwise ready to take action.
• Make a plan. Communicate the plan. Over-communicate it. Explain why you’re doing it. Explain what you need from people. Be specific.(for example: “Rich, you’re up next. I need you to accept the invitation to bill.com. From now on, please make sure your vendors’ invoices are submitted by mailing them to the billing address. If you need any help figuring out the approvals process, please ask.” (thanks Rich, for your part in making this transition work!))
• Pain drives change. It had been recognized for years that bill approval in Subversion was sub-optimal. But when the problem made it nearly impossible to keep up to date with our payments for one of our vendors, we had to start fixing it.
• It’s the person who feels the pain who will put in the effort to make the change. The accountant had more work than necessary from the Subversion process, but that wasn’t new. They were being paid for their work, and they knew how to do it (sort of), so they weren’t going to drive change to a process to reduce their workload. As the Treasurer, I was the one with the real pain from the process.
• The person who wants the change needs to put in the work to lead it. That was me. I needed to spend my Easter entering vendors. I needed to spend my weekends explaining next steps to The Foundation officers, or this would never have happened. Having done that, I could ask for minor inconveniences from my fellow volunteers.
• You can’t drive change alone. Greg was there with me the whole time, helping evaluate options, doing data entry with me, coordinating with his vendors, pointing out problems and solutions. David Nalley, the newly-minted ASF President, was supporting and providing cover in the background.
• …and despite all of the above: it is relatively easy to drive change at a small organization like The Apache Software Foundation once you get started. The officers almost immediately loved the new solution. This made it possible for me to complete this transition while also completing my master’s thesis on Vendor Neutrality in Open Source and turning it in on the 31st of July 2020.