Work Stories | Matthew Devney

What's This?

A resume is intentionally compressed. Interviews are also short on time. This page is the long-form version: what the challenge was, why it mattered, how I approached it, and what happened next. Think of this page as an extension of the interview itself, with enough space to tell the full story.

Click any role to expand the context.

Fast Facts

1,700+

Security findings remediated through an AI-assisted SAST program.

30%

Annual cloud cost reduction from audit and remediation work.

~40 hires

L1/L2 operations team members hired and trained as founding lead.

~30 years

Production systems, DevOps, site reliability engineering, and incident response leadership.

Tools I Have Worked With

Roles I Have Held

Customer Success Engineer - L2 Team Lead

06/2021 – 06/2022

Nike had a problem. Nike had literally over 1,000 technical teams, and they generally need similar things. Every team needed a kubernetes cluster (for example), so every team spun up a kubernetes cluster, and we end up with 1,000 kubernetes clusters all configured differently, and who knows whether they're patched or secure or use proper RBAC or anything, really.

Nike's answer was to create a management interface so when a team needs a kubernetes cluster, they clicky-click through a form and a cluster gets spun up for them. This cluster starts in compliance with all the corporate policies, and it's maintained centrally so it remains in compliance. We called this a "platform". My team supported 14 of these platforms.

Support is part of any mature service offering, and that's where I came in. They created a "Customer Success Engineering" team and I was the team lead. They told me they were going to hire 40 engineers and I don't want to train that many people, so I created a handbook of how to be a Customer Success Engineer: all the access we need and how to request it; CSE-focused documentation on how to support our 14 platforms; I documented policies about how to interact with customers and dev teams, but first I had to create those policies. Met with dev teams and worked out SLIs and SLOs and SLAs, and made dashboards to visualize them.

I interviewed over 100 candidates, and we hired about 40 engineers in 6 time zones. The other major focus of this role was training and mentoring these engineers. In my role as Lead with decades of experience, I tried to pass on as much knowledge as I could, but more importantly I tried to teach these engineers modes of thinking that help with troubleshooting and problem solving.

At some point a corporate decision was made to outsource this work, so I smoothly passed the whole enterprise off to an external contract partner and moved on to my second role at Nike.

Lead Software Engineer - Build and Agile Tools

06/2022 – 04/2026

After a short stint on the GitHub team where I implemented some Actions to automate housekeeping, I landed on the Build And Agile Tools (BAAT) team which manages Jira and Confluence and a third Atlassian app Crowd which helps to manage those. My main focus was the Jenkins pipelines that roll out these apps.

Over the next year I added features to these pipelines: blue/green deployments, proper tagging, unit tests and code coverage, and of course optimizing for reduced cost. I was able to bring these pipelines into 100% compliance with all applicable corporate requirements and standards, and reduce cost by 30%.

Once I implemented code scanning our code scanner identified about 1,700 vulnerabilities across all the pipelines. I found AI to be very useful in resolving these, and quickly got the number down to 0.

Corporate decisions above my pay grade put our team in charge of Miro, Lucid, and Figma, three near-identical whiteboarding SAAS tools. Once again there were no processes, so the first thing to do was work out who does what and how things like escalations are done. The next thing was just to get a handle on the tool itself, so I used Cursor AI to create several dashboards to display availability and license info.

Bloomfire just had not concentrated on their software infrastructure for a while and it was all creaking with old age.

The version of ssh that they were using was so old that all the ciphers it supported had been deprecated, and chef-solo couldn't ssh to modern servers. Each time they spun up an instance they would manually ssh to that instance and enable old, deprecated ciphers.
The version of chef-solo they used depended on that old ssh, so they couldn't upgrade ssh. Also it wouldn't run on modern ubuntu because of openssl and ohai version incompatibility, and some dependency headers mismatch modern libraries.
The product apps were set up to start from Upstart, which was unavailable in modern versions of Ubuntu.

We called this the chicken-egg-egg problem, because each of these 3 parts (ssh, ubuntu, chef-solo) could easily be upgraded on its own, but that would break the system unless the other 2 were also upgraded at the same time.

First I switched the product apps from upstart to systemd. This theoretically allowed our apps to run on Ubuntu 18, if everything was set up properly. It was set up by chef, which wouldn't run on modern Ubuntu.

So I re-implemented it in Ansible. The head engineer wanted to move to ansible anyway, so I spent months with 2 windows up on my screen: reading chef on the left, writing ansible on the right. This got rid of both the ssh problem and the chef version problem both at once.

Once I had a minimum viable subset of chef re-implemented in ansible, then I did the ubuntu upgrade in production to address security concerns in the old ssh, old chef, old ubuntu. Then I completed the chef-to-ansible re-implementation project.

I completed one other major task here. The running installation of the SaaS product was running in a bespoke VPC hand-crafted years ago by the head engineer and added to by hand ever since. I created a series of rake tasks and terraform modules to roll out a new instance of the product in a different region.

There comes a time in a dot-com's life when it must stop being a dot-com and grow up into a professional company, and SambaTV was just at that cusp. I was hired to midwife that change:

Get the team to adopt agile processes and sprint cycles
Promote Infrastructure as Code and modern DevOps practices
Go through the whole SRE process of defining metrics, agreeing SLIs and SLOs, creating graphs and dashboards, all the way through updating architecture and actively improving response times and user experiences.
Move the entire installation. Where? That had not yet been decided when I was hired.

SambaTV had a couple hundred servers each at 3 physical datacenters around North America. Each of them was hand-configured so they were all a little different, and they each had seemingly unsolvable minor bugs. My very first day on the job they flew me to Las Vegas to visit a brand new datacenter they had leased as a potential replacement, and I spent the next week there mostly configuring network switching things like vlans and spanning tree.

Remote hands installed the physical servers and I set up preseed to install Ubuntu. Next came automation to install our app and register this server with the zookeeper cluster, load balancer, etc.

Our ops team had been hand-crafting nagios configs, and that doesn't scale. So one project was to replace Nagios with Sensu, which included updating and rewriting plenty of check scripts. We ended up ripping out the entire monitoring system top to bottom: write a Prometheus exporter to expose metrics, Grafana graphs and dashboards, created our own ELK stack to replace expensive Splunk.

There was a prevailing opinion among some of the engineering team that due to specific properties of this specific app, it could never operate fast enough in cloud. That theory had never been tested because nobody had ever gotten the server to run in a proper cloudy way. Sure we could spin it up on static ec2 instances, but it's better to have everything auto-scale and auto-tag and auto-everything. Which turned out to be not hard, just kind of a lot. I spent 3 months making it happen. In the end we found that AWS was plenty fast enough, but the datacenter cost less.

This whole time I was mentoring the team to help them change from an engineer who uses Ansible to run a command across 100 servers, to the kind of engineer who writes an Ansible playbook that we can commit to Git and run from a pipeline. That's what Infrastructure as Code means.

Senior Linux System Administrator

02/2014 – 11/2015

CNE Media was a small specialty video production company that did everything in-house: casting, shooting, pre- and post-production, writing, directing, marketing. They had 3 studios on site. They had some very unique issues. How do you store and back up multi petabytes of video going back 20+ years? (We settled on ZFS snapshots)

There were many processes that were undocumented and poorly understood, so my first task was software archaeology: read the code, find out what it does, and re-implement it in a modern, highly-available and reliable configuration. Implement all this in Puppet.

Since I was the infrastructure engineer I was also tasked with migrating the VM infrastructure from VMware to Citrix, as well as generally tuning anything that felt slow. Often MySQL, PostgreSQL, MongoDB, Redis, or Jenkins.

Site Reliability Engineer

12/2015 – 11/2016

I read about Google's idea of Site Reliability Engineering and convinced my manager to try it. I officially became a Site Reliability Engineer. I was tasked to implement enterprise wide backups with Bacula supplemented by custom Bash scripts around mysqldump and pg_dump. Implemented asset management software GLPI and OCSinventory; deployed agents with Puppet. Spun up and maintained virtual servers in VMware and Citrix XenCenter as necessary. Installed and maintained what services they needed, mostly LAMP. Did it all with Puppet. Infrastructure as Code is just so much easier and better than the old way of doing things and I was sold on the idea.