Cleaned up, templatized, and modernized monitoring infrastructure, going from an unmaintained Nagios 3.x installation to Icinga, Graphite, Graphana, and NLMA-ng. Quintupled the number of checks and integrated alerting with our existing paging system. Upgraded an existing, unmaintained Atlassian stack (JIRA, Confluence, Fisheye/Crucible) to the latest versions, migrated to SSO (Crowd), added Git support (Bitbucket Server) and moved from an older server in a closet to a highly secure environment in a DC, all with no downtime or interruption in service. Became the Subject Matter Expert (SME) on a new internal tool, helped architect the stack, wrote documentation, deployed it, and maintained it in production, within 2 months of starting. Became the SME on an existing service that is used world-wide by Apple, taking over duties including reviewing configuration, deploying, monitoring, and troubleshooting. Mentored junior members of the team, in both technical and social/personal matters. Helped transition multiple projects from SVN to Git, including training and coaching developers who had never previously used Git. Represent the team at multiple meetings a week, working closely with developers and engineers to set priorities and direction.
Stark & Wayne ( Sept. 2014 – Apr. 2015)
Worked with large companies, including Intel, Cisco, and GE to help them roll out deployments of CloudFoundry, an opensource Platform-as-a-Service (PaaS). On a two-person team that succesfully implemented a system to have a “one-click” deploy of CloudFoundry on Amazon’s AWS platform, using BOSH, terraform.io, and shell scripting. Participated in community building by open-sourcing various git repositories, and attending/contributing monthly CloudFoundry conference calls. Learned enough about CloudFoundry and BOSH to help teach a one-week course for 20 students in January.
Lead Systems Administrator
Synacor — Systems Infrastructure ( Apr. 2009 – Sept. 2014)
Originally hired as a Senior Systems Administrator, to help maintain and expand the linux infrastructure, support dozens of websites written in PHP and Perl, and provide technical direction for the group. Mentored other sysadmins on the use of puppet for maintaining servers. Integrated puppet into kickstart and almost all day-to-day tasks. Evangelized and trained release engineers, systems architects and developers on using puppet to automate installation and configuration. Completed expanding a central authentication system using ldap, with replication across seven datacenters, to all machines and most internal web applications. Took over packaging duties, and expanded the number of packages built in-house from around 200 to over 2400. Integrated existing bind configuration with source control, wrote scripts to automate syntax checking, configured staging servers, and documented the process of releasing updated zone files, drastically reducing the frequency and impact of DNS errors. Helped expand from two datacenters to seven, and more than quintupling the number of servers from 600 to 4000, while the number of admins was reduced from eight to seven. Was in a two-man 24×7 oncall rotation for eight months, and during that time reduced the average number of calls from eight per weekend to two. Proposed, designed, and taught a course on Basic Linux Administration to operators in our NOC, including two people who became sysadmins on my team. Was the Systems representative in almost all architectural decisions made over the last five years. Was the Lead Systems Administrator with seven direct reports including all standard management responsibilities (approving PTOs, writing annual reviews, recommending promotions and pay raises/bonuses). Managed not only my team, but also the DBAs and the Storage Infrastructure team for six weeks, reporting directly to the VP of IT, while my boss was out on medical leave. Received multiple performance-based bonuses, as well as additional stock option grants, outside of the annual review process.
Freelance Unix Systems Administrator
GNUconsulting (1999 — 2014)
Installed, configured, and maintained multiple Linux servers for 30+ clients. Helped launch a website that needed to stand up against traffic from “Shark Tank” viewers (http://geekchichq.com). Used multiple cloud service offerings, including Amazon Web Services (AWS), Linode, DigitalOcean, and Rackspace. Helped migrate companies from service to service, including helping to architect the new setup to use Elastic Load Balancing (ELB), Simple Email Service (SES), Route53 (hosted DNS), Amazon’s RDS (hosted database) and CloudFlare (hosted DNS+caching).
Hired to help with the website’s transition from Solaris/SPARC to Linux/x86. Helped maintain the main website, running on ATG (a java application server), as well as multiple secondary websites running on Tomcat. Implemented OpenLDAP to manage accounts and passwords. Implemented automated, network-based installation of servers. Helped configure and implement host-based firewalls. Participated in a two-man on-call rotation (i.e., at one point it was down to two people in my group). Helped in migration to a new colo. Was the lead admin in charge of installing, configuring, and maintaining a Tealeaf cluster and Scene7 (dynamic imaging). Helped with migration from gigabit ethernet to ethernet over InfiniBand. Helped train multiple new Sysadmins in unix skills and Cabelas.com specific knowledge. Was the “go-to geek” in a group of Unix Sysadmins. Wrote multiple production-level scripts, including a multi-threaded load tester for Scene7. Helped maintain four 9′s uptime on a website with over 100 million dollars in annual revenue. Received multiple performance-based bonuses.
Unix Systems Administrator
Micron — Research & Development (May 2004 — Sept. 2006)
Brought on to manage the department’s transition from Solaris/SPARC to Linux/x86 on desktops and servers. Grew from less than 60 Redhat WS3 workstations and servers to over 850. Implemented automated, network-based installation. Took initiative to implement network and performance monitoring software (Nagios) covering all 1000+ machines on our network. Took initiative to install and implement CUPS, a network printing system, for all Linux and Solaris machines. Wrote modular testing program in Python which was used worldwide within Micron to check for standards compliance. Installed and maintained multiple apache web servers, utilizing modules from php, SSI, to Siteminder (a Single Sign-On solution). Performed routine system administration tasks, such as adding and removing users in NIS, troubleshooting issues, installing, upgrading, and moving machines, installing and upgrading software and writing small scripts in Perl and Python to automate repetitive tasks. Took initiative to rollout and use VNC for remote troubleshooting and end-user help. Took initiative to write and maintain FAQs for end-users new to Linux or our network, and for the other System Administrators on Linux/Solaris differences. Received $35,000 in performance-based raises. Received multiple performance-based bonuses.
Micron — Product Engineering (Oct. 2000 — May 2004)
Installed and maintained multiple Apache and Tomcat web servers. Wrote and maintained multiple CGIs in Perl and Python. Installed and maintained network and performance monitoring software (Big Brother) for 100+ Solaris machines. Helped maintain and update in-house software which used Redbrick and Sybase databases. Maintained software written in Perl that generated postscript files from test data. Led conversion from sccs to cvs for revision control. Helped convert 100K+ line C++ Solaris program to run on Linux. Helped maintain 200+ Solaris workstations. Installed and maintained pilot program of 5 Linux servers. Received multiple performance-based raises.
Member of WNY Ruby Brigade November 2011 - April 2015
Spoke at the Buffalo Database User Group on Puppet and Databases October 2014
Presented “Systems Administration for Developers” at Buffalo Barcamp April 2014
Presented “How to take down a 30,000 person company with 1 Line” during lightning talks at Nickel City Ruby September 2013
Spoke at Basho HQ in Boston, MA about Synacor’s Riak Usage for a meetup August 2013
Presented “Why Systems Administration is Awesome” at Buffalo Barcamp February 2013
Education & Training
Red Hat Jboss for Administrators JB336 June 2008 Red Hat Linux Networking & Security Administration RH253 June 2006 A.A.S. in Computer Networking, I.T.T. Tech March 2001