My Journey to Secure the Business from Within

A Lost Decade

For much of the first decade of my career, I worked on security teams that were “outside” the business — meaning they were understaffed, under-appreciated, and somewhat mysterious — constantly trying to justify their existence and value.

In the early to mid 2000s, folks knew “security was important” (and were definitely attracted the idea of hackers and hacking, as they still are today) but nobody knew how or why or what to do with it. They kept you around through the layoffs and reorgs, but didn’t do much with your team, neither growing nor killing it, but it was never aligned with the business.

These were the early years when billion dollar security BUs were still a just dream — the days of the first DoS attacks and of rampant Internet worms — long before computer/network security became called “Cyber.”

As a member of a product security team working closely with PMs and engineering to define and implement assurance activities within the development cycle (threat modeling, attack-based security testing, etc.) or as an external consultant performing vulnerability assessments, penetration testing, or design reviews — I was still an outsider. I was not an active, participating member of the design, development, and operation of products, services, systems or components that were delivering business value.

Sure, you might work with product teams early in the (waterfall) development process but you were still an outsider. Best case, you address some issues prior to release to avoid having to disclose some vulnerabilities or dealing with independent security researchers. Worst case, you are just checkbox or rubber stamp or feel good step prior to the final release. Occasionally, a product marketing team might want to you to do competitive vulnerability research for sales teams, which most of us viewed as unethical and dangerous. Once I had a Director admit their product line was “insecure”, but didn’t care. He just needed me to find dirt on up and coming competitor that was rapidly gaining market share.

In many security roles, the job was just to “raise awareness” and “identify and communicate risks” but ultimate the Business must decide what to do — or not to do. They are always right. Maybe you convince the Business. Maybe you don’t. Whether it is it “secure enough” is not your call. The product or service will be released with design flaws because you must gain market share. You must acknowledge “there is a business to be run” and security (and even compliance) goals can be overruled if they put the Business at risk. As in The Phoenix Project, if the enterprise goes out of business (or the product doesn’t launch) there will be nothing to secure. So step aside and let the business do what they need to do.

As a security professional, this was a recipe for failure and frustration. It is no surprise that I began to feel restless. I wanted something more. Yes, it was still fun to find stupid security holes, but writing reports that would just get filed away started to get old. Really old. Dealing with the slow moving SCADA security during “the lost decade” exacerbated this frustration, so I left security work to actually build and run things.

First Steps

In late 2006, I left the world of vulnerabilities and security testing (mostly) for good. (I would briefly come back to this in 2010 for a few projects and realize, once again, that I didn’t like this sort of work anymore.)

This was also my first time doing Ops and the first time I’d ever worked in an IT organization or been on call. The Ops part was cool, being part of a large, extremely siloed, 3000 person IS organization with helpdesks and NOCs and so many teams with poor handoffs, not so much.

But building BSD-based security appliances for a large Enterprise was not only fun, it was building and testing and deploying a product. The other engineer and I who worked on this thought of our work as a product with releases. More importantly it got me thinking about managing large numbers of systems across multiple environments, which I would need later. Most importantly, it saved the business money. Even paying 2–3 senior engineers to develop and maintain high availability security appliances built on commodity server hardware had clear and demonstrable value to the business. It was much cheaper than the hardware and software and support costs. But it was not really “doing security.”

Three years (and 2 jobs later) would be the first time I started to work from within the business to secure things, but this was also (once again) in the context of building things. Security is about building from the inside not breaking from the outside. This is how you find value.

I was on a project to design and deploy a new wireless metering system for military bases in the Washington D.C. area. My team (and the initial reason my company partnered with another integrator) was to help complete the DIACAP package and to navigate the compliance bureaucracy necessary to connect our systems to the military wide area network.

Like any good contractor does (especially when doing T&M work), we expanded our efforts beyond the initial SOW and took on a greater role in the project. Instead of just running the customary Retina and Gold Disk scans and completing the required paperwork, we became critical members of the solution using compliance as a foundation for real security, helping to design the wireless network, the security zoning, architect application and infrastructure components to balance security, usability, and manageability.

Because we deeply understood the technical and business problems of our solution (and the necessary security controls) we were able successfully negotiate and justify our design to DoD Information Assurance (IA) teams.

The same is often (and unfortunately) true for siloed corporate security teams who have no understanding, awareness, or interest in the business or the underlying technology or applications when it strays from their Enterprise comfort zone. This problem just as true for Industrial Control Systems or the Public Cloud. What you do not understand you cannot possibly secure.

Help Me Secure the Business

Fast forward to my current role (skipping the good parts where I decide to leave my cush and lucrative defense contractor role, taking a sizable pay cut for the chance to make a bunch of mistakes to start to “learn DevOps” thing at console gaming company has never done an online before and then go onto to launch a SaaS security analytics platform) where now I not only have technical responsibilities for a global Ops team but am the product owner for a services business. I am responsible for driving the transformation from an “on premise” to “services” mindset across the value chain from development through delivery and operation.

So what has changed? I know I have changed. Changes come through mistakes, failures, bad decisions and flawed perspectives.

But the industry has changed, too. With Cloud, DevOps, Rugged Software, and devsecops I’ve never been more confident that security can find its rightful home. And that home is within the business, as close as possible to where business value is defined, prioritized and created — where there is context. But this change requires centralized, top-down, security teams to be dismantled and re-oriented. Security must be pushed from the cost centers to the profit centers where the real work of building and running things really happens. Security must no longer exclusively be an auditor or an advisor, but actively build things and be part of building things.

And this is easier said than done. I’m just getting started. I need help. And I’m sure I’m still getting many things wrong.

I’ve just opened a DevOps Security Engineer position. I’m hoping to find someone that is less cynical, less jaded, than I was when I jumped ship. My ideal candidate has 4–5 years in a security role and is beginning to grow sick of a being part of a centralized security/complaince team whose job is to makes sure the boxes are checked or say no — but who often gets overruled.

Ideally I’m looking for somebody in the Maryland, DC, or Northern Virginia area so we can get together face to face once a week. But depending on the candidate I’m open to locations outside the region.