Hagai Bar-El on Security

I have been running a security research group at Sansa Security since 2006, and while I think about it often, I never bothered to publish any post about how to run an effective security research team. So here is a first post on this topic, with an anticipation for writing additional installments in the future.

I will address a few random topics that come to my mind this moment, about staffing, external interaction, being in the know, and logging. Feel free to bring up other topics of interest as comments to this post.

Team members and roles

There are tons of books on hiring in general, and I have no intention of replicating text of any of them. I will try to address only some specific requirements for security research teams, stemming from their nature. Of course, the people need to be security professionals, but there is just a little more.

Security is a wide domain with many sub-disciplines. Therefore, focusing on "security" in CVs of candidates is not enough; you need to be more specific. On the other hand, recruiting professionals in each relevant domain that you may encounter, especially if your area of research and development is wide, is impractical. So what should you do? From my experience, you should hire based on horizontal skills rather than on vertical ones. Focus on people who can familiarize themselves with new domains, that have a wide enough background, and most importantly -- people who are capable of carrying out the horizontal tasks that the team deals with.

For example, if the team has to research security protocols in several domains to build suitable security products, then your people shall have wide backgrounds and the ability to learn new subject areas, as well as have the specific horizontal capability of protocol analysis. This is not a groundbreaking principle; it is common in engineering domains to hire based on task-centric capabilities rather than on subject matter familiarity; programmers, for example, are hired based on their programming skills, even if lacking any experience in the domain. I still find it necessary to bring up this recommended approach because in research teams this approach is less common. A security research team is probably somewhere between academic research and engineering, and I believe that hiring based on horizontal capabilities (task skills rather than domain familiarity) is the more effective way.

It follows that you need to be able to tell what operations the security research team deals with, and based on those to allocate your racks. Typically, there are three classes of operations that require skillful personnel, at least in the type of security team I manage:

Design and system engineering

If the team is part of a company that produces security products, then aside of knowing how to secure systems, you would need to know how to devise useful security products. For companies that produce security-related products, security is not just a property of the product (i.e., "a secure product"), but also its core functionality. The product is not only secure, but it actually implements security as its function.

Deciding what the product or component should do to be worthwhile is more difficult than making the product secure. Even if the company has a "Product" group, the core of products that solve security problems is the novel mitigation technology that the company offers, and this likely falls within the responsibility of the security research team. For example, if a company makes a firewalling product, then aside of having security people checking that the product is secure, it will have security people defining what the core technologies of this firewall should be to make it valuable in its target market.

Someone in your team will need to have the creativity, positive problem solving mindset, and system knowledge to handle this activity.

Security Engineering

Regardless of whether or not your research team invents new technologies, it is very likely to be tasked with the role of reviewing design and implementation of systems, against security flaws. This often consists of three types of review:

• review of design documentation,
• review of implementation (a.k.a., white-box testing), and
• adversarial testing (a.k.a., black-box testing).

It is possible that the person on board for carrying out the security engineering task above will also review design documentation. It is less likely (yet possible) that the same person doing the design will also carry out the source-code review and/or adversarial tests. Security engineers, as they gain their experience with high-level architectures, often drift away from coding (if ever been there), and by the time they become experienced system engineers, their coding ability is a bit rusty. For source-code review, the reviewer shall be a top notch programmer on top of being a security practitioner. Otherwise, he might not be able to properly understand the code written by other excellent programmers and his findings may be limited to trivial memory leaks and missing bounds checking.

The people who do security reviews must also love what they are doing. Reviewers need to be people who have an adversarial mindset and who love uncovering broken stuff. A bad security review can only be told at hindsight, so if the reviewer is not motivated to do a good job -- you have zero chance of getting one.

Intellectual property

This topic seems to not belong here, but it does. A research team that does not merely review the specifications and code of others also generates intellectual property, and may be involved with the intellectual property of others. Obviously, the help of a legal firm is required, even if just for filing patents. However, a lot of IP-related activity is best taken care of internally.

Whereas filing and prosecution of patents is managed by a law-firm, for patents with actual essence and higher value, the technologist that invented the technology, or someone well versed in it, shall be the first one to point out the special merits of the invention. Also, when a provisional patent application is filed on a two-day notice, it is best to maintain the capability of drafting it properly within the team.

Lastly, if there is a desire to carry out a patent-search, those searches are often best carried out by a technologist in the field who is experienced with IP search, rather than by a search firm. The security and secrecy of the process is yet another concern.

When patents are filed, the actual claim wording shall be done (or confirmed) by a patent attorney, who shall also care for the legalese. However, specifying the technology in a way that is properly usable by a patent attorney, pointing out what the key claimable elements are, where a level of dependent claims is required, and what elements shall remain unclaimed until, say, a continuation application is filed, are all aspects that are usually better served by a technologist who understands IP, rather than by a lawyer who understands technology.

Interaction

A security research team is a service team, and requires efficient means of interaction with other parts of the organization. First, part of the technology developed by this research team gets into larger systems and products managed by others. Second, the team often engages with security review of work done by other teams.

Few organizations have well-established protocols that specify the interaction with the in-house security research team. The level of involvement of the security team with the rest of the organization more often depends on the charisma of the team leader and on the security mindset of the other managers. The exact points in the development processes where the security team is consulted are not always well-defined. Therefore, I sometimes find it essential to have a liaison of the security team work closely day-to-day with other teams. This liaison, who often is not even formally appointed, hints when a wider involvement may be desirable, and may form a first-line of review when the need arises.

Being in the know

People and processes shall be put in place to ensure that participants do not lose touch with their domains of security expertise; otherwise the value of the research team deteriorates over time. This may be true for everyone and particularly for every research team, also outside the domain of security. However, since security engineers cannot neglect their familiarity with the subject matter domains, other than the security domain per-se, the magnitude of this mission increases.

You had better not hope that people find the spare time to keep abreast of new developments on their evenings and weekends. I spend some time every weekend reading books on security and otherwise, and still have a pending reading list in front of me that will take the rest of my lifetime to complete. Members shall actually have time assigned for studying, along with priorities (that must match their own interests). Aside of sync meetings and round-tables, there is little control over how this time is spent. However, if you do not trust your people to want to maintain their knowledge, then you simply need to hire other people.

While you can never monitor studies, and probably should not want to anyway, it could be useful to lead by example. Building up a digital library using Calibre, for example, and sharing your own preferences and recommendations, can certainly help.

What should participants read? Aside of some very good textbooks, there is a lot of content online in professional blogs and podcasts. I remember once even establishing an internal RSS syndication where I put links to posts that I found interesting, for the team (and other teams) to consume. Other than professional blogs and podcasts, there are also press releases, but those I found to be less of interest, other than for overall market know-how.

Individual vs. team work

This is an important issue. How much work should each individual do on his/her own and how much shall be done as a team? I know of managers who call in all members to discuss every single topic and others that treat team members as individual silos. Like with everything else, I believe the sweet spot is somewhere in between.

Getting many team members involved in every action has the obvious advantages of multiple views. However, it multiplies the cost of work, adds friction at times, and if done incorrectly might also violate another important principle which is that of accountability. So where is the sweet spot? My personal view is as follows:

• Security reviews are done individually. The work may be done by multiple people, and moreover, critical parts may be checked by more than one person, but still, every piece of the work is done by an individual alone. I personally see no room for round-table work on security reviews, other than for sharing the findings.
• Design and new technology development is divided into two parts from this respect: the core hard stuff, which are the problems that you really don't know yet how to solve, and all the surrounding technology and specification, which takes work, but is more "linear" and less risky. The core hard part is done by multiple people working entirely together, until a solution is found. The rest is done individually by multiple people, yet not in an isolated fashion. Each participant is responsible for some parts of the design and the rest act as his/her reviewers.

Logging

I have a lot to say about how I think IT issues are best handled, but for a moment it will suffice to say that proper logging cannot be over estimated. By "logging" I refer to the practice in which everything is documented, not just the eventual deliverable. Here are a few reasons for keeping note of the entire processes rather than just of its outcomes.

1. It just makes sense. Knowledge generation is much of what you do, and knowledge is not just in the outcome.
2. Any process in which intellectual property is generated and may need to be protected needs to be such that leaves a clear trail. This does not necessarily refer to the legally binding "lab notebooks" that have some legal value as admissible evidence in patent cases, but more to exercising "due process", which allows you to ascertain the order of events to yourself as well as to others. Such records are useful, for example, when determining inventorship and when dealing with IP that does not have a clear ownership model.
3. When security reviews are carried out, they usually apply to living products that are re-submitted once in a while. Tools for "diff"-ing submitted revisions need to be handled with care, as most security issues live in the cracks, but having a clear record of the modules that were already reviewed with certain results can save an immense amount of work.
4. Whether we like it or not, the work of a non-academic security research team, both as a review team and as an inventing team, is highly influenced by priorities of its customers and by the priorities of the organization it serves. As a result, context-switches are a frequent event. A project can be suspended and resumed a dozen times before it gets complete, with the same people switching tasks back and forth. It is not desirable, of course, but it happens and will keep happening as long as the team remains relevant in the organization. The role of the leader is to assure that such context switches have the lowest possible cost. Working with proper journals assures that when a participant gets back to a previously-suspended project, he/she can effectively resume from the point at which he/she stopped.