I have been running a security research group at Sansa Security since 2006, and while I think about it often, I never bothered to publish any post about how to run an effective security research team. So here is a first post on this topic, with an anticipation for writing additional installments in the future.
I will address a few random topics that come to my mind this moment, about staffing, external interaction, being in the know, and logging. Feel free to bring up other topics of interest as comments to this post.
There are tons of books on hiring in general, and I have no intention of replicating text of any of them. I will try to address only some specific requirements for security research teams, stemming from their nature. Of course, the people need to be security professionals, but there is just a little more.
Security is a wide domain with many sub-disciplines. Therefore, focusing on "security" in CVs of candidates is not enough; you need to be more specific. On the other hand, recruiting professionals in each relevant domain that you may encounter, especially if your area of research and development is wide, is impractical. So what should you do? From my experience, you should hire based on horizontal skills rather than on vertical ones. Focus on people who can familiarize themselves with new domains, that have a wide enough background, and most importantly -- people who are capable of carrying out the horizontal tasks that the team deals with.
For example, if the team has to research security protocols in several domains to build suitable security products, then your people shall have wide backgrounds and the ability to learn new subject areas, as well as have the specific horizontal capability of protocol analysis. This is not a groundbreaking principle; it is common in engineering domains to hire based on task-centric capabilities rather than on subject matter familiarity; programmers, for example, are hired based on their programming skills, even if lacking any experience in the domain. I still find it necessary to bring up this recommended approach because in research teams this approach is less common. A security research team is probably somewhere between academic research and engineering, and I believe that hiring based on horizontal capabilities (task skills rather than domain familiarity) is the more effective way.
It follows that you need to be able to tell what operations the security research team deals with, and based on those to allocate your racks. Typically, there are three classes of operations that require skillful personnel, at least in the type of security team I manage:
If the team is part of a company that produces security products, then aside of knowing how to secure systems, you would need to know how to devise useful security products. For companies that produce security-related products, security is not just a property of the product (i.e., "a secure product"), but also its core functionality. The product is not only secure, but it actually implements security as its function.
Deciding what the product or component should do to be worthwhile is more difficult than making the product secure. Even if the company has a "Product" group, the core of products that solve security problems is the novel mitigation technology that the company offers, and this likely falls within the responsibility of the security research team. For example, if a company makes a firewalling product, then aside of having security people checking that the product is secure, it will have security people defining what the core technologies of this firewall should be to make it valuable in its target market.
Someone in your team will need to have the creativity, positive problem solving mindset, and system knowledge to handle this activity.
Regardless of whether or not your research team invents new technologies, it is very likely to be tasked with the role of reviewing design and implementation of systems, against security flaws. This often consists of three types of review:
It is possible that the person on board for carrying out the security engineering task above will also review design documentation. It is less likely (yet possible) that the same person doing the design will also carry out the source-code review and/or adversarial tests. Security engineers, as they gain their experience with high-level architectures, often drift away from coding (if ever been there), and by the time they become experienced system engineers, their coding ability is a bit rusty. For source-code review, the reviewer shall be a top notch programmer on top of being a security practitioner. Otherwise, he might not be able to properly understand the code written by other excellent programmers and his findings may be limited to trivial memory leaks and missing bounds checking.
The people who do security reviews must also love what they are doing. Reviewers need to be people who have an adversarial mindset and who love uncovering broken stuff. A bad security review can only be told at hindsight, so if the reviewer is not motivated to do a good job -- you have zero chance of getting one.
This topic seems to not belong here, but it does. A research team that does not merely review the specifications and code of others also generates intellectual property, and may be involved with the intellectual property of others. Obviously, the help of a legal firm is required, even if just for filing patents. However, a lot of IP-related activity is best taken care of internally.
Whereas filing and prosecution of patents is managed by a law-firm, for patents with actual essence and higher value, the technologist that invented the technology, or someone well versed in it, shall be the first one to point out the special merits of the invention. Also, when a provisional patent application is filed on a two-day notice, it is best to maintain the capability of drafting it properly within the team.
Lastly, if there is a desire to carry out a patent-search, those searches are often best carried out by a technologist in the field who is experienced with IP search, rather than by a search firm. The security and secrecy of the process is yet another concern.
When patents are filed, the actual claim wording shall be done (or confirmed) by a patent attorney, who shall also care for the legalese. However, specifying the technology in a way that is properly usable by a patent attorney, pointing out what the key claimable elements are, where a level of dependent claims is required, and what elements shall remain unclaimed until, say, a continuation application is filed, are all aspects that are usually better served by a technologist who understands IP, rather than by a lawyer who understands technology.
A security research team is a service team, and requires efficient means of interaction with other parts of the organization. First, part of the technology developed by this research team gets into larger systems and products managed by others. Second, the team often engages with security review of work done by other teams.
Few organizations have well-established protocols that specify the interaction with the in-house security research team. The level of involvement of the security team with the rest of the organization more often depends on the charisma of the team leader and on the security mindset of the other managers. The exact points in the development processes where the security team is consulted are not always well-defined. Therefore, I sometimes find it essential to have a liaison of the security team work closely day-to-day with other teams. This liaison, who often is not even formally appointed, hints when a wider involvement may be desirable, and may form a first-line of review when the need arises.
People and processes shall be put in place to ensure that participants do not lose touch with their domains of security expertise; otherwise the value of the research team deteriorates over time. This may be true for everyone and particularly for every research team, also outside the domain of security. However, since security engineers cannot neglect their familiarity with the subject matter domains, other than the security domain per-se, the magnitude of this mission increases.
You had better not hope that people find the spare time to keep abreast of new developments on their evenings and weekends. I spend some time every weekend reading books on security and otherwise, and still have a pending reading list in front of me that will take the rest of my lifetime to complete. Members shall actually have time assigned for studying, along with priorities (that must match their own interests). Aside of sync meetings and round-tables, there is little control over how this time is spent. However, if you do not trust your people to want to maintain their knowledge, then you simply need to hire other people.
While you can never monitor studies, and probably should not want to anyway, it could be useful to lead by example. Building up a digital library using Calibre, for example, and sharing your own preferences and recommendations, can certainly help.
What should participants read? Aside of some very good textbooks, there is a lot of content online in professional blogs and podcasts. I remember once even establishing an internal RSS syndication where I put links to posts that I found interesting, for the team (and other teams) to consume. Other than professional blogs and podcasts, there are also press releases, but those I found to be less of interest, other than for overall market know-how.
This is an important issue. How much work should each individual do on his/her own and how much shall be done as a team? I know of managers who call in all members to discuss every single topic and others that treat team members as individual silos. Like with everything else, I believe the sweet spot is somewhere in between.
Getting many team members involved in every action has the obvious advantages of multiple views. However, it multiplies the cost of work, adds friction at times, and if done incorrectly might also violate another important principle which is that of accountability. So where is the sweet spot? My personal view is as follows:
I have a lot to say about how I think IT issues are best handled, but for a moment it will suffice to say that proper logging cannot be over estimated. By "logging" I refer to the practice in which everything is documented, not just the eventual deliverable. Here are a few reasons for keeping note of the entire processes rather than just of its outcomes.
A brilliant article, has extremely depth understanding on executing and running a security team in real business.
Hiring based on task-centric capabilities rather than on subject matter familiarity is an excellent point. It shouldn’t be limited and only applied for engineering domains but all.
The sweet spot of this article that I enjoy is to the joint and balance of academic and non-academic!
Hi Hagai, like the insights just want to add on that in parallel to the above there is also an evaluation in the nature of the management positions and expectation from them towards the activity of security, and security research just as an example the logging topic, or any data related topic there is a strong need to bring value and intelligence rather than mastering the ability to create correlations etc. etc.
Fabulously enlightening.
Will you talk about conflict resolution?