For years, the first time the Department of Defense’s cyber forces faced high-end digital attacks was not in practice or in a classroom, but in actual operations.
For the cyber teams that focused on offense, a playbook developed from years of National Security Agency operations guided their work. But on the defensive side, standards and processes needed to be created from scratch meaning, in part, there was a lack of uniformity and little tradecraft to follow.
Because cyber leaders had focused on staffing, training opportunities for defensive cyber operators had been sparse.
To help solve that problem, the Department of Defense is expected to award a contract worth roughly $1 billion later this year for a global cyber training environment. But in the meantime, some units across the joint force have gone so far as to create their own small-scale training events and exercises to keep their forces’ skill sets sharp.
Perhaps the best example of these efforts are the 567th Cyberspace Operations Group’s “Hunt Event,” which has quickly grown to become one of the largest cyber exercises across the department. The bi-monthly exercise pits teams against each other in a competition for the coveted Goblet of Cyber trophy and bragging rights.
The group aims to better train defensive hunters, improve defensive tactics, techniques and procedures and develop defensive tradecraft.
“The point of this was that we didn’t really have a good range space to play on that had an active and live adversary so we could, in theory, replay traffic and we could go in and generate some easy kill, low hanging fruit signatures for detection,” Capt. Reid Hottel, training flight commander at the 837th Cyber Operations Squadron, told Fifth Domain.
“If we are supposed to be the primary counter to advanced persistent threats, the way that we were training was not like how we were fighting.”
The exercises started roughly a year ago to teach operators how to hunt on networks. It’s now evolved to where participants also work on leadership skills and build custom exploits on a large range with multiple stakeholders.
In addition to the Air Force CPTs — the defensive cyber teams each service provides to U.S. Cyber Command — members from the Air Force Office of Special Investigations and Mission Defense Teams, specialized defensive cyber teams that will protect critical Air Force missions and local installations, also take part. At the most recent exercise in January, a representative from NASA participated. Now, the exercises have become so popular Hottel said other services are interested in participating in the future. This includes a Marine Corps CPT at Scott Air Force Base.
Building better leaders and hunters
To be the best, cyber leaders recognized their teams would have to beat the best and that meant training against the world’s most advanced cyber threats.
Some other forms of training — such as the popular capture the flag game, which involve teams trying to find “flags” such as files or scripts inside a network — are not always the most realistic form of training.
“When we were fighting, we’re up against advanced adversaries. We’re up against adversaries that are using tactics, techniques and procedures that are just above and beyond what simple little [scripts] … we were using in the past,” Hottel said. “This hunt exercise allows us to do that, whereas in the past, particularly in other flag exercises, we are not training at the APT level. We [were] training at the script kiddie kind of level and here we’re training at a much higher difficulty, which stretches and grows our operators into being true hunters.”
He added that the exercises are also helping develop tradecraft.
“That’s one thing that nobody really teaches, there’s no commercial course that you can go buy that teaches tradecraft, that teaches the military away, that teaches the way that we use to find the APT, which in theory, should be ever evolving because our adversary is as well,” Hottel said. “These exercises have been really eye-opening to provide tradecraft development, to become hunters, to understand what it means to be a cyber protection team.”
The exercise has evolved to include custom exploits, custom root kits, custom attacks and zero-day exploits within a real-world mission where in some cases hunters don’t have any indictors of compromise that exist in the public domain. This means that there is no public reporting available on the exploits or tactics the adversary is using.
Participants can hone their skills, by actively hunting on a network in order to find anomalies that could lead to trouble.
“As hunters,” he said, “we don’t necessarily have singular methodology, we don’t necessarily have a unique way that we can go about finding advanced threats mostly because we haven’t really been training like that.”
The training is also helpful for new mission defense teams, which are just being officially resourced within the Air Force around local installations. By having those teams sit next to CPTs, who are using generally the same tools, they can learn about tradecraft and what to look for at the local level.
During the most recent exercise, officials said it was the first time they intentionally tried to trip up participants. Organizers created fake attack chains to see how the players scoped an investigation into a network and deducted points for the amount of time they wasted following that lead. This technique helps teach teams how to scope investigations without going down “rabbit holes,” and not adequately planning, Lt. Christopher Trusnik, chief of training at the 835th Cyberspace Operations Squadron, told Fifth Domain.
Beyond the technical hunting, this approach helped team leader to flex leadership muscles.
“It was more of teaching that leadership technique of you plan for this, how do you investigate quickly and how do you triage your investigation,” Trusnik, whose unit ran the January exercise, said.
Hottel explained that following this most recent event, teams focused on leadership and organization.
At one point, someone on his team previously had been coached on what they needed to include such as specific indictors that might be valuable to their mission partners to understand. At this exercise, they included those indicators.
In another instance, one team member who had never run a hunt mission struggled at first. Hottel stepped in and with just a little guidance, the leader became more disciplined and was able to find things much better in the last three days.
Benefits of cyberspace in training
Training in cyberspace has benefits that other domains don’t offer.
For one, forces don’t need a dedicated battlespace such as the Army’s National Training Center or the range used at Nellis Air Force Base for the Air Force’s Red Flag. With cyber, a custom range can be built and forces from all across the world can come in and participate.
The range used for the hunt exercises stays up weeks after the formal event so individuals or teams can try their hand, though they obviously won’t be eligible for the Goblet of Cyber trophy.
All of this could change with the Persistent Cyber Training Environment (PCTE). PCTE is a major program being run by the Army on behalf of Cyber Command and the joint force to provide a web-based cyber training environment where cyber warriors can remotely plug in around the world and conduct individual training, collective team training or even mission rehearsal — all of which does not exist on a large scale currently.
Hottel said that his forces haven’t been limited thus far without PCTE. Though, once the platform is online, they can upload the range they used for a competition and it can be accessed by anyone across the joint cyber mission force.
Testing new concepts
But in the meantime, smaller, unit level exercises like those run by the 567th allow forces to test concepts and learn from others. Unlike larger exercises that have requirements and stated objectives, smaller exercises can serve as a proving ground for staying sharp and pushing the envelope. This allows local units more control over what their personnel do but can also allow teams to test new concepts in a relatively risk-free environment.
“Let’s say that a national [cyber protection] team wants to test out … whatever they’re currently using because they feel like it would provide them an advantage so they want to test out something,” Hottel said. “We can throw that on the range as well and they can utilize an entirely defensive tool set. We’re not trying to make people tool experts, we’re trying to make them tradecraft, defensive hunters.”
Hottel also said that personnel playing on the archived range can bring new ideas, which can then be tested during the next exercise. In some cases, they may come up with an idea on their own and bring it to the next exercise to see if it actually works.
Ultimately, the event is designed to create better cyber warriors.
“We’re not trying to make people tool experts, we’re trying to make them tradecraft, defensive hunters,” Hottel said.
Mark Pomerleau is a reporter for C4ISRNET, covering information warfare and cyberspace.
