Using artificial intelligence to reduce DoD’s procurement timelines

Terry Gerton I guess applying AI to the federal acquisition process really seems to be a question of when and how, not if. I mean, commercial firms are already doing it. What would be the most significant benefits from deploying AI into the [Defense Department]’s procurement process?

Wilson Miles Yeah, so good question. So there are a range of benefits. And to kind of frame the problem just a bit, it’s quite well known around the community that the acquisition workforce is quite overburdened. They’re swimming in data. Simultaneously, they’re asked to move faster and faster. And there are some worries that the contracting workforce in particular is not at the adequate size for the pace of global threats and the pace of technology. And so there are ways to support that workforce, which is what the paper is trying to get at through using different AI tools to do things like market research faster, use ChatGPT-style tools to assist in developing questions for vendors, for example. And so this is really an idea of moving faster throughout the acquisition process to deliver capabilities to the warfighter.

Terry Gerton Are there other challenges that you think AI could solve beyond merely the understaffing? And I say merely; that’s a huge problem. But are there other aspects of the process that you would think AI would bring a competitive advantage to?

Wilson Miles Yeah, absolutely. So like you said, there’s the staffing part, there is also the investigation benefit to these tools. So it’s really good at pulling and analyzing large amounts of data to help answer previously unanswerable questions, or questions that would previously have required a lot of human ingenuity that frankly, acquisition professionals just don’t have time to do. And so there’s the staff augmentation aspect and then there’s the investigatory benefit as well.

Terry Gerton Your paper reports that DoD’s tried a couple of pilots in the Army and the Air Force. What have they learned from those pilots in terms of a broader deployment of AI in this space?

Wilson Miles Certainly that there are benefits. I will say that one of the challenges with those pilot programs that we’ve heard is that there’s not a solid baseline for understanding the before and after equation in this. So for example, one of the things that you want for these tools is to show a concrete return on an investment. And so there’s a good understanding that these pilot programs are indeed helping address some understaffing problems, for example, but in the organization writ large, it’s hard to understand how much time certain tasks are taking. And so there needs to be a clear distinction of “this tool is helping this specific problem and that the gains are X amount.” And so I think that’s a really interesting problem to think through, which is anecdotally, there have been some ideas of how long certain tasks take, but there’s nothing quantitative about how how long different responsibilities are and how AI has improved that particular task.

Terry Gerton So you could imagine different AI tools for different segments of the acquisition process and a measure of success might be how much time they save, but you don’t know how much time it takes on the front end to start with, right? You don’t have the baseline.

Wilson Miles Correct. I mean, time is obviously one aspect to this, but there’s also competition. There’s also retention of contracting professionals, for example. There’s this worry that because contracting professionals are so overburdened that they only stay at the positions for a couple of years. But if you start to see these individuals stay longer, that’s also a measure of success. And so when you think about different pilots or building your own tool, you need to be very clear about what is the problem that you’re solving. And this gets into questions like understanding and mapping your workflow and understanding what the gains you’re trying to get are and then evaluating the success of that tool.

Terry Gerton I’m speaking with Wilson Miles. He’s an associate research fellow at the Emerging Technologies Institute with the NDIA. So Wilson, the paper makes some specific recommendations to resolve these kinds of challenges. Tell us what you’re suggesting.

Wilson Miles Yeah, so I think that there’s been a clamoring for a program of record-style effort, specifically targeting acquisition tools. And so I know that there are these pilot programs. You mentioned the Army, [Air Force Research Lab] has their NIPRGPT, but there are different lines of effort. And so I think having something in a program element, for example, having a specific place for these, for the development of these tools to live would be one really important need. And then the second piece that I would mention is that offices that are interested in exploring using an AI tool, they really need to understand and map every step of their particular workflow, understand why each one exists, pick a high-frequency responsibility. That takes a lot of human effort, a lot of people, for example, were expressing frustration over justification and approval documentation, for example. And focus on predictability and understand that an AI agent that completes 20% of a test with high explainability, and zero surprises will outperform one that is 90% capable, but unpredictable. And so it’s just an understanding of how these things can fit. And then that will inform your investment. The third aspect that I will point to is that more pilot programs are needed. And so we recommend that Congress drafts a provision for the FY26 National Defense Authorization Act, urging DoD to develop a pilot program addressing a specific need, whether it’s market research, source selection, and outline a plan to use commercially available tools with a plan to scale that tool based upon certain metrics of success that we discussed earlier in hopes that that tool will eventually transition.

Terry Gerton So the very first one of those recommendations you mentioned was creating a program of record. I’m kind of surprised to hear you say that because that is the most un-agile way to deliver agile capability through AI back into DoD. Are you suggesting that the pilot program should be part of the program of record and that it should proceed through all of the regular milestone reviews and those kinds of things?

Wilson Miles No, it’s an interesting point. I think the complaint is that, for example, there’s no [Program Executive Office] designed to develop acquisition tools. And so we’re really looking for a more coordinated effort where different projects can live that are designed for acquisition tools specifically, more of those business operations tools that you don’t really see in a coordinated effort. And so I hear your skepticism and it makes sense to me, but I think our primary concern is finding a way to coordinate different efforts together because this is something that doesn’t receive a lot of attention. The acquisition community or the public debate right now is very much over the flashy platforms, but we really do need to improve the way that DoD approaches its business systems. A program of record might be one of those paths forward.

Terry Gerton I appreciate that clarity, and I want to circle back to your second point about workflows, but more broadly workforce. What kind of training would you imagine would need to go along with this kind of deployment to overcome some of the resistance of the acquisition workforce and really prepare them to optimize the use of these tools?

Wilson Miles It’s a really good question and I think that’s a question that a lot of people that I personally spoke with struggled over. It’s how do you incentivize a workforce that works in a particularly highly regulatory field to take those risks and understand what the benefits are that they can get from using these tools. And I admittedly will say that I’m not sure that people have a good answer for it. And we’re trying to figure out ways to force the workforce to use these tools. But in terms of hard skills that are needed when using different AI tools, it’s things like prompt engineering. So, thinking about how do I interact with this large language model, how do I ask good questions, how do I give a good context so that I get the information and analysis that I need to do my job? There’s one particular worry about skill decay, which is let’s just say we have this tool that helps someone do good market research. If this becomes so ingrained in our workflow there is a worry that people will lose the ability to do that on their own. And so there needs to be also some contingency planning of, hey, if this tool goes down, we’ll still be able to do our job. So I think those are just a couple of examples that I would raise.