Introduction
Agile means “quick, well coordinated – marked by an ability to think quickly.” It is often used to imply the flexible ability and willingness to be nimble and change direction.
Federal procurement officials are facing “Agile” more and more, particularly in the information technology (IT) arena. Agencies such as DHS, HHS, and GSA indicate that they rely on the Agile approach to IT development 42%, 49%, and 72% of their software respectively (versus a target of 60%). As well, several agencies, including GSA, are experimenting by asking their procurement offices to operate on Agile principles – in addition to buying from vendors who use those same Agile principles. (Please see http://agilemanifesto.org/principles.html.)
As pressure mounts to use agile, Federal CIOs frequently note that they see acquisition processes and habits as an obstacle. In response, this white paper describes the challenges and opportunities associated with Agile procurement and identifies ways to overcome those challenges in the Federal Government.
What is Agile IT?
In brief, the Agile IT approach calls for breaking large, complex projects into small pieces, collaborating with customers to design each piece as it is being built, inviting customers/users to try it, and provide feedback (or new ideas) before continuing to build the next small piece. Each small piece is informed by real use of the previous group of pieces.
The core benefit of the Agile IT approach is that a product evolves in real time to meet the needs of real users. It results in better software, more responsive to the user’s needs, faster and at less cost and overhead than the traditional waterfall method of software development. There is less risk that a committee will design and build a clunker when what the customer really wanted is a sports car. Unfortunately, it is rarely possible to precisely specify all of the requirements – what is to be built – in advance of the procurement. Thus, it can appear to be a challenge to get good price competition and to tell if an agency is getting what it wanted and what it paid for. Instead, a solicitation and requirements should accommodate flexibility, user collaboration, and incremental builds that are offered by Agile software development.
Acquiring Services from an Agile Provider (Buying Agile)
Many Government procurement officials are nervous about contracts to purchase Agile software and IT solutions, and perhaps for good reason: It’s hard to feel comfortable buying something when you aren’t sure what it will be.
But most Federal agencies report (in OMB PortfolioStat briefings) that more than half of their IT development efforts use the Agile approach. And, the benefits of Agile are powerful. An effective Agile approach brings innovation as development takes place, moves more quickly than other approaches, and rarely produces a solution that an agency finally rejects and discards.
Thus, learning to procure services from vendors who use an Agile approach is important now and will grow increasingly important in the coming years. This white paper suggests how to do so effectively.
If the Scope is Vague, How Can We Get Competition on Price?
A key benefit of Agile is a scope that can be vague and that is allowed to change. A vague scope reduces the time and effort needed to develop precise requirements. It also allows requirements to change as users learn and provide feedback incrementally while a system is developed. But, that ideal of precise requirements in advance has offered comfort to Government that it will get strong competition on the basis of price – thus a lower risk of “paying too much.” Vendors also like the idea of precise requirements because they perceive less risk – less fear that they have made a mistake in setting their price.
However, the ability to set complete requirements accurately has proven elusive. The Agile approach recognizes that precise requirements introduce risk, too – risk that requirements can’t be known perfectly, and that fast-evolving users won’t adopt a system built to the specification of outdated requirements. Thus the effort required to create precise requirements isn’t the best way to obtain price competition.
In theory, an agency should prefer to define outcomes it seeks (“customer use-cases” in Agile parlance), and seek price competition from vendors who may propose new ways to achieve those outcomes. However, outcomes may also be challenging to define precisely, and there is typically some risk faced by vendors in determining how to achieve a defined outcome. Thus, price competition based on a well-defined outcome is desirable but has been difficult to obtain.
Since Agile development occurs in short periods of time called ‘sprints” that are usually set to intervals of one or two weeks, price competition should be based on the vendor’s proposed cost of a standard Agile team week, and the number of weeks quoted to complete a project. Some agencies may choose to define the “standard” team composition, but different vendors may naturally prefer to offer different teams (smaller or larger, more or fewer design staff vs. programmers). As long as a vendor defines the team they will deliver, and quotes the duration of effort needed, the agency will be able to determine the total estimated price multiplying the team’s weekly price by the number of weeks the vendor proposes. The Government should anticipate 10-20% variance in the number of weeks estimated. Of course this price competition should be balanced (probably in a best-value format) against the qualifications of team members, technical approach, and satisfaction of previous customers with past performance in Agile development efforts.
Finally, to balance price concerns against desires for team expertise, quality, and performance, agencies may invite several vendors to actually perform the first sprint of a major project. (The first small increment of work.) Following that, the agency may select a vendor based on real performance, rather than on written promises and assertions.
Success Factors and Mistakes
Key success factors include:
- Identify user groups and real users to participate in developing use cases, review designs, and provide feedback on the product as it is developed incrementally.
- Break development into increments that each have real value, that can be used by customers, and that build upon each other.
- For a large program, engage several vendors to provide Agile teams so that an agency can continue/increase use of effective vendor teams but decrease/ discontinue use of ineffective teams.
- Specify the types of work that are needed well enough so that vendors can assign the best-suited team members.
The most significant mistakes include:
- Procurement awards based entirely on low price. This will eliminate a vendor’s motivation to accommodate evolving agency and customer needs/learning.
- Failure to identify a clear end date, completion criteria, and Product Owner. Each of these is critical to controlling cost and establishing mutual understanding.
- Agency staff fail to adequately participate in the Agile process (especially as a “Product Owner”). If key Government individuals do not have time to participate, the schedule or the entire process may be compromised.
- Hesitance to design and release partial functionality to real users. (Prevents real feedback, and incremental adoption.)
- Acceptance of a design that does not provide valuable working features at each sprint. (Allowing an agency to terminate an agreement mid-way, but still have a valuable product.)
How Can We Tell if We Got What We Paid For?
When an agency procures an Agile IT solution, the Government and vendor must collaborate, accepting mutual responsibility for risks and outcomes, rather than expecting one or the other to shoulder all of the risk. Thus, some firms offering Agile services will expect to be paid on the basis of effort provided (i.e., team-weeks worked).
Value should be determined several ways:
- Was the solution developed quickly, within schedule, and on-budget?
- Did users adopt the solution (likely because they collaborated in designing, testing, and providing feedback on it throughout development)
- Did the solution achieve the desired mission impact? (likely because the “Product Owner” directly guided development, while accepting user- and developer-proposed improvements as the project progressed).
Since Agile relies upon user collaboration, and since many IT projects fail because users do not adopt the result, adoption and user satisfaction may be the most important and observable measures of success. Thus, the Government could award a portion of contract value (perhaps 10% to 20%) upon achievement of an agreed-upon level of user adoption. This would make an Agile contract, in effect, a performance-based contract. An adoption measure, for example, could be “90% user adoption within 90 days after launch.”
As a bonus, this focus on user adoption will motivate vendors to consider organizational change management – including potential behavioral obstacles for users to change and use the new solution – thus motivating inclusion of users, customers, and “change champions” up front. It will also likely produce better design and quality assurance, and will likely result in more effective user training/ support as each Agile increment is launched.
What Type of Contract Should an Agency Use to Buy Agile Services?
In most instances, an agency will want Agile IT development services because it is uncertain of precise requirements, wants innovation from the contractor, and expects greater value from collaboration between developers and customers. This expectation of uncertainty most naturally lends itself to a time and materials (T&M), cost-plus-fixed-fee (CPFF), or cost-plus-award-fee (CPAF) contract format. Typically, agencies using these contract formats also set a dollar-value ceiling and a specific, finite duration to minimize the risk of over spending.
In some circumstances, agencies will prefer a firm-fixed-price (FFP) contract format. This makes sense if requirements can be described clearly but there is a desire for Agile services because the agency values incremental delivery and collaboration with customers/users during development.
A hybrid alternative (though rarely used) is to contract based upon a fixed price per sprint. This allows price competition, but it may be challenging to define how many sprints are needed, and to ensure that the same product quality will accrue from sprints performed by different vendors.
Buy Based on a Sample
Since the Agile framework delivers useful increments of software (or other project deliverables), purchasing agencies have the opportunity to ask for a demonstration or sample in order to help them evaluate vendors. This can be useful to obtain ideas, to evaluate vendors’ capabilities, and to meet a specific team that would support the agency. Recent procurements by GSA and USCIS have each used this method (GSA for its Agile Delivery Services Blanket Purchase Agreement and USCIS for Flexible Agile Delivery Services II).
An agency’s contract should likely be valued at $20 million or more before inviting multiple vendors to deliver a real Agile product as their proposal. For a vendor to do so, its team of four developers might spend a two-week sprint to create the product at a cost of at least $40,000 (80 hours x 4 developers x $125/hour, each). If the vendor expects a 20% chance to win, she would thus plan to spend $200,000 for each such win. Typically, IT vendors earn about 6-8% profit for T&M Government contracts, and expect to dedicate no more than 1% of revenue to business development. Thus, for a contractor to be willing to spend that $200,000, the value of the contract should be $20 million. If an agency offers a smaller contract, it may find few vendors willing to provide a real Agile deliverable as their proposal.
Finally, however, where the agency obtains a real Agile deliverable as the proposal from vendors, the agency will likely receive many creative ideas to address the problem(s) it faces. For example, GSA’s 18F Agile Delivery Services solicitation received a working software artifact from more than 100 proposers (with no proprietary restrictions on use of that artifact). Many of the ideas represented by those proposals/artifacts were unexpected and quite creative. (Note that GSA invited vendors to develop software in connection with a Food and Drug Administration (FDA) data set, rather than using a real problem faced by GSA itself.)
For a smaller-scale procurement, agencies could use a “hack-a-thon” model – a single session in which vendors receive a challenge, and then plan, design, develop and present their working software solution all within an 8-hour period. This reduces vendor cost (8 hours x 4 staff x $125/hour = $4,000), and the approach may thus attract vendors with a smaller contract value, but, the ideas will not likely be as well developed or executed, thus selecting the best vendor may be more challenging.
Using Agile Methods to Acquire IT Services (Being Agile)
At the same time agencies are being asked to buy services from Agile vendors, leadership is asking whether Agile principles can be used within the agency itself – often starting with the Procurement Office, because that Procurement Office is expected to know how Agile works, since it is buying Agile IT development services from Agile vendors. Unfortunately, the Agile methods used to develop software do not all translate directly into methods that can be used to purchase goods and services. There are, however, a number of parallels, and some beneficial lessons to be had.
What does Agile mean with respect to Procurement?
There is no consensus – the term and practice are too new. But, here’s what we believe Agile procurement should evolve to mean:
- Ensure that procurement efforts are aligned to your agency’s mission – and sufficient to achieve mission success.
- Break big procurement efforts into smaller tasks and pieces – pieces that can be accomplished incrementally.
- Procure quickly – but accommodate changes to the needs of program officials and their customers/users.
- Use a challenge model to try a “free sample” from several vendors (under a BPA, or an open competition), rather than simply relying upon written assertions and descriptions of what the vendors propose.
- Make it possible to add (or remove) vendors to support a program, and to switch between vendors, so that the agency can adjust direction quickly as program increments are delivered.
- Collaborate with users to try the results of each increment in their real-world setting, and to provide feedback and new ideas that can help inform subsequent pieces of the effort.
- Accept (in fact, welcome) the occasions when customers ask for a change – it means they are paying attention, care about what you’re doing, and that they plan to use the result.
- Measure success based upon outcome achieved, and upon user adoption – use of goods and services purchased versus those that are rejected or discarded.
Experience at REI Systems
REI has begun managing our own Contracts Office (as well as other corporate functions such as HR and Finance) using Agile principles. Each team is focused on four goals this year – which helps avoid fragmenting our attention across too many goals.
REI’s Contracts Director reports that “What makes this groundbreaking, in my opinion, is for the first time in my 20+ year career, there is a level of communication and cooperation all focused on achieving the goals that incorporate short-term goals into achieving the long-term goals.”
Hurdles in the Federal Context, and How to Address Them
A focus on the lowest unit price doesn’t always produce the lowest cost. Federal procurement often focuses on low cost for the individual item or service. This focus neglects to recognize individual items that are rejected or discarded after purchase. An agency that buys ten different software applications, but discards six because they don’t meet user needs should not consider their procurements a success regardless of the price for each of the ten applications. If an Agile developer charges a unit price that is 50% higher for each of six applications, the agency will still be better off than if it purchased all ten from a traditional vendor, but only six were accepted by users. The agency that bought ten each for a price $1X (total cost: $10X), paid more than the agency that bought six for a price of 1.5X each (total cost: $9x).
There are good reasons for contracting staff to remain independent from program staff – but they can still collaborate. The separation of responsibility in Government between procurement and program staff ensures that a single person cannot determine the need for an item, and then negotiate the purchase of that item – this reduces the risk of corruption. But, a key benefit of the Agile approach comes from collaboration between those doing work (procurement staff) and their customers (program staff) – and from the ability and willingness of Agile team members to step into each other’s roles to make sure work is done quickly. (They could jointly serve as “Product Owner” during service delivery by an Agile vendor.) Although Agile procurement should promote collaboration between procurement and program staff, care must be taken to avoid allowing team members to perform each other’s roles.
Measuring success in procurement will always be challenging – the Agile approach will not change that. As with Agile IT, user acceptance is an important supplement to traditional measures of value and procurement success, and cost must be considered, but considered carefully. Ultimately, however, the core measures of procurement success tend to be: did we get the right items and services that we need to carry out our agency’s mission?, and did the outcome actually advance our agency’s mission? These are subjective – satisfaction measures – and thus may be drowned-out by focus on driving costs down, ensuring compliance, and avoiding corruption. Nevertheless, these difficult-to-capture satisfaction measures may best demonstrate benefits from Agile procurement.
About REI Systems
Since 1989, REI Systems has developed and sustained decades-long customer relationships by providing innovative solutions that ultimately impact millions of peoples’ lives. From supporting the infrastructure and software that disburses more than $20B in grants for more than 1,700 Federal programs each year to building and sustaining advanced analytics and data visualization platforms supporting the last two US Presidents’ Open Data initiatives, our solutions are innovative and key to the infrastructure of our nation. As a missions-first Government technology solutions provider, we specialize in agile software development, CI/CD, DevOps, application modernization, and platform-based solutions. Our more than 500 employee-owners pride themselves in delivering meaningful and sustainable results that consistently exceed our customers’ expectations.
Authors
Pradeep Krishnanath, REI Senior Account Director
Kevin White, REI Senior Director of Contracts
Jeff Myers, REI Senior Director
