Los Angeles County has spent the last 10 years creating what it hopes is the voting system of the future, a $300 million fleet of cutting-edge machines built from scratch.
But as it prepares to roll out the new equipment for the first time when early voting in California's Democratic primaries kicks off next week, the county is in a race against the clock to shore up critical vulnerabilities highlighted in an alarming third-party assessment.
The technical report commissioned by the California secretary of state identified a wide variety of security flaws and operational issues, including insecure ballot boxes and exposed USB ports that rogue actors could exploit to alter votes.
"At first reading, it's terrifying," said Richard DeMillo, a computer science professor at Georgia Tech University who specializes in voting security. "There are things that are clear security vulnerabilities in the system that are brushed aside."
For more on this story, tune in to NBC Nightly News with Lester Holt tonight at 6:30pm ET/5:30pm CT or check your local listings.
L.A. County Registrar Dean Logan, who is in charge of the system, said the majority of the security flaws have been fixed, and the county has complied with the requirements set out by California Secretary of State Alex Padilla. Padilla just last month approved the system for use in the Democratic primaries so long as certain conditions are met.
But the questions raised by the independent vulnerability assessment are likely to hang over Los Angeles County until all of the ballots are tallied on March 3—and well beyond. Padilla gave the county until August to resolve some of the more complex security flaws.
While it's routine for testing to reveal kinks in new voting systems, and conditional certification is standard, experts say it's rare for so many issues to be identified this close to a big election.
"It's not unusual in certification testing for independent examiners to find issues with a system. That's why you do it," said Eddie Perez, a former executive at a major election systems manufacturer.
"But to be finding and reporting on those issues in late December before a February presidential primary when the system is being used for the very first time in a county this large, that's the concern," added Perez, who is now a global director of technology development for the Open Source Election Technology Institute (OSET), a nonprofit that conducts election technology research.
Voting systems around the country are under heightened scrutiny as a result of Russian election meddling in 2016. The Iowa Democratic caucus debacle, where technical snafus and communication breakdowns delayed the reporting of results, underscored the potential for mishaps.
But perhaps no municipality is facing more pressure than Los Angeles County. The largest voting district in the country undertook an audacious plan to revamp its antiquated voting machines amid growing concerns over malfunctioning equipment and inadequate security. The system was ripe for an overhaul—the technology dates back to 1968 when Richard Nixon defeated Hubert Humphrey for president.
Los Angeles County's goal? Build a publicly-owned and operated system and make it easy to use for all voters, including those with disabilities.
"If we make it inaccessible and complicated, or if we scare voters to the point that they think their vote doesn't matter, then we've destroyed the whole intent of having elections in the first place," Logan said.
The system — the first of its kind in the nation — relies on modern technology. Voters make their selections on a sleek touch screen, then the machine generates a paper ballot. The individual voter is expected to review the ballot to ensure its accuracy before casting it.
Official ballots are counted independently on a separate tally system.
The system, called Voting Solutions for All People (VSAP), gives voters the freedom to cast ballots at any of the county's 1,000 voting centers.
Logan said he believes it's uniquely suited for the size and diversity of Los Angeles county, which contains 5.4 million more voters than the combined total in 42 states. He emphasized that no special arrangements are needed for people with disabilities.
"This is the first system in the nation that allows a voter with a disability to vote completely independently" without "having to disclose that they have a disability and be relegated to a corner in the room and be treated differently," Logan said.
Election experts say handmarked paper ballots are the gold standard to prevent tampering and allow for recounts and audits. Some 70% of the country still votes that way.
The VSAP system relies on voters to ensure the accuracy of the machine-generated ballots. Inaccurate ballots caused by technical errors or any other issue would not otherwise be caught. But experts say the reliance on voter due diligence is problematic; studies show most don't take the time to inspect ballots.
Logan said his system produces a "human readable paper ballot" that is just as secure and still gives voters the chance to catch any errors. "The voter is in the driver's seat," he said.
But the technical report, released on Christmas Eve, identified several significant flaws:
— Open USB ports in which an attacker could insert malware and potentially alter votes.
— Widespread dissemination of source codes, too many people have access
— Ability to insert or remove ballots from ballot transfer boxes.
— Serious paper jams that require a full system reboot
The machines were in fact found to jam at five times the allowable rate in California. The malfunctions often destroyed the ballots during testing, meaning a potential voter would have to go through the process again. And the jammed machines would have to be stopped and restarted, resulting in significant delays.
"Those things were mitigated after testing," Logan said. "And one of the conditions is that we need to continue to look at that and improve on that."
Logan said most of the other issues have been corrected—through measures like adding tamper-resistant seals, locking vulnerable ports and restricting system access to a smaller pool of personnel.
But the county is also facing a lawsuit from the city of Beverly Hills over the way the touch screens display the names of candidates. The screens only display four at a time, requiring voters to press a button marked "MORE" to see the others. If voters instead press next, they won't see those names — a big problem in a place that has 30 contests with five or more candidates.
Logan said a large yellow circle has been added around a larger "MORE" button to make it stand out better. But he acknowledged that some of the more complex concerns, such as encrypting all server and workstation hardware, won't be addressed until closer to the general election.
As a fallback plan, Padilla, the secretary of state, has mandated in his conditional approval of the new system that all 1,000 new voting centers have paper ballots available on site.
Los Angeles County also has a vibrant vote at home program. Any voter can request to vote on paper at home and mail the ballot in or deliver it. Logan said that historically about 60 percent of voters ask for such ballots and roughly 30 percent of them end up voting that way.
In a statement, Padilla expressed confidence in the system.
"California is home to the most rigorous voting system certification testing standards in the nation," he said. "LA VSAP — like any other voting system used in California — had to be tested for months and identified vulnerabilities addressed before it could be certified for use."
But Perez, of the Open Source Election Technology Institute, said the number of outstanding issues and the limited time to resolve them poses a daunting challenge.
"It's very high stakes," said Perez. "It's the sort of thing that would keep any clerk recorder awake at night."