The Dirty Little Secrets of Voting System Testing Labs

by Avi Rubin

The following URL is to a Blog by Avi Ruben. He recently attended, as 

A presenter, the California Secretary of State's Summit on voting 

Systems testing. What he reveals about the current state of the testing of

proposed voting systems is scandalous. We have repeatedly been told by

local election officials, who have been told by the vendors, that the

voting systems have been thoroughly tested by federal Independent

Testing Laboratories.

Avi Rubin :  Dr. Aviel D. Rubin is Professor of Computer Science and

Technical Director of the Information Security Institute at Johns

Hopkins University. Professor Rubin directs the NSF-funded ACCURATE

center for correct, usable, reliable, auditable and transparent

elections....

http://tinyurl.com/9cpbk

-----------------------------------------------------------------------

Full text:

The Dirty Little Secrets of Voting System Testing Labs

Avi Rubin, 12/16/2005

http://www.huffingtonpost.com/avi-rubin/the-dirty-little-secrets-_b_12354.h

tml

A couple of weeks ago, I spoke at a voting system testing summit 

hosted by the Secretary of State of California, Bruce McPherson. It 

was an event that included members of the US Election Assistance 

Commission, Secretaries of State, local election officials, vendors, 

voting machine testers, representatives from NIST, social scientists 

who study voting issues, and computer scientists, such as myself.

Most notable by their absence were Wyle Laboratories and Ciber Inc. 

Let me explain.

Before election officials can purchase voting systems, those systems 

need to be certified by a federally accredited lab called an 

Independent Testing Authority (ITA). There are three such labs in 

the US: Ciber, Wyle Labs, and Systest. These labs are tasked with 

testing any proposed voting systems against federal standards, in 

this case, the 2002 federal standards, soon to be replaced by the 

2005 voluntary voting system guidelines (VVSG). You would think that 

these labs would be very interested in attending a summit such as 

this, and in fact, they were all invited. Only Systest showed up.

There were several overriding themes that emerged at the voting 

systems testing summit. Perhaps the most prevalent one was that the 

ITAs consistently decline to appear at these meetings. Why? Well the 

main reason is that they are fraught with conflict of interest and 

incompetence. In fact, had they shown up, they would have been raked 

over the coals by some of the voting system examiners that attended 

the summit. For instance, an examiner from Pennsylvania wanted to 

know how come so many systems that passed the ITA testing still had 

serious security and even operational flaws. The Systest 

representative, who had the misfortune of representing his entire 

industry alone, replied that they were only required to test against 

the standard. When pressed about whether or not the ITAs would fail 

a system if a serious flaw was found, the reply was that a memo 

would be written, but that the system would still pass. I couldn't 

believe it. The company that was tasked with certifying machines for

  elections in the United States would still pass them, even if a 

 serious flaw was found, as long as the machine did not violate any 

 aspects of the standard. Unbelievable.

Now, let me talk a bit about the conflict of interest. As a friend 

of mine put it, the ITAs are not independent and they have no 

authority. So Independent Testing Authority is a misnomer. 

Thankfully, NIST is going to change the name next year. Here's where 

it gets bad. The ITAs are hired by and paid by -- the vendors. That 

is, when a vendor has a voting machine that they want certified, 

they find an ITA who is willing to certify the voting machine. Any 

memos about flaws that are discovered remain confidential. There is 

no requirement to disclose any problems that are found with the 

machines. In fact, the entire ITA report is considered proprietary 

information of the voting machine vendor. After all, they paid for 

it. This provides an incentive for ITAs to certify machines, to 

satisfy their clients.

Two years ago, my research team got our hands on the code that runs 

inside of Diebold's Accuvote machines. We performed a source code 

analysis and reported all kinds of serious security problems (see 

http://avirubin.com/vote/analysis/). It was incredible to me that 

such machines were actually deployed and used in elections. Equally 

confounding was that a national testing lab, in this case Wyle Labs, 

actually certified this machine. Either they did not know the first 

thing about cryptography and security, or they did not look at the 

source code. In fact, according to the 2002 standards, they were not 

required to examine the code.

So, the current state of affairs is grim. The ITA model provides an 

incentive to certify bad systems, and clearly such systems are being 

certified all the time. When the ITAs find a serious problem, it is 

relayed, confidentially to the vendor, and the only thing that the 

public ever learns is that a machine was certified. If a machine is 

not certified, nobody ever learns about it. The ITAs are aware 

enough of how broken the system is that they mostly hide from public 

events where they might be taken to task.

Here's how I would reform the system. First off, I would have all 

the vendors pay a tax to NIST. NIST would then hire real independent 

testers to examine any voting machine proposed by a vendor. The 

testers would be paid more for finding problems with the machines 

than for certifying them. Thus, you can be sure that the testers 

tried every way of failing a machine before passing it. Everything 

done by the testers, every test performed, as well as the result, 

would be public. Occasionally, to keep the testers on their toes, 

NIST would throw a machine at the testers with a known serious 

problem, just to see if the testers could find it, and testers who 

did not find the problem would be penalized. The whole process would 

be open and transparent to the public. I doubt systems such as the 

2003 Diebold Accuvote would have ever made it to a polling station 

in that model.

I learned a lot at the voting system testing summit, and I applaud 

Secretary McPherson for the dialogue that he opened up. I sincerely 

hope that in such events in the future, there will be no 

stakeholders who are afraid or ashamed to show their faces.