Pierce County Washington is up to "software bug number 3" in their new Instant Runoff Voting system. We blogged earlier this year about two other major software bugs in Pierce County's IRV voting system, first in June and then in September of 2008. Now there is a report of a third software bug in Pierce County's instant runoff voting software. This time it is a capacity issue - the software can't handle more than 200,000 votes at a time. This is a real problem since with IRV you end up with often 3 times as many votes as normal, capacity is a very real issue. These flaws may also affect San Francisco's IRV voting systems, as the voting systems are Sequoia Insight Optical scanners using WinEDS tabulating software. And San Francisco has around 430,000 registered voters.
Once you legislate IRV, you are backed into a corner. Politicians and the media press hard for "instant results" that can't be provided for weeks.
When a jurisdiction adopts "instant runoff voting" they find out that it is too laborious and complicated to count by hand. That leaves them searching for voting machines that can do the job. Right now, there are no voting systems in the US that are federally certified to count IRV. Earlier this year, Pierce County officials requested emergency certification of IRV software that had bugs in it because "they tried hand-counting just 14 RCV ballots with seven ranked contests and found that it was “horrendous.” Using software to tally this sort of balloting was absolutely essential." Ironically, it took Pierce County Washington and San Francisco California several weeks to count their "instant runoffs".
Bug number 2: Saturday, September 13, 2008 Pierce County Instant Runoff System has new bug, says Washington SOS - may affect San Francisco ...This email from the Secretary of State of Washington outlines another bug in the new IRV voting system for Pierce County. It affects "rank choice voting" (IRV/Instant Runoff) only.
Bug number 1: Friday, June 27, 2008 Instant runoff forces Pierce County Washington to use uncertified voting systems ... As you can imagine, we objected to having a system certified when, in testing, it reported that there were "zero votes" to start, but actually had 56 votes already in its secret, unobservable "ballot box"!The Sequoia IRV software has a capacity issue in uploading and tabulating about 200,000 votes. Here is communication from an office of the Washington Secretary of State's office regarding the problem.
#
Bug number 3: From John Gideon on November 10, 2008
For those of you who are in states that use Sequoia and that may, in the future, be using WinEDS 4.0 here is some information from WA state where that version was used to support IRV in Pierce Co, WA. This is a string of emails between Ellen and a member of the SOS voting systems staff:
F.Y.I. – Sequoia has updated their documentation regarding the issue Pierce County ran into on Election Night -
Section 7.2.2 now contains the following recommendation:
Note: Excessively large 400-C batch loads may impede tally performance on election night. In extreme cases, the SQL memory, which is utilized for processing the tally, may be exceeded. In this situation, the following error message will display:
Tally process terminated early because of a trappable error.
If you receive this error, reduce the 400-C results batch size and re-import the results. Many factors, including the complexity of the election definition, can influence the allowable batch size that may be loaded. To ensure successful election night operations, Sequoia Voting Systems recommends loading 400-C results in batches not to exceed 25,000 ballots.
I have added this note to my certification folder so that we can make sure it is highlighted when we certify the WinEDS 4.0 system.
Thanks,
Patty Murphy Voting Systems Support
Office of the Secretary of State
(360) 902-4188 Fax (360) 664-4619
PO Box 40229 520 Union Ave NE Olympia, WA 98504
From: Murphy, Patty Sent: Monday, November 10, 2008 12:26 PM
To: 'Ellen Theisen' Cc: John Gideon; Miller, Paul; Hamlin, Shane
Subject: RE: Pierce County RCV
Hello Ellen,
Regarding the Nov. 6 article (http://www.thenewstribune.com/news/local/story/529926.html ) concerning delays with election results reporting in Pierce County – they were not hardware related. The memory that Pierce County added to the computer system was not actual physical memory. And it did not relate to the RCV algorithm, but it related to a miscommunication from the vendor to the county regarding moving results to the reporting system.
This is the vendor’s response -
Pierce encountered a delay on election night when they attempted to import their 400C results from WinETP (the tabulation server). WinEDS 4.0 (the reporting software) is designed to import 400C results in batch sizes on the order of 25,000 ballots or less. Pierce had consolidated all of their 400C results on WinETP into a single batch size of over 200,000 ballots. When they encountered the import constraint, we instructed them to increase their SQL Server memory configuration in order to successfully process the batch. This was a change made to the MS SQL Server configuration parameters only and had nothing to do with physical memory on the server computer.
This is another case of a vendor not communicating system usage limitations to a county. It was in no way Pierce County’s fault, but the vendor’s lack of proper communication regarding this upload protocol. However, the vendor responded quickly, and helped them solve the issue on election night. I’m sorry we didn’t catch this during testing – in retrospect, if I had thought to test combining all of our volume testing into one batch, and then uploading it to Win EDS for reporting, it may have caught this. This was partly an oversight on my part regarding the test scenario. I don’t think the national test laboratory would have caught this either because they would have uploaded results to Win EDS based on the vendor’s recommendation of 25,000 ballot batches.
So yes, reporting of results was delayed, but it was not due to a limitation in processing power, but a software limitation that was not properly communicated to the county. It is now properly documented in county procedures, and I have made a note to alert the other Sequioa counties to this protocol when the software is nationally certified and available for their use. (And when I say ‘limitation’ – it is not necessarily a bad thing. Uploading smaller batches has other benefits – more selective and efficient back out of batch loads, and audit comparison advantages when comparing Win ETP reports with Win EDS uploads.)
Let me know if you have any other questions. Patty Murphy Voting Systems Support Office of the Secretary of State (360) 902-4188 Fax (360) 664-4619 PO Box 40229 520 Union Ave NE Olympia, WA 98504 pmurphy@secstate.wa.gov
From: Ellen Theisen
Sent: Monday, November 10, 2008 9:24 AM To: Murphy, Patty Cc: John Gideon Subject: Pierce County RCV Patty, I was very disappointed to read in a November 6 news article regarding the memory that had to be added to the Pierce County system in the middle of tabulation:
"McCarthy said that while her office had tested the software previously, it did not test it with anywhere near the volume of ballots processed Tuesday. She said Pierce County is one of the first to use the software, so the problem could not have been foreseen."
But it was foreseen. If you remember, after the May hearing, I asked about that very issue.
I emailed you: "On the drive home, I was thinking about Denver in 2006, and trying to anticipate what RCV processing problems might come up that could be prevented. I began wondering whether the Pierce County system had sufficient processing power, disk storage, RAM, and cache to successfully do the processing required for 400,000 RCV ballots. It has to be significantly more stressful on a system than processing traditional ballots."
You responded: "Second, regarding processing power - the county purchased a new server, and when we extrapolated the processing power needed to cover the RCV ballots - they have plenty of power.
The RCV algorithm itself takes very little power in the CPU world." I do hope the extrapolation wasn't entrusted solely to Sequoia. Their estimate for what would be needed in Denver fell quite short. VotersUnite is in a position where we can sometimes pass on information to election officials based on the experiences of others. With that in mind, what else was learned from the experience in Pierce County that might be useful for other jurisdictions introducing RCV in the future?
Thanks,
Ellen Theisen Co-Director http://www.votersunite.org/
-- John Gideon Co-Executive Director VotersUnite.Org http://www.votersunite.org/
