FOR THE RECORD: Human Error, Not Technological Failure, Causes MPD Radio Outage
Milwaukee Police Department
Office of Media and Communications
October 7, 2010 - The outage this morning to the Milwaukee Police Department’s radio system was the result of human error and not failure of technology or equipment.
A Harris Corporation Employee performing a system upgrade to the Milwaukee Police Department’s radio system inadvertently caused an outage at about 11:20 a.m. on October 7. The system was completely restored and operational in approximately 30 minutes.
The Harris Corporation employee remotely accessed our system without first notifying our Communications Center. The upgrade inadvertently corrupted the system server, which activated a security feature. Shortly after the security feature was activated a back-up server was brought online that began to reregister the 1800 system-wide radios.
Officers were able to communicate using their mobile data computers and cell phones during the transition to the backup system. The Milwaukee Police Department received about 20 calls for service during the outage. All of the calls for service were successfully dispatched to squads.
The Milwaukee Police Department swiftly established protocols to ensure that this type of situation does not happen in the future.
Milwaukee Police Chief Edward A. Flynn discussed the issue on WTMJ Radio 620. Click to hear the radio interview.
Harris Corporation, the radio vendor, made the following statement to the Milwaukee Police Department about the issue:
“We understand the impact to the safety of your officers and appreciate the fact that the system is under significant public scrutiny. We sincerely regret the event this morning and have already taken initial steps internally to prevent a similar occurrence in the future. This will not happen again.
The outage began at approximately 11:20AM and the system was fully restored by 11:50AM. The outage was not the result of failure in the technology itself. A Harris staff member proceeded with a maintenance process that should have been completed during a time of low call volume.
As part of the preparation for a system upgrade, the radio database server (UAS) had been taken offline [earlier this week]. A staff member brought the server back online – a process that normally would have no impact on live operations. In this situation however, recent changes to the network caused a port to shut down which prevented normal database synchronization. This in turn caused the live database to be unavailable, which disconnected all users from the system. The staff member restored the system by activating the port and re-provisioning the database to all radios. In short, this event occurred because of the system upgrade preparation, and is not a possible failure during normal operations.
It is because of the potential for unforeseen errors such as this, that this process should have been completed during a time of low-call volume and after full communication with your staff. We understand and agree with your decision to remove remote access, although we recommend that it remain available for the Harris Technical Assistance Center on an as-needed basis.
As you know, Harris is committed to professional and competent service and has been in the public safety business for many years. We deeply regret the error today and assure you it will not happen again.”
