Broadband Maintenance

Concord Broadband performs routine maintenance on the equipment that provides internet connectivity to customers, from the Optical Network Terminals (ONTs) in customers' homes and in businesses, to the servers and network switches/routers that provide the backbone for all communications. This service ensures the highest level of security and reliability of our network.

Much of the maintenance performed does not impact customers, but we always notify people of any work that could impact our customer's experience. Unless it is deemed an emergency and when possible, maintenance is scheduled for 1-6am on Sunday mornings.

October 2022 Outage Information

On Monday evening, October 24, 2022, Concord Broadband experienced an outage for 63 confirmed customers lasting in some cases up through 4:35pm on Tuesday, October 25, 2022. This is inconsistent with the service we strive to provide to our customers, and as such we have compiled the following comprehensive report that outlines what happened, what staff and contractors did to resolve the problem, and how we can improve afterward.

10/25/2022 Concord Broadband Post-Incident Analysis Report

Because the report is long and highly technical, we have assembled the following summary, but please see the full report should you have questions after reading the summary.

Background
On Thursday night, October 20th, a crew from a third-party vendor was on-site in Concord to replace of a cable damaged by rodents last year. When this cable was cut, we now know it triggered a software glitch in the equipment used to provide internet service to customers.

The glitch manifested in several different ways, affecting areas of the network not in the vicinity of the Laws Brook repair work. The impacted customers were returned to service quickly, and the manufacturer of the equipment began troubleshooting. This work continued all weekend by multiple Concord Broadband staff members.

Outage
By Monday, a plan was in place to attempt to remedy the situation when suddenly a number of customers lost connectivity after no intervention or actions on the part of staff or our vendor. Because this was not a disruption due to customer hardware failures or a break in the fiber, Concord Broadband was unable to determine the precise number of customers impacted. Based on total bandwidth in use, we knew the total did not exceed 10% of customers (~160).

Concord Broadband staff worked with our vendor non-stop to determine the cause and an appropriate response. The vendor was reluctant to point to the fiber repairs as the root cause, and absent a reason, they had to escalate the case until developers and engineers were involved.

By 4pm on Tuesday, there was enough data to suggest which physical device had suffered a software failure, and the decision was made to reboot it. There had been a reluctance to reboot it earlier without more data because an ill-advised reboot could have widened the outage.

After the reboot, the issue was immediately resolved. No configuration changes were necessary. Any temporary changes made for a small number of customers during the investigation phase were reversed. Concord Broadband staff worked with customers who had issues after the resolution, most of them requiring a reboot of their ONTs.

Issues
While Concord Broadband believes the response and communication were much improved from the last unplanned outage, communication – in particular once the helpdesk vendor implemented a recorded phone message – could have been better. A total of 3 News and Notices posts and 10 Tweets were issued during and after the outage, and all customers who provided an email address were kept up-to-date during restoration efforts.

Thanks
Thank you to all the staff who spent countless hours in service of the team and our customers.

Should you have any questions after reading the full report, please reach out to us at broadband@concordma.gov. Thank you for your continued support of this community resource.



Upcoming Maintenance

No maintenance is scheduled at this time.