Rackspace hosted Exchange suffered a devastating blackout beginning December 2, 2022 and is still continuous since 12:37 AM December 4th. Initially described as connectivity and login issues, the guidance was ultimately upgraded to reveal that they were dealing with a security occurrence.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be dealt with.
Customers on Buy Twitter Verification reported that Rackspace was not responding to support emails.
This has been quite the day with #Rackspace. Every hosted exchange client has been down for 14 hours or so. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they came down with something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Unsure how many business that is, however it’s considerable.
They’re serving a 554 long hold-up bounce so individuals emailing in aren’t aware of the bounce for numerous hours.”
The official Rackspace status page used a running update of the blackout however the initial posts had no details other than there was a failure and it was being examined.
The very first authorities upgrade was on December second at 2:49 AM:
“We are examining a concern that is impacting our Hosted Exchange environments. More information will be published as they appear.”
Thirteen minutes later Rackspace began calling it a “connectivity issue.”
“We are examining reports of connectivity problems to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates described the continuous problem as “connection and login problems” then later on that afternoon at 1:54 PM Rackspace revealed they were still in the “examination stage” of the outage, still attempting to find out what failed.
And they were still calling it “connectivity and login problems” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
4 hours later Rackspace described the situation as a “considerable failure”and began using their consumers totally free Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround till they understood the issue and might bring the system back online.
The official guidance specified:
“We experienced a substantial failure in our Hosted Exchange environment. We proactively shut down the environment to prevent any further problems while we continue work to bring back service. As we continue to overcome the root cause of the issue, we have an alternate service that will re-activate your ability to send and receive emails.
At no cost to you, we will be providing you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 up until more notice.”
Rackspace Hosted Exchange Security Occurrence
It was not till nearly 24 hours later on at 1:57 AM on December 3rd that Rackspace officially revealed that their hosted Exchange service was experiencing a security incident.
The statement further revealed that the Rackspace service technicians had actually powered down and detached the Exchange environment.
“After more analysis, we have identified that this is a security incident.
The known effect is separated to a portion of our Hosted Exchange platform. We are taking essential actions to examine and protect our environments.”
Twelve hours later on that afternoon they updated the status page with more information that their security team and outdoors experts were still working on fixing the outage.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not released details of the security event.
A security occasion normally includes a vulnerability and there are two serious vulnerabilities presently in the wile that were patched in November 2022.
These are the two most current vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack permits a hacker to read and change data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assaulter is able to run malicious code on a server.
An advisory released in October 2022 described the effect of the vulnerabilities:
“An authenticated remote assailant can carry out SSRF attacks to escalate opportunities and execute arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the assailant can possibly access to other resources by means of lateral movement into Exchange and Active Directory site environments.”
The Rackspace failure updates have actually not indicated what the particular problem was, only that it was a security event.
The most existing status update since December 4th stated that the service is still down and consumers are encouraged to move to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make development in resolving the occurrence. The accessibility of your service and security of your data is of high significance.
We have actually devoted extensive internal resources and engaged world-class external expertise in our efforts to reduce unfavorable impacts to consumers.”
It’s possible that the above noted vulnerabilities are related to the security incident impacting the Rackspace Hosted Exchange service.
There has been no statement of whether consumer information has been jeopardized. This occasion is still ongoing.
Included image by Best SMM Panel/Orn Rin