Root Cause Analysis
Product: Harmony
Issue: Service disruption for login service, APIs on APIM, Agent communications
Impact: Major
Services Impacted:
- Cloud Login
- Rest API
- Operations
- APIs running on APIM
Location: NA (North America Cloud)
Problem Description
Login and App access: Users encountered an internal server error when attempting to log in to Identity, hindering access to various apps such as Cloud Studio, APIM, Marketplace, and the Management Console.
Cloud and Private Agents: Harmony Cloud and Private agents faced connection issues and were unable to log in, rendering them incapable of processing any operations.
API on APIM: The APIs experienced intermittent issues, ranging from degraded execution to complete unresponsiveness.
Duration
March 2, 2024 9:05 PM UTC - March 3, 2024 02:20 AM UTC
Root Cause
This high-traffic volume was confirmed to be the result of a distributed denial-of-service (DDoS) attack from a malicious third party targeting our AWS Cloud Service.
We detected unusually high traffic volume on our firewalls and load-balancers. The servers behind the firewall were unable to handle this overwhelming surge in traffic, resulting in disruptions to user logins, agents processing operations, and APIs on APIM.
Our Web Application Firewalls (WAF) failed to detect the unique fingerprint of the traffic and, consequently, were unable to block it.
Immediate Action
Once the elevated traffic was detected, the Jitterbit team, per security protocol, took countermeasures by blocking all traffic and safely restoring each service by thoroughly reviewing with our Information Security and AWS Security Team.
During this time, services to API, Login and Agent communications were disrupted. Once the fingerprint was derived, a rule was put in place to mitigate the attack.
We implemented rate-limit rules as an additional safety layer while collaborating with the AWS Security Team. Services were safely restored by thorough review by both the Jitterbit and AWS teams. No customer data was lost or at risk during the disruption.
Strategic Action
The Information Security team will run security scans and review the Intrusion Detection System (IDS) to confirm this was an isolated scenario.
Jitterbit Security Teams are continuing to work with the AWS Security team to further analyze and implement proactive safety measures to mitigate further disruptions.