Responding to CitrixBleed (CVE-2023-4966): Key Takeaways from Affected Companies

The Cybersecurity and Infrastructure Security Agency (CISA) recently sounded the alarm on the widespread exploitation of the Citrix Bleed vulnerability. This critical security flaw has had a significant impact across various industries in the United States, including credit unions and healthcare services, marking it as one of the most critical vulnerabilities of 2023. Its relatively straightforward buffer overflow exploitability has raised major concerns. 

In response to the escalating threat, the US Health Department has urgently called upon hospitals to promptly mitigate the Citrix Bleed vulnerability. This move underscores the broad spectrum of sectors at risk. 

Citrix Bleed Background

Since its discovery, numerous notorious and well-established criminal groups, such as Lockbit and AlphV / BlackCat, have been exploiting the vulnerability known as CVE-2023-4966 , or "Citrix Bleed". This vulnerability primarily affects Citrix NetScaler ADC (Application Delivery Controllers) and NetScaler Gateway products. Citrix released security updates on October 10, 2023. 

While extensive information on how to exploit CitrixBleed is available (including descriptive blog posts and proof-of-concept scripts on GitHub), there's been limited sharing of the subsequent steps attackers are taking post-exploitation. 

Morphisec’s Incident Response (IR) team had the unique opportunity to analyze several such incidents, assisting customers in swiftly containing these breaches. In this advisory, we aim to illuminate the tactics and techniques used in selected IR activities dealing with post-exploit CitrixBleed incidents.  
 
This post provides a structured walkthrough of the incidents, providing for each phase key insights and recommending best practices for managing exposure to such threats.   

Exclusive: Join Morphisec for a FLASH Citrix Bleed webinar to learn directly from our security team about this Citrix Bleed analysis.

CitrixBleed Attacks phases from external reconnaissance, Citrix NetScaler vulnerability exploitation to data exfiltration – the threat actor used multiple IP addresses from different countries in order to evade detection 

External Reconnaissance for vulnerable NetScaler ADC  

In the incident we're examining, as well as in numerous others, hackers commonly initiate their attack by scanning the external networks of their target organizations. They specifically look for devices vulnerable to the CitrixBleed exploit. These scans are often traced back to IP addresses originating from Russia, with a notable pattern of 10-20 minutes between each scan request. 

The accompanying graph illustrates the range of IP addresses involved in the reconnaissance phase targeting the CitrixBleed vulnerability. It also highlights a specific IP address that we've identified in one of the incidents. This data underscores the systematic approach employed by attackers in identifying potential targets. 

An effective measure in managing the exposure to such threats is geo-blocking. By blocking access from high-risk geographic regions, organizations can significantly lower the risk of attacks like CitrixBleed.  

CVE-2023-4966 | CitrixBleed Attacker Ips IDed by GreyNoise 

Exploitation of CitrixBleed 

Following the initial reconnaissance phase, the attackers moved to the next stage of their operation. Just two days later, over the weekend, they shifted their strategy for the actual network penetration. Notably, they changed their IP address for this phase, switching from the Russian IP used during reconnaissance to a French IP for the initial entry. This practice of using different IPs for reconnaissance and entry is a deliberate tactic to evade detection. 

The exploitation process involved using a buffer overflow vulnerability to extract active session IDs from the system's memory. The attackers pinpointed a session that was active outside of normal working hours, belonging to an external service provider, which served their stealth approach well. 

This incident underscores the importance of having strong monitoring processes for network access, particularly for activities occurring outside standard working hours. Such measures are vital for the early detection and prevention of unauthorized access. 

It also illustrates the complexity of these attacks, often involving collaboration between multiple cybercriminal groups and strategic timing, such as choosing weekends or periods when slower organizational responses are expected. 

Internal Network Reconnaissance 

Once inside the network via a Citrix-accessible virtual machine (VM), the attackers swiftly moved to escalate their privileges. This task was made easier due to the enabled local administrator account on the VM, facilitating straightforward privilege escalation. 

With knowledge of the local administrator's enabled status, they sought out other servers with similar configurations. They used 'Find-LocalAdminAccess' from PowerSploit for this purpose.  

(PowerSploit: an open source, offensive security framework comprised of PowerShell modules to assist with multiple attack phases) 

During their lateral movement and after compromising additional service accounts, they employed SharpHound to gather data for BloodHound, enhancing their ability to navigate and exploit the network. 

(SharpHound/BloodHound: a tool used to analyze AD [Active Directory] rights and relations, focusing on ones for potential abuse) 

Evidence indicated that BloodHound activity was detected by the installed EDR (a market-leading vendor). However, this did not deter the attacker. 

To test remote command execution capabilities, the attacker tried to execute ‘ping 8.8.8.8’ on accessible devices. The first attempt involved using PowerShell's ‘Invoke-Command‘ with ‘Test-NetConnection’.

When this method proved ineffective, the attacker switched to using 'Test-WSMan'.

At some point the attacker tried to register Mimikatz as a windows security package, this failed and the EDR quarantined the dll. 

(Mimikatz is a program for extracting credentials from Windows memory)  

To reduce the exposure to this threat, immediate recommendations include: 

• Disabling local administrator accounts or employing LAPS (Local Administrator Password Solution) to manage local account passwords. 
• Disabling WinRM (Windows Remote Management) functionality. While WinRM can be used for IT management, it's important to weigh the risks and consider whether it needs to be enabled across the network. 

Lateral Movement 

Upon identifying servers accessible to the local administrator, the attacker initiated lateral movement using PowerShell commands: ‘Enter-PSSession’ and ‘new-PSSession’ for persistence. We observed instances of double nested sessions in this process. The attacker would connect from the initially compromised machine to a server via a remote PowerShell session, and then execute additional nested PowerShell sessions from there. Although the EDR vendor detected potential database services tampering on one server, this activity was carried out through a nested session originating from the initial VM. 

The attacker predominantly operated through the PowerShell ISE (Interactive Scripting Environment), using powershell_ise.exe. This approach allowed for the direct insertion of scripts without creating any files, complicating some aspects of PowerShell's basic security logging. 

At a certain point, the attackers also attempted to use the mstsc command within a PowerShell session for RDP (Remote Desktop Protocol) access. 

To reduce the attack surface, it is possible to disable PowerShell Remoting by utilizing Disable-PSRemoting. This action will prevent unauthorized users from initiating remote PowerShell sessions. 

Accessing Databases 

Typically, attackers like those in this scenario often prioritize targeting database servers, especially backup servers, as their first step post-lateral movement due to the lucrative nature of the data stored there. This data potentially can be leveraged as part of an extortion attempt. In this case, the attackers' initial objective was to extract database user passwords to gain access to the contents. They successfully accessed an older Veeam database containing usernames and passwords. This was achieved using PowerShell's Invoke-Sqlcmd to query and decrypt credentials stored on the local server instance, primarily through basic base64 decryption and the Unprotect command. 

To mitigate such risks, it's crucial to restrict the accessibility of these files within the network segment. Additionally, enhancing password security through salting can significantly reduce exposure. By implementing these measures, organizations can better protect sensitive database information from unauthorized access. 

Exfiltrating data 

While data exfiltration is often one of the simpler steps in a cyber-attack, the attackers in this scenario faced a challenge due to restricted outbound access within the target network. To successfully send the data to their FTP server (hosted on DigitalOcean with a German IP), they needed to identify a device with unrestricted outbound access. 

In their search for such a device, the attacker tried to remotely execute a scheduled PowerShell task. This task, located under Microsoft/Windows, was initiated by a PowerShell script and executed with System privileges. The attackers leveraged Microsoft OpenSSH for the connection to the FTP server, using a downloaded digital certificate for authentication (BatchMode=Yes). 

This situation highlights a critical point: while many organizations focus primarily on controlling inbound traffic, managing and monitoring outbound traffic is equally vital. It’s crucial to not only control outbound traffic but also to carefully track any exceptions to these restrictions. Such vigilance can prevent unauthorized data exfiltration and bolster overall network security. 

Citrix Bleed Summary 

The CitrixBleed incident underscores the importance of continuous cybersecurity vigilance: 

• Network Monitoring: Ongoing monitoring and timely system updates are crucial in preventing exploitations like CitrixBleed. 
• Access Control: The incident highlights the need for strict control and surveillance of privileged accounts to prevent unauthorized access. 
• Data Security: Securing sensitive information, especially on database servers, is vital. Employing password salting and network segmentation can significantly bolster defenses. 
• Traffic Management: Controlling both inbound and outbound traffic is essential in detecting and preventing unauthorized activities and data breaches. 
• Collaborative Security Strategies: The multifaceted nature of cyber threats necessitates industry-wide collaboration for effective cybersecurity measures. 

In addition to these practices, it's critical to continuously manage threat exposure and implement in-depth, multi-layered security defenses. By doing so, organizations can significantly reduce their risk profile and enhance their ability to respond effectively to evolving cyber threats. This incident serves as a reminder that cybersecurity is an ongoing process, requiring consistent evaluation and adaptation to new challenges. 

Note: Since this post represents a collection of real-life incidents, specific technical details including IOCs have been concealed.  Please contact Morphisec if you require detailed information.  

Want to learn more? Join our FLASH webinar to receive exclusive details about this Citrix Bleed attack analysis  

Or, Schedule a demo today and explore how Morphisec AMTD and on-demand incident response services can support your security operations.