When ChatGPT Becomes a Phishing Tool: The Story Behind the ChatGPhish Vulnerability
Overview of the campaign
For many years, phishing has primarily appeared as fake emails, SMS messages, or fraudulent websites. Organizations have invested significantly in Secure Email Gateway (SEG), Safe Links, URL Filtering, and awareness programs to mitigate the risks from these attack forms. However, a new study from Permiso Security has revealed a completely different attack approach: instead of deceiving users through email, attackers can exploit ChatGPT itself to display phishing content directly within the interface users trust. This technique is called ChatGPhish.
Notably, ChatGPT is neither compromised, nor is malware executed, nor is control taken over. Instead, attackers exploit how ChatGPT processes content from external websites to turn the AI interface into a "trusted bridge" between the victim and the phishing infrastructure. This can be seen as one of the clearest examples of AI-Mediated Phishing, where AI becomes an intermediary link in the attack chain.
When website summarization becomes an attack surface
Nowadays, using ChatGPT for a range of tasks such as summarizing articles, analyzing README files on GitHub, reading technical documents, researching competitors, and translating foreign reports has become a daily activity for organizations and businesses.
Let's take a basic example with a common action like: "Summarize the content of this website." This prompts ChatGPT to retrieve, analyze, and present content from an external source to the user. The issue arises when the website's content contains directives specifically designed to manipulate how the AI responds. According to Permiso's research, the web version of ChatGPT's renderer overly trusts certain Markdown elements from external sources, particularly hyperlinks, images from external URLs, QR codes, or content formatted to resemble system alerts.
This facilitates phishing attacks appearing directly within the ChatGPT conversation window.
The true nature of ChatGPhish
ChatGPhish is not a traditional XSS vulnerability. Nor is it Remote Code Execution. Essentially, it is a clever combination of Indirect Prompt Injection, UI Redressing, and Trust Exploitation.
The attacker does not target the ChatGPT system itself but instead exploits ChatGPT to target users.
The danger lies in the psychological factor. If a link appears in an email, users are often suspicious. But if that link appears in a response from ChatGPT—a tool they use daily—their level of caution typically decreases significantly. This is what researchers refer to as Trust Transfer. Users' trust in ChatGPT is inadvertently transferred to the content the AI is displaying.
How does the attacker exploit it?
Prepare Payload
First, the attacker creates or edits a public webpage. This could be a blog, wiki, GitHub README, user guide, or FAQ page. They insert Markdown or content designed for the AI to interpret and display. The goal is to make ChatGPT treat them as a legitimate part of the source document.
Users request ChatGPT to summarize
Once the environment is successfully prepared, the attacker waits for users to use ChatGPT to summarize any content.
Example: Summarize this page: https://example.com/documentation ChatGPT accesses the website and processes the content. The malicious payload is then loaded into the context.
AI displays content controlled by the attacker.
The final result appears directly in the ChatGPT interface. Users see: security notifications, account verification links, images, or a QR code. They often don't realize that this content actually comes from the source website, not from OpenAI.
Attack scenario
Scenario 1: Tracking users with a Tracking Pixel
This is a simple but effective technique. Here, the attacker embeds an image stored on their server. When ChatGPT displays the content and the browser loads the image, an HTTP connection is sent to the attacker's server. Through this request, the attacker can collect: IP address, User-Agent, access time, browser identification, and some metadata related to the session. Although not enough to take over an account, this data is very useful for reconnaissance activities and building target profiles.
Scenario 2: Faking OpenAI System Alerts
This is a more dangerous method. The attacker creates content identical to system alerts: Security Warning Your ChatGPT session has expired. Please re-authenticate. Accompanying this message is a phishing link. From the user's perspective, the notification appears directly in the ChatGPT interface. There are no clear signs indicating that this content actually comes from an external website. If the victim clicks the link and enters their login information, the entire credential theft process is completed.
Scenario 3: Mobile Pivot via QR Code
This technique is considered the most dangerous in the study. Instead of displaying a direct link, the attacker embeds a QR code. Typically, when users are working on a computer and see a QR code in ChatGPT, they use their phone to scan it.
At this point, the attack shifts to the mobile device. This allows the attacker to bypass various protection mechanisms: URL Preview, Browser Reputation, Safe Browsing, Password Manager, DNS Filter, and even EDR on the computer, making it difficult for SOC experts to detect and record.
The entire defense chain deployed on the workstation is almost completely bypassed. This is why Mobile Pivot is becoming increasingly popular in modern phishing campaigns.
Does ChatGPhish have a CVE yet?
As of the study published by Permiso Security on 05/29/2026, there is no official information about this vulnerability being assigned a CVE. The reason is that it is not entirely a traditional software flaw. ChatGPhish falls into the categories of logic flaw, trust boundary issue, indirect prompt injection, and unsafe rendering design. Therefore, assigning a CVE and scoring it with CVSS is still under discussion. There is currently no official information from OpenAI or NVD about the CVSS score for this issue.
Why is this a concerning signal for AI security?
ChatGPhish shows that the attack surface is shifting. In the past, phishing simply involved sending emails to users. Now, attackers follow a different flow: Website -> LLM -> AI Interface -> User AI is becoming a new intermediary layer between the internet and humans. This means all data processed by AI can directly influence user behavior. Without mechanisms to separate data sources, mark external content, or control the rendering process, AI systems could inadvertently become tools for phishing and social engineering.
MITRE ATT&CK Mapping
T1566.002 – Spearphishing Link
T1204.001 – Malicious Link
Expert opinion
Although ChatGPhish has only been demonstrated in a research environment, from a practical operational perspective, it is a scenario that organizations in Vietnam should pay attention to. Over the past two years, AI platforms like ChatGPT have gradually become part of the daily workflow in many businesses. From technical teams, SOC operations, Threat Intelligence analysis, software development to business, marketing, or market research departments, using AI to summarize documents, analyze websites, read GitHub source code, or process foreign language reports has become common.
In particular, technical teams in Vietnam, such as SOC Analysts, Threat Hunters, Malware Researchers, Developers, or DevOps Engineers, may be at the highest risk. These individuals frequently use ChatGPT to analyze technical documents, read README files on GitHub, evaluate source code, or research IOCs from public sources. Their high frequency of interaction with external data increases their likelihood of encountering content designed to exploit AI display mechanisms.Một điểm đáng chú ý khác là kỹ thuật Mobile Pivot thông qua QR Code được đề cập trong nghiên cứu. Trong môi trường doanh nghiệp Việt Nam, việc sử dụng đồng thời máy tính làm việc và điện thoại cá nhân là rất phổ biến. Khi người dùng nhìn thấy mã QR xuất hiện trong phản hồi của ChatGPT, hành động quét mã thường diễn ra gần như theo phản xạ và ít bị nghi ngờ hơn so với việc nhấp trực tiếp vào một liên kết lạ.
Recommendations
Do not consider ChatGPT as a source for identity verification.
The most important principle is: ChatGPT can display content sourced from the Internet, but not everything that appears in the chat window is created or verified by OpenAI.
If notifications appear such as:
"Your session has expired"
"Security Alert"
"Re-authenticate your account"
"Verify your credentials"
"Account suspension warning"
Users should not take any immediate action but instead need to
Open a new browser tab
Visit the official website directly
Check the account status from an official source
Absolutely do not log in through links that appear in AI responses.
Do not click on links that appear in website summary content.
Before accessing:
Check the full domain name
Compare with the official website
Verify the URL by manually typing it
Do not click directly if the link requires login
Do not provide a password or MFA code
Do not scan QR codes that appear in AI responses.
If ChatGPT displays a QR Code:
Consider it unreliable data
Find the original URL using an independent source
Do not scan QR codes just because they appear in the ChatGPT window
Be cautious of unusual images or security alerts.
If an AI response appears:
Security banner
Image requesting authentication
"Login" button
"Continue" button
"Verify" button
Consider it a sign of irregularity. ChatGPT does not require users to log in again through content displayed within the model's response.
Do not enter credentials after being guided by AI.
- Simple rule: If AI directs you to a page requesting a password, stop and verify the source before proceeding.
Refer
ChatGPhish Vulnerability Turns ChatGPT Web Summaries Into a Phishing Surface
