Cybersecurity lessons: Flaw in Zendesk file-upload feature

Editor’s note: This post is part of our series for cybersecurity professionals and hobbyists, written by Aaron E., head of cybersecurity at ExpressVPN. 

Late last year we received a bug report via our bug bounty program regarding a security issue with our support system, Zendesk.

Using a specific API call, an attacker could upload a malicious file to Zendesk and would receive in response a URL pointing to this file. Files uploaded in this manner would be accessible both to support agents and anyone the attacker shared the resulting URL with. 

ExpressVPN is a Zendesk customer, so we were particularly interested in the impact of this bug. The most troubling aspect for us was that the URL would be of the form expressvpn.zendesk.com/path/to/malicious/file, which may appear authoritative to users. 

Specifically, we were concerned about the scenario where an attacker creates a malicious version of our VPN application and then abuses the Zendesk flaw to publish it under the expressvpn.zendesk.com domain. Unsuspecting users could be convinced that such an application is legitimate due to its URL. 

There was also a real risk that an attacker would simply use our subdomain at Zendesk to host malware. Even though these files had a limited lifetime, an attacker would have been able to simply upload any payload they wanted (e.g., powershell files, shellcode, binaries, javascript files, etc.) and then use the highly reputable domain in drive-by download attacks, phishing attacks, or even to host second stages of their attack tooling and more.

ExpressVPN disclosed this vulnerability to Zendesk and also changed our support site and Zendesk configuration to mitigate the issue. Additionally, Zendesk rolled out a customer-controlled feature to require authentication for the requests and uploads API endpoints. In what follows, we’ll explain in more detail exactly how this bug was triggered and our remediation and disclosure timeline.

The bug

Background

ExpressVPN’s Support team currently fields tens of thousands of support requests per week. To handle this capacity, we need a large team and robust customer service software.

For the latter we use Zendesk, essentially a ticket management system with multiple different interfaces. Several of these interfaces are publicly facing. For instance, on our support page you can chat with one of our support agents or you can email us (support@expressvpn.com)—messages sent through these channels are actually funneled to our support agents via Zendesk.

We previously supported a third option, “File a Support Ticket,” which was non-interactive and allowed a user to contact us in a manner similar to email but without actually using their email account. This option included the ability for someone to attach files, e.g. screenshots or logs to help us troubleshoot their problem.

Vulnerability

In October 2020, a security researcher named anonymouse_360 contacted us and disclosed a bug with the subject “crafting harmful links in www.expressvpn.com/support that can deliver highly harmful files.” In their reproduction steps they described manually uploading a file via the “File a Support Ticket” Zendesk widget and intercepting the response via Burp Suite. An example response is below.

In this example response, the upload.attachment.content_url field contains a URL to download the previously attached file.

A more reliable and easier proof-of-concept is the cURL command:

The $subdomain is expressvpn and $filename is the name of the file (and above we use the value malicious_file). Here we pipe the output of the cURL command into jq to parse the JSON response, so we print only the URL to the uploaded file. The output of this command would be something like:

This the download URL for the attachment.

Note that this is unauthenticated; anyone can upload files and receive an expressvpn.zendesk.com URL from which to download them later. There’s also no obvious limitation when it comes to filetype or contents.

Interestingly, Zendesk documents a similar API interaction for authenticated users under https://developer.zendesk.com/rest_api/docs/support/attachments#upload-files

Impact

Any Zendesk customer that had the “Anybody can submit tickets” setting enabled was vulnerable to this attack. This setting was on by default, and our testing of a random sampling of domains we found via Google Dorking all had this setting enabled and were therefore vulnerable. An attacker could use a myriad of valid and trusted Zendesk subdomains to distribute malicious payloads of any type in a large variety of attack scenarios.  

Resolution

Once we received the bug report, we quickly validated it and created the simple proof of concept above. Since we identified it as a bug in Zendesk, a third party, we triaged it as out of scope for our bug bounty program. However, we rewarded the researcher regardless, as the issue potentially impacted us, and there were some steps we could take to mitigate the risk, detailed below.

Removing the “File a Support Ticket” widget

One of the first things we decided to do was to remove the “File a Support Ticket” widget from our support page. This widget was originally intended to cover cases in which our live chat support was unavailable for any reason. However, we had implemented Snapengage offline to cover the lack of online chat, rendering “File a Support Ticket” redundant and acceptable for removal. 

Although this was a positive change, as it removed some unnecessary code from our support page, it didn’t actually fix the issue, since the API was still accessible.

Removing the ability to upload files

We submitted a bug report to Zendesk on behalf of the security researcher and also inquired about immediate remediation steps. Zendesk product security responded with two options: reduce the lifetime of the URL validity or disable unauthenticated file uploads altogether. We chose to disable file uploads, and Zendesk implemented the change for us on their backend. At that time we validated that the issue was resolved for our account.

During a later attempt to validate if Zendesk had fixed the issue in May, we discovered that our subdomain was once again enabled for unauthenticated uploads. We notified Zendesk, which told us we needed to enable at least one or two more settings to make it effective—but those settings would require our users to log in to Zendesk to create tickets, which would have significantly changed our customer support process. We decided to wait for a real fix.

Removing the indirect object reference

Ultimately, the root cause for this bug was not that unauthenticated users can upload files but that unauthenticated users can download them. Resolving this issue doesn’t just protect our users but also anyone who might be tricked into downloading malicious files from any organization that has a Zendesk account with an authoritative-looking subdomain.

Fortunately, there was a setting available for customers to require users to authenticate before they can download attachments by selecting Enable Secure Downloads under the ticket attachment settings.  That setting would technically also solve the issue of serving anonymously uploaded files, but the change has a large impact on the way customers can use and interact with Zendesk, requiring them to create an account to see their tickets, for example, which creates additional barriers to a successful and frictionless customer service experience.   

Zendesk notified us in May that there is a mitigation in place, which is that all non-image attachments use the Content-Disposition: attachment header when serving files. ExpressVPN has verified that the mitigation negates cross-site scripting (XSS) vulnerabilities on the *.zendesk.com domain. However, this vector can still be used to upload publicly accessible malicious files on the *.zendesk.com domain. An example of how this can be exploited is by uploading a malicious javascript file, which can then be used as an injection for XSS into other domains.

Zendesk rolled out a configuration option to resolve the issue to our Zendesk instance on June 25, 2021, which was rolled out globally to their customers by July 30, 2021.  The fix requires customers to enable “Require authentication for requests and upload APIs,” which will require authentication for any API requests using the /api/v2/requests.json and /api/v2/uploads API endpoints. This was fairly easy for us to implement since we already deprecated the use of the Zendesk Web Widget Contact form and any integration we have is already authenticated. Once we enabled the feature, we retested the vulnerability and confirmed that we could no longer upload files anonymously.  

If you’re a Zendesk customer, it is important to understand that you will need to enable this feature manually, as it is not enabled by default.  You can follow the instructions on the Zendesk support page to implement this feature and disable this vulnerability.

Acknowledgements 

ExpressVPN would like to thank Zendesk for working with us on this issue and ultimately resolving it. We’d also like to thank anonymouse_360, the researcher who initially reported this issue through our Bug Bounty program on Bugcrowd. We take all reports seriously, and we were determined to get this upstream issue fixed not just for us but for everyone else who was impacted.

Timeline

Please note, due to the way the bug was being handled on HackerOne, ExpressVPN had an alternative communication channel via Zendesk’s support where we communicated with Zendesk.

• 2020-10-27: A security researcher, anonymouse_360, contacted ExpressVPN regarding the unauthenticated file upload and indirect object reference bug.
• 2020-10-28: The ExpressVPN security team investigated the report and determined that the bug resided in Zendesk and was thus out of scope for ExpressVPN.
• 2020-10-29: ExpressVPN submitted the bug to Zendesk via HackerOne (#1021298) on behalf of the security researcher.
• 2020-10-30: HackerOne incorrectly marked the issue as a duplicate. ExpressVPN reviewed the duplicate and notified HackerOne that the issues were not the same.
• 2020-11-02: ExpressVPN responded to the security researcher, anonymouse_360, and confirmed the bug but marked it as out of scope due to it being with a third party.
• 2020-11-05: ExpressVPN contacted the Zendesk CISO, who acknowledged that the issue was a valid security bug.
• 2020-11-09: Zendesk product security responded and offered to decrease the attachment token expiry time or to disable unauthenticated file uploads.
• 2020-11-10: ExpressVPN removed the “File a Support Ticket” widget from the support page.
• 2020-11-13: ExpressVPN responded to Zendesk and confirmed that we were ready to disable unauthenticated file uploads.
• 2020-11-14: ExpressVPN confirmed that unauthenticated file uploads were disabled and the proof of concept now fails.
• 2020-11-14: ExpressVPN rewarded the security researcher for bringing this bug to our attention despite it being out of scope.
• 2020-11-17: ExpressVPN recommended a 90-day disclosure window for the fix, with a disclosure date of January 27, 2021.
• 2020-11-18: A Zendesk Staff member, thepomerium, validated and reopened the HackerOne security bug.
• 2020-11-18: Zendesk responded to ExpressVPN’s recommendation, stating that it needed to check how long the fix would take and requested for a 120-day disclosure window instead. 
• 2020-11-19: ExpressVPN recommended sticking to a 90-day disclosure window as a target and stated that we were willing to be flexible.
• 2021-01-15: Zendesk responded and recommended March 4 as the disclosure date, based on 120 days from the opening of the support case.
• 2021-01-18: ExpressVPN recommended March 1, based on 120 days after the October 29 opening date in HackerOne.
• 2021-01-26: Zendesk notified ExpressVPN they planned to roll out a full fix on February 8, 2021.
• 2021-04-06: ExpressVPN validated that the bug was still in production and responded to the support request asking for an update.
• 2021-05-17: With no response from Zendesk since January 26 and the bug still present in production, ExpressVPN notified Zendesk of our intent to publish this report with a target date of May 24. 
• 2021-05-17: ExpressVPN identified that the unauthenticated file upload setting was re-enabled without our knowledge and requested additional information from Zendesk.
• 2021-05-19: ExpressVPN contacted the Zendesk CISO and requested an update, as we weren’t getting any from the support ticket. The Zendesk CISO acknowledged and stated he’d get an update from his side.
• 2021-05-20: Zendesk CISO replied and said he had met with his team and we’d get an update from them.
• 2021-05-20: Zendesk responded to us through the support ticket and provided us 3 options for disabling unauthenticated uploads on our account. They also promised an updated timeline on the release, which was delayed due to some technical issues.
• 2021-05-21: Zendesk set out specific TTL for our attachment expiry to three hours to limit the length and usability of these anonymous links.
• 2021-05-24: Zendesk notified ExpressVPN that they expected to ship a fix in June. We agreed on a weekly update cadence.
• 2021-06-03: Zendesk notified ExpressVPN that the global rollout was planned before the end of July.
• 2021-06-11: ExpressVPN shared this report with Zendesk for comment and edits.
• 2021-06-22: Zendesk asked ExpressVPN if we were willing to enable the new feature on our instance and test it. We agreed.
• 2021-06-25: Zendesk notified ExpressVPN that a new feature that fixes the issue was enabled on our account. We tested the fix and validated that it successfully mitigated the issue.
• 2021-06-25: Zendesk notified ExpressVPN they had started to roll the new feature that fixes the issue out to all customers, with a plan to have it fully rolled out by July 16, 2021.
• 2021-07-13: Zendesk notified ExpressVPN that they will not make the July 16 target, and requested an extension until July 30.
• 2021-07-28: Zendesk fixed the issue by rolling out a customer-controlled feature, not enabled by default, that when enabled will require authentication for the requests and uploads API endpoints.