Embedded Quote Data Leak
Incident Report for Vanilla Forums
Postmortem

Overview

On the evening of Friday November 15th, Vanilla's development team was informed of a vulnerability within our embedded quote functionality. We immediately took steps to rectify the issue. Out of an abundance of caution and following industry best practice, we also reset all Vanilla Forums user passwords. This was then patched and deployed later that evening and further mitigation steps were taken throughout the weekend.

Thus far we have seen no evidence of this vulnerability having been exploited, however we are still doing further investigations.

The vulnerability and the steps taken to rectify the issue and mitigate data leaking are described below.

Some Background

Vanilla stores records of user information in its databases, for display and authentication purposes. The full record is generally not visible to the public and is guarded by various permission checks through Vanilla's controllers and API endpoints. In Vanilla's APIv2 endpoints additional validation exists to ensure that only certain, predefined fields are returned from any particular endpoint. This is called a Schema.

What happened

A bug in sanitization logic caused the schema not to be applied to a single APIv2 endpoint. For customers using our "Rich Text Editor" this caused full user records to outputted to the HTML source sent to browsers while quoting comments and discussions. While this data was not visible to the eye, it could be accessed by:

  • Inspecting the network requests while quoting some user content.
  • Inspecting the HTML of rich comment or discussion quotes.

In addition, for all Vanilla customers regardless of text editor in use:

  • Calling the /api/v2/media/scrape endpoint directly (with permission to view the scraped discussion or comment). We are currently evaluating logs to determine if any evidence of malicious action against this API is present, thus far no evidence is present in the log reviews performed to date.

Affected data includes entire user records made up of:

  • Usernames.
  • Passwords (salted & hashed)

    • Our hashing mechanism is BCRYPT with a cost of 10
    • Users connected over a SSO (Single Sign On) connection or through social connect don't have this data stored with Vanilla, and could not have had this field shared.
  • Emails

  • IP addresses

  • User preferences

  • Users roles and ranks

Timeline

Oct. 30th - Vulnerability in the code is introduced to master branch.
Oct. 31st to Nov. 13th - Vulnerable version, 2019.016 is rolled out to Vanilla Cloud.
Nov. 15th, 17:20 - Vanilla Developers informed of the vulnerability.
Nov. 15th, 17:27 - A fix has been tested, reviewed and merged.
Nov. 15th, 17:40 - A patched version of Vanilla begins to roll out across Vanilla's infrastructure.
Nov. 15th, 18:42 - A patched version of Vanilla is deployed to all clusters, which mitigates the immediate issue.
Nov. 15th, 18:50 - Vanilla begins to reset the password of all potentially affected users* and signs them out.
Nov. 16th, 09:30 - Vanilla Developers prepare a patch to filter cached user data out of authenticated API endpoint responses.
Nov. 16th, 11:00 - This patch begins to be deployed to all clusters.
Nov. 16th, 11:50 - The patch is deployed.
Nov. 16th, 12:00 - Vanilla publishes an incident on its status page, along with this RCA.

* As SSO users do not have their passwords stored in Vanilla, this information was not leakable. These users were not affected and no changes were made.

Findings

Prioritize additional unit tests against models over integration tests against API endpoints

  • The code that did not properly sanitize the output was only tested through integration tests with other API endpoints. These endpoints provided their own Schema to sanitize output and as a result the issue was not uncovered at testing time.

If you have any additional questions or concerns about any of this please don't hesitate to contact us.

Posted Nov 16, 2019 - 11:53 EST

Resolved
A bug in sanitization logic caused the schema not to be applied to and APIv2 endpoint. This caused full user records to outputted to browser while quoting comments and discussions.
Posted Nov 15, 2019 - 17:00 EST