Samepage Messages Incident & Response
Dear Samepage users,
On February 14, we had a serious, though a brief, incident in which open team chat message content was sent to all guests in your organization via email and/or push notifications who should not have received those notifications.
This was due to a bug that we pushed into production a short time ago.
I want to assure you that we were NOT hacked: this was a bug on our part. No data was lost. And those messages never allowed access to those teams. Unfortunately, the message content was exposed to guests who did not have access rights to see those messages.
On behalf of the entire team here at Samepage, I want to apologize for this serious error. We are committed to learning from this failure and improving our systems, processes, and procedures.
The full technical explanation is below.
At 8:04:47 UTC, we deployed a new version of the server service responsible for processing email as well as push notifications.
Soon after, we started receiving complaints from some customers that they are getting notifications of messages not intended for them.
At around 8:20 UTC, we identified the issue and immediately started working on a fix.
At 8:30:48 UTC, the server service was fixed, and all pending notifications, i.e., those that hadn't been delivered yet, were canceled.
Unfortunately, we had no control over the push and email notifications that had already been sent.
The affected service is responsible for deciding whom to notify about each chat message posted to Samepage.
Normally only members of the team get notified. In the case of open teams, all members of the organization may get notified (depending on their notification preferences).
The coding error caused that not only organization members but also all guests were considered. Those guests should normally not receive any notifications from open teams where they are not added as members. Thus during the period, all organization guests got notified on their mobile devices as well as email. Those notifications (email and push) did include the texts of the messages, but at no time were the guests able to access the actual team content.
The problem occurred in open teams only. Private and protected teams were not affected.
If no one in your tenant posted any messages to an open team during the period of this incident, you were NOT affected.
The error made it to the server even though we have a strict code review policy and a wide range of automated tests. Both the engineer and the reviewer failed to spot the issue. At the same time, the test suites weren't complete enough to cover this particular scenario.
Only organizations where someone did post a chat message in an open team were affected. Guests and members of other organizations were not affected.
What Happened in numbers
Thankfully, we are mainly releasing in night PST hours, so the main portion of our customers was only affected by notifications from our tenant.
During the time period (26 minutes):
- total organizations affected - 66
- including our own organization, which unfortunately has thousands of guests
- total chats where messages were posted - 185
- total notifications dispatched - 170,451
- this number includes both correct and incorrect notifications
Lessons Learned & Actions Taken
Our release process is very flexible and agile. We can and do release changes multiple times a day. We rely on our automated tests framework as well as the code review and pair programming to avoid errors like this.
Considering this was our first serious issue in many years now, one could argue the process is strong and secure. Apparently, we have become a bit too overconfident in our trust in the process.
In our drive for releasing as many new features and improvements as possible, as fast as possible, we became less focused on the tests. This was the cost.
We re-commit to the following goals:
- Spend more time on writing new tests.
- Review all the code paths, especially in the security-related code, are always covered by the automated tests.
- Don't push new code unless it is fully covered.
Thank you for the love
Let's enjoy the rest of Valentine's day peacefully. We are not going to release anything new today :) We will take the time to review and learn from this experience.
Thank you so much for reporting to us shortly after the problem began. We are really thrilled to see such a great community of customers and friends that believe in our vision building a world where everything is on the same page.