Desktop app? Yes please!

The Samepage Desktop App for makes it even easier to hop into Samepage. One click and you're in. Plus, it'll help you synchronize your team files to your computer for easy updating and offline access.

Samepage iOS App

Sign your team up for free with our iOS app!

Samepage Android App

Sign your team up for free with our Android app!

Samepage Messages Incident & Response

February 14, 2020

Samepage Messages Incident & Response

Dear Samepage users,

On February 14, we had a serious, though a brief, incident in which open team chat message content was sent to all guests in your organization via email and/or push notifications who should not have received those notifications.

This was due to a bug that we pushed into production a short time ago.

I want to assure you that we were NOT hacked: this was a bug on our part. No data was lost. And those messages never allowed access to those teams. Unfortunately, the message content was exposed to guests who did not have access rights to see those messages.

On behalf of the entire team here at Samepage, I want to apologize for this serious error. We are committed to learning from this failure and improving our systems, processes, and procedures.

The full technical explanation is below.

Sincerely,

Scott

Timeline

At 8:04:47 UTC, we deployed a new version of the server service responsible for processing email as well as push notifications.

Soon after, we started receiving complaints from some customers that they are getting notifications of messages not intended for them.

At around 8:20 UTC, we identified the issue and immediately started working on a fix.

At 8:30:48 UTC, the server service was fixed, and all pending notifications, i.e., those that hadn't been delivered yet, were canceled.

Unfortunately, we had no control over the push and email notifications that had already been sent.

What Happened

The affected service is responsible for deciding whom to notify about each chat message posted to Samepage.

Normally only members of the team get notified. In the case of open teams, all members of the organization may get notified (depending on their notification preferences).

The coding error caused that not only organization members but also all guests were considered. Those guests should normally not receive any notifications from open teams where they are not added as members. Thus during the period, all organization guests got notified on their mobile devices as well as email. Those notifications (email and push) did include the texts of the messages, but at no time were the guests able to access the actual team content.

The problem occurred in open teams only. Private and protected teams were not affected.

If no one in your tenant posted any messages to an open team during the period of this incident, you were NOT affected.

The error made it to the server even though we have a strict code review policy and a wide range of automated tests. Both the engineer and the reviewer failed to spot the issue. At the same time, the test suites weren't complete enough to cover this particular scenario.

Only organizations where someone did post a chat message in an open team were affected. Guests and members of other organizations were not affected.

What Happened in numbers

Thankfully, we are mainly releasing in night PST hours, so the main portion of our customers was only affected by notifications from our tenant.

During the time period (26 minutes):

  • total organizations affected - 66
    • including our own organization, which unfortunately has thousands of guests
  • total chats where messages were posted - 185
  • total notifications dispatched - 170,451
    • this number includes both correct and incorrect notifications

Lessons Learned & Actions Taken

Our release process is very flexible and agile. We can and do release changes multiple times a day. We rely on our automated tests framework as well as the code review and pair programming to avoid errors like this.

Considering this was our first serious issue in many years now, one could argue the process is strong and secure. Apparently, we have become a bit too overconfident in our trust in the process.

In our drive for releasing as many new features and improvements as possible, as fast as possible, we became less focused on the tests. This was the cost.

We re-commit to the following goals:

  • Spend more time on writing new tests.
  • Review all the code paths, especially in the security-related code, are always covered by the automated tests.
  • Don't push new code unless it is fully covered.

Thank you for the love

Let's enjoy the rest of Valentine's day peacefully. We are not going to release anything new today :) We will take the time to review and learn from this experience.

Thank you so much for reporting to us shortly after the problem began. We are really thrilled to see such a great community of customers and friends that believe in our vision building a world where everything is on the same page.