API service degradation

Incident Report for Banked

Postmortem

On July 28th, 2023 between 9:38 -10:35 UTC, Banked experienced an outage in the payments API and checkout product.

During the outage, most attempts to create payments failed.

This issue occurred during a platform upgrade to our clusters that caused a failure in one of our core components.

Our on-call team responded immediately to the failure and started a recovery process to return the system to healthy state.

We have already begun and are continuing to work on improving our infrastructure components and their resilience.

Posted Jul 31, 2023 - 15:36 UTC

Resolved

This incident has been resolved.

Posted Jul 28, 2023 - 11:01 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jul 28, 2023 - 10:36 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jul 28, 2023 - 10:32 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Jul 28, 2023 - 10:03 UTC

Investigating

We are currently experiencing a performance degradation to the API and checkout service.

Investigations are ongoing and an update will be provided as soon as more information becomes available.

Posted Jul 28, 2023 - 09:44 UTC

This incident affected: Global (API, Checkout).