payments service degradation

Incident Report for Banked

Postmortem

On October 30th, 2023 between 11:18 -11:43 UTC, Banked experienced a partial outage in the payments API and checkout product in the UK/EU region.

During the outage, about half of the attempts to create payments and checkout failed.

This issue occurred due to a failure in a routine upgrade of a component in our infrastructure.

Our on-call team responded immediately to the failure and stopped the upgrade to investigate with our cloud service provider.

We are currently working with our cloud service provider to prevent similar issues from occurring in the future.

Posted Nov 01, 2023 - 19:20 UTC

Resolved

This incident has been resolved.

Further details will be provided in the postmortem in due course.

Posted Oct 30, 2023 - 16:04 UTC

Update

We are still continuing to monitor the results following the earlier fix.

Posted Oct 30, 2023 - 14:43 UTC

Update

We are still continuing to monitor the results following the earlier fix.

Posted Oct 30, 2023 - 14:07 UTC

Update

We are still continuing to monitor the results following the earlier fix.

Posted Oct 30, 2023 - 13:14 UTC

Update

We are still continuing to monitor the results following the earlier fix.

Posted Oct 30, 2023 - 12:39 UTC

Update

We are continuing to monitor for any further issues.

Posted Oct 30, 2023 - 12:04 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Oct 30, 2023 - 12:01 UTC

Update

We are continuing to work on a fix for this issue.

Posted Oct 30, 2023 - 11:58 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Oct 30, 2023 - 11:47 UTC

Update

We are continuing to investigate this issue.

Posted Oct 30, 2023 - 11:45 UTC

Investigating

We are currently experiencing a performance degradation to the payments and checkout service.

Investigations are ongoing and an update will be provided as soon as more information becomes available.

Posted Oct 30, 2023 - 11:28 UTC

This incident affected: EU Payments (API, Checkout).