API and checkout service degradation

Incident Report for Banked

Postmortem

On June 14th, 2023 between 17:54-18:55 UTC, Banked experienced an outage in the payments API and checkout product.

During the outage, most attempts to create payments failed.

This issue occurred during a patch upgrade to our infrastructure that caused a failure in our streaming component.

Our on-call team responded immediately to the failure and started a recovery process to return the system to healthy state.

Moving forward, we are working to improve the resilience of our components and their self-healing abilities.

Posted Jun 20, 2023 - 18:59 UTC

Resolved

The issue affecting the API and checkout service has been fully resolved.

We have confirmed, following a period of monitoring, that the service has returned to normal processing and no further degradation has been observed. Further details will be provided in the postmortem in due course.

Posted Jun 14, 2023 - 19:10 UTC

Update

A fix has ben deployed and the intermittent failures have stopped. We will continue to monitor the situation and update once the situation has resolved.

Posted Jun 14, 2023 - 19:01 UTC

Update

We're continuing to investigate intermittent issues with payment creation and completion the team is investigating and deploying a fix. We will update next by 20:00.

Posted Jun 14, 2023 - 18:52 UTC

Update

We're continuing to investigate intermittent issues with payment creation and completion the team is investigating and deploying a fix. We will update next by 19:45.

Posted Jun 14, 2023 - 18:36 UTC

Monitoring

A fix has been implemented and we are starting to accept payments again and payments are completing. The Incident response team are continuing to monitor the situation.

Posted Jun 14, 2023 - 18:22 UTC

Identified

We have isolated the issue on the API and checkout service; which is calls to create payments to fail and checkouts to complete. Our Incident Response team is actively working to resolve this as high priority.

We will post an update at 19:30 UTC unless there is further information to share before this time.

Posted Jun 14, 2023 - 18:11 UTC

Investigating

We are currently experiencing a performance degradation to the API and hosted checkout service.

Investigations are ongoing and an update will be provided as soon as more information becomes available.

Posted Jun 14, 2023 - 18:02 UTC

This incident affected: Global (API, Checkout).