On July 20, 2024, from 12:35 AM to 1:30 AM IST, we experienced an incident that affected our call services. This resulted in a 55-minute service disruption for some of our customers.

Incident Summary:

• A configuration change in our production Kubernetes cluster caused certain critical pods to terminate unexpectedly.
• New pods were unable to start due to authentication issues with the cluster.
• This led to a temporary unavailability of our call functionality for both ongoing and new calls.

Root Cause:

The incident was caused by an unintended removal of necessary permissions in our cluster configuration during a routine maintenance operation.

Resolution:

Our team quickly identified the issue and restored the required permissions, allowing the affected services to recover fully.

Actions Taken to Prevent Recurrence:

• We are implementing stricter review processes for configuration changes.
• We're enhancing our monitoring systems to detect similar issues more quickly in the future.
• We're updating our documentation and training to prevent similar oversights.

We sincerely apologize for any inconvenience this may have caused. We appreciate your patience and understanding as we worked to resolve this issue. Our team remains committed to providing you with the highest level of service reliability.

If you have any questions or concerns, please don't hesitate to contact our support team.

Thank you for your continued trust in our services.