Date and Time: Friday (19th July 2024) 4:15 AM to 4:30 AM IST (11 PM Thursday 18th July UTC)
Duration: 15 mins
Issue: /v2/meetings API was returning 500 response
Root Cause: It was investigated that one of our read routes experienced a huge request spike and caused our read DB to choke. Since this was not our service critical route, status alerts did not go up. However, secondary alerts were in place, which alerted the team. However, the spike had died down, and the system had recovered before taking any measures.
Resolution: System auto recovered
Next Steps:
Our team is designing rate limits on non-critical critical paths to avoid such errors in the future.