остання проблема, в діагностиці якої я приймав участь - all the threads are waiting on connection pool. Everyone is tormenting the DB connection pool team to get them to make connection pool more efficient. Looking at the heap dump it became clear there are no connection at all, and it is not about efficiency of borrowing connections. What then? Digging through the logs we see login timeouts during connection establishment. Looking at the flight recordings for the said time, we see the connection getting closed, but we do not see long IO operation causing the timeout. What then?
Well...looking at the flight recordings, we see CPU spiked for minutes, and we see almost all of it is spent on DH key negotiation - during DB connection establishment. Why does it take minutes? Because it switched from compiled and optimised to interpreted mode. So the fix is in working out why it switched to interpreted mode all of a sudden.
I don't know how many production issues caused by NPEs and the absence of monads were fixed without me lending a hand. But I get a feeling that stuff just breaks.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
timeasmymeasure
no subject
Date: 2021-12-26 12:33 am (UTC)остання проблема, в діагностиці якої я приймав участь - all the threads are waiting on connection pool. Everyone is tormenting the DB connection pool team to get them to make connection pool more efficient. Looking at the heap dump it became clear there are no connection at all, and it is not about efficiency of borrowing connections. What then? Digging through the logs we see login timeouts during connection establishment. Looking at the flight recordings for the said time, we see the connection getting closed, but we do not see long IO operation causing the timeout. What then?
Well...looking at the flight recordings, we see CPU spiked for minutes, and we see almost all of it is spent on DH key negotiation - during DB connection establishment. Why does it take minutes? Because it switched from compiled and optimised to interpreted mode. So the fix is in working out why it switched to interpreted mode all of a sudden.
I don't know how many production issues caused by NPEs and the absence of monads were fixed without me lending a hand. But I get a feeling that stuff just breaks.