Call the police one day ： A service deployed by a machine suddenly cannot be accessed . Remember the first reaction, log in to the machine and view the log , Because the service hung up , Probably because OOM. At this time, the following information is found in the machine log ：
nio handle failed java.lang.OutOfMemoryError: Direct buffer memory at
org.eclipse.jetty.io.nio.xxxx at org.eclipse.jetty.io.nio.xxxx at
org.eclipse.jetty.io.nio.xxxx
Indicates that it is indeed OOM, Which district caused the problem ? It can be seen that ：Direct buffer memory
, And see a lot jetty Related method call stack , Just these logs , Can analyze OOM reason .
<>Direct buffer memory
Out of heap memory ,JVM A piece of memory outside the heap , Not by JVM Administration , but Java Code can be in JVM Some memory space is used outside the heap . These spaces are Direct buffer
memory, Direct memory , This memory consists of os Direct management . But it's strange to call it direct memory , I don't like to call it “ Out of heap memory ”.
Jetty As JVM The process runs the process of the system we have written ：
this time OOM yes Jetty Caused when using off heap memory . Can be calculated ,Jetty Off heap memory may be in constant use , Then out of heap memory space is insufficient , Unable to use more off heap memory , Just OOM Yes .
Jetty Non stop use of off heap memory ：
<> solve OOM Underlying technology of
Jetty Since it is used Java Written , How did he pass Java Code to request out of heap memory ? Then, how can the out of heap memory space be released ? This involves Java of NIO bottom .
JVM The performance optimization of is relatively easy , But if it's solved OOM, Except for some mentally retarded and simple , If someone keeps creating objects in the code . Many other products OOM problem , There are some technical difficulties , Need solid technology .
<> How is out of heap memory applied , How is it released ?
If in Java A block of off heap memory space should be applied for in the code , Is to use DirectByteBuffer This class , You can build one through this class DirectByteBuffer Object of , The object itself is JVM In heap memory .
But while you're building this object , A memory space will be set aside in the off heap memory to associate with this object , Let's look at the picture below , You know their relationship very well .
Therefore, when allocating off heap memory , That's the basic idea .
<> How to free off heap memory
When you DirectByteBuffer Object is not referenced , After garbage , At some point YGC or Full GC Recycled when .
Just recycle one DirectByteBuffer object , It frees its associated off heap memory ：
<> Then why is there an out of heap memory overflow ?
If you create many DirectByteBuffer object , Takes up a large amount of off heap memory , Then these DirectByteBuffer Object not yet GC Thread to recycle , Then it won't be released !
When out of heap memory is heavily DirectByteBuffer Object association usage , If you need to use additional off heap memory , Report memory overflow ! When will there be a large number of DirectByteBuffer The object is always alive , As a result, a large amount of off heap memory cannot be released ?
It is also possible that the system is highly concurrent , Create too many DirectByteBuffer, Use a lot of off heap memory , Continue to use off heap memory at this time , Will OOM! But this is clearly not the case with the system .
<> The real reason for out of heap memory overflow
Can use jstat Observe the online system operation , At the same time, check the processing time of some requests according to the log , Analyze the past gc journal , After taking a look at the time-consuming call of each interface of the system , The analysis idea is as follows .
First, look at the time-consuming interface call , System concurrency is not high , But it takes more time to process each request , Average per request 1s.
then jstat find , Continuously called with the system , Various objects are always created , include Jetty It keeps creating itself DirectByteBuffer Object to apply for off heap memory space , Then until Eden full , Will trigger YGC：
But often in progress GC For a moment , Maybe some requests haven't been processed yet , There are many at this time DirectByteBuffer The object is alive , It hasn't been recycled yet , Of course, many before DirectByteBuffer The request corresponding to the object may have been processed , They can be recycled .
There must be some at this time DirectByteBuffer Objects and some other objects are alive , You need to transfer in Survivor area . Remember when the system went online , Memory allocation is extremely unreasonable , Gave the young generation one or two hundred M, The old generation gave seven or eight hundred M, Leading to Survivor only 10M. So often in YGC after , Some surviving objects （ Including some DirectByteBuffer） Will exceed 10M, Can't put it in Survivor, Direct entry Old：
So the process is repeated , Lead to some DirectByteBuffer Objects enter slowly Old,Old of DirectByteBuffer
More and more objects , And these DirectByteBuffer Are associated with a lot of off heap memory ：
In these older generations DirectByteBuffer In fact, many are recyclable , But because the older generation hasn't been full , So it didn't trigger full
gc, It will naturally not recycle these from the elderly generation DirectByteBuffer Yes ! Of course, these are not recycled in the elderly generation DirectByteBuffer The association has occupied a large amount of off heap memory space !
Until the end , When you want to continue using off heap memory , All out of heap memory is heavily used by older generations DirectByteBuffer It's occupied , Although they can be recycled , But helpless, because it has not triggered the old age full
gc, Therefore, out of heap memory can never be recycled . Finally lead OOM!
<> this Java NIO Why does it look so sand sculpture ?
Java NIO Didn't you think this would happen ?
Considered ! He knows a lot DirectByteBuffer The object may not be used , But it was not triggered gc As a result, they always occupy off heap memory .Java
NIO The following treatment has been done , Every time a new out of heap memory is allocated , All call System.gc(), remind JVM Proactively perform the following GC, To recycle some garbage that no one cites DirectByteBuffer object , Free out of heap memory space .
As long as it can trigger GC To recycle something that no one quoted DirectByteBuffer, Some off heap memory will be released , Naturally, more objects can be allocated to off heap memory . But because we're here again JVM Set ：
-XX:+DisableExplicitGC
Cause this System.gc() Not effective , As a result OOM.
<> Ultimate optimization
The project has the following problems ：
* Unreasonable memory setting , cause DirectByteBuffer Objects have been slowly entering the elderly generation , Out of heap memory has been unable to be freed
* Set -XX:+DisableExplicitGC, cause Java
NIO Some garbage can't be recycled actively DIrectByteBuffer object , It also leads to the failure to free off heap memory
This should be ：
* Allocate memory reasonably , Give the younger generation more memory , Give Way Survivor There is more space in the area
* let go -XX:+DisableExplicitGC This restriction , Give Way System.gc() take effect
After optimization ,DirectByteBuffer Generally, they will not continue to enter the elderly generation . As long as he stays in the younger generation , along with young gc The memory outside the heap will be recycled normally .
Just let go -XX:+DisableExplicitGC limit ,Java
NIO Out of heap memory was found to be low , It will pass naturally System.gc() remind JVM Active garbage collection , Recycle some DirectByteBuffer, This frees up off heap memory .
Technology