The Problem
Today when I run our Solr application in one machine, during start, it reports warning:
Oct 6, 2014 7:25:15 PM org.eclipse.jetty.server.AbstractConnector doStart
WARNING: insufficient threads configured for SelectChannelConnector@0.0.0.0:12345
Trying http request in browser, no response, just hang forever.
Inspect the solr server in Visual VM. In threads tab, it shows there is 238 live threads, and a lot of selector(128) and acceptors(72). This looks very suspiciours:
qtp1287645725-145 Selector127
java.lang.Thread.State: BLOCKED
java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked (a sun.nio.ch.Util$2)
"qtp1287645725-217 Acceptor71 SelectChannelConnector@0.0.0.0:12345"
java.lang.Thread.State: BLOCKED
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:134)
- waiting to lock (a java.lang.Object) owned by "qtp1287645725-221 Acceptor71 SelectChannelConnector@0.0.0.0:12345" t@221
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
Then check the code: When start jetty, the code sets acceptors to number of cpu cores * 2. in this machine, There is 64 cores. This will cause jetty to start 64*2 = 128 selectors and acceptors.
connector.setAcceptors(2 * Runtime.getRuntime().availableProcessors());
The default acceptors is:(Runtime.getRuntime().availableProcessors()+3)/4 which is 16 in this case.
setAcceptors(Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4));
So to fix this issue, I just comment or cusom acceptors code: connector.setAcceptors(2 * Runtime.getRuntime().availableProcessors());
Lesson Learned
Be careful when tune server performance, make sure you truly understand its meaning.
Configure ThreadPool in jetty.xml
During startup, jetty will start acceptors(2=128 in this case) selectors:
org.eclipse.jetty.server.nio.SelectChannelConnector.doStart()
protected void doStart() throws Exception
{
_manager.setSelectSets(getAcceptors());
super.doStart();
}
org.eclipse.jetty.io.nio.SelectorManager.doStart()
protected void doStart() throws Exception
{
_selectSet = new SelectSet[_selectSets];
for (int i=0;i<_selectset .length="" i="" p=""> _selectSet[i]= new SelectSet(i);
super.doStart();
// start a thread to Select
for (int i=0;i {
final int id=i;
boolean selecting=dispatch(new Runnable()
{
public void run()
{
// ....
}
});
}
}
Today when I run our Solr application in one machine, during start, it reports warning:
Oct 6, 2014 7:25:15 PM org.eclipse.jetty.server.AbstractConnector doStart
WARNING: insufficient threads configured for SelectChannelConnector@0.0.0.0:12345
Trying http request in browser, no response, just hang forever.
Inspect the solr server in Visual VM. In threads tab, it shows there is 238 live threads, and a lot of selector(128) and acceptors(72). This looks very suspiciours:
qtp1287645725-145 Selector127
java.lang.Thread.State: BLOCKED
java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked (a sun.nio.ch.Util$2)
"qtp1287645725-217 Acceptor71 SelectChannelConnector@0.0.0.0:12345"
java.lang.Thread.State: BLOCKED
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:134)
- waiting to lock (a java.lang.Object) owned by "qtp1287645725-221 Acceptor71 SelectChannelConnector@0.0.0.0:12345" t@221
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
Then check the code: When start jetty, the code sets acceptors to number of cpu cores * 2. in this machine, There is 64 cores. This will cause jetty to start 64*2 = 128 selectors and acceptors.
connector.setAcceptors(2 * Runtime.getRuntime().availableProcessors());
The default acceptors is:(Runtime.getRuntime().availableProcessors()+3)/4 which is 16 in this case.
setAcceptors(Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4));
So to fix this issue, I just comment or cusom acceptors code: connector.setAcceptors(2 * Runtime.getRuntime().availableProcessors());
Lesson Learned
Be careful when tune server performance, make sure you truly understand its meaning.
Configure ThreadPool in jetty.xml
<Set name="ThreadPool"> <New class="org.eclipse.jetty.util.thread.QueuedThreadPool"> <Set name="minThreads">10</Set> <Set name="maxThreads">200</Set> <Set name="detailedDump">false</Set> </New> </Set>Jetty Code:
During startup, jetty will start acceptors(2=128 in this case) selectors:
org.eclipse.jetty.server.nio.SelectChannelConnector.doStart()
protected void doStart() throws Exception
{
_manager.setSelectSets(getAcceptors());
super.doStart();
}
org.eclipse.jetty.io.nio.SelectorManager.doStart()
protected void doStart() throws Exception
{
_selectSet = new SelectSet[_selectSets];
for (int i=0;i<_selectset .length="" i="" p=""> _selectSet[i]= new SelectSet(i);
super.doStart();
// start a thread to Select
for (int i=0;i
final int id=i;
boolean selecting=dispatch(new Runnable()
{
public void run()
{
// ....
}
});
}
}