Jetty: insufficient threads configured for SelectChannelConnector


The Problem
Today when I run our Solr application in one machine, during start, it reports warning:
Oct 6, 2014 7:25:15 PM org.eclipse.jetty.server.AbstractConnector doStart
WARNING: insufficient threads configured for SelectChannelConnector@0.0.0.0:12345

Trying http request in browser, no response, just hang forever.

Inspect the solr server in Visual VM. In threads tab, it shows there is 238 live threads, and a lot of selector(128) and acceptors(72). This looks very suspiciours:
qtp1287645725-145 Selector127
   java.lang.Thread.State: BLOCKED
   java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked (a sun.nio.ch.Util$2)

"qtp1287645725-217 Acceptor71 SelectChannelConnector@0.0.0.0:12345"
   java.lang.Thread.State: BLOCKED
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:134)
- waiting to lock (a java.lang.Object) owned by "qtp1287645725-221 Acceptor71 SelectChannelConnector@0.0.0.0:12345" t@221
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)


Then check the code: When start jetty, the code sets acceptors to number of cpu cores * 2. in this machine,  There is 64 cores. This will cause jetty to start 64*2 = 128 selectors and acceptors. 
connector.setAcceptors(2 * Runtime.getRuntime().availableProcessors());

The default acceptors is:(Runtime.getRuntime().availableProcessors()+3)/4 which is 16 in this case. 
setAcceptors(Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4));

So to fix this issue,  I just comment or cusom acceptors code: connector.setAcceptors(2 * Runtime.getRuntime().availableProcessors()); 

Lesson Learned
Be careful when tune server performance, make sure you truly understand its meaning.

Configure ThreadPool in jetty.xml
<Set name="ThreadPool">
  <New class="org.eclipse.jetty.util.thread.QueuedThreadPool">
    <Set name="minThreads">10</Set>
    <Set name="maxThreads">200</Set>
    <Set name="detailedDump">false</Set>
  </New>
</Set>
Jetty Code:
During startup, jetty will start acceptors(2=128 in this case) selectors:
org.eclipse.jetty.server.nio.SelectChannelConnector.doStart()
protected void doStart() throws Exception
{
    _manager.setSelectSets(getAcceptors());
    super.doStart();
}
org.eclipse.jetty.io.nio.SelectorManager.doStart()
protected void doStart() throws Exception
{
    _selectSet = new SelectSet[_selectSets];
    for (int i=0;i<_selectset .length="" i="" p="">        _selectSet[i]= new SelectSet(i);

    super.doStart();

    // start a thread to Select
    for (int i=0;i    {
        final int id=i;
        boolean selecting=dispatch(new Runnable()
        {
            public void run()
            {
            // ....
            }

        });
    }
}

Labels

adsense (5) Algorithm (69) Algorithm Series (35) Android (7) ANT (6) bat (8) Big Data (7) Blogger (14) Bugs (6) Cache (5) Chrome (19) Code Example (29) Code Quality (7) Coding Skills (5) Database (7) Debug (16) Design (5) Dev Tips (63) Eclipse (32) Git (5) Google (33) Guava (7) How to (9) Http Client (8) IDE (7) Interview (88) J2EE (13) J2SE (49) Java (186) JavaScript (27) JSON (7) Learning code (9) Lesson Learned (6) Linux (26) Lucene-Solr (112) Mac (10) Maven (8) Network (9) Nutch2 (18) Performance (9) PowerShell (11) Problem Solving (11) Programmer Skills (6) regex (5) Scala (6) Security (9) Soft Skills (38) Spring (22) System Design (11) Testing (7) Text Mining (14) Tips (17) Tools (24) Troubleshooting (29) UIMA (9) Web Development (19) Windows (21) xml (5)