Jetty: insufficient threads configured for SelectChannelConnector

The Problem
Today when I run our Solr application in one machine, during start, it reports warning:
Oct 6, 2014 7:25:15 PM org.eclipse.jetty.server.AbstractConnector doStart
WARNING: insufficient threads configured for SelectChannelConnector@0.0.0.0:12345

Trying http request in browser, no response, just hang forever.

Inspect the solr server in Visual VM. In threads tab, it shows there is 238 live threads, and a lot of selector(128) and acceptors(72). This looks very suspiciours:
qtp1287645725-145 Selector127
   java.lang.Thread.State: BLOCKED
   java.lang.Thread.State: RUNNABLE at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked (a sun.nio.ch.Util$2)

"qtp1287645725-217 Acceptor71 SelectChannelConnector@0.0.0.0:12345"
   java.lang.Thread.State: BLOCKED
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:134)
- waiting to lock (a java.lang.Object) owned by "qtp1287645725-221 Acceptor71 SelectChannelConnector@0.0.0.0:12345" t@221
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)


Then check the code: When start jetty, the code sets acceptors to number of cpu cores * 2. in this machine,  There is 64 cores. This will cause jetty to start 64*2 = 128 selectors and acceptors. 
connector.setAcceptors(2 * Runtime.getRuntime().availableProcessors());

The default acceptors is:(Runtime.getRuntime().availableProcessors()+3)/4 which is 16 in this case. 
setAcceptors(Math.max(1,(Runtime.getRuntime().availableProcessors()+3)/4));

So to fix this issue,  I just comment or cusom acceptors code: connector.setAcceptors(2 * Runtime.getRuntime().availableProcessors()); 

Lesson Learned
Be careful when tune server performance, make sure you truly understand its meaning.

Configure ThreadPool in jetty.xml
<Set name="ThreadPool">
  <New class="org.eclipse.jetty.util.thread.QueuedThreadPool">
    <Set name="minThreads">10</Set>
    <Set name="maxThreads">200</Set>
    <Set name="detailedDump">false</Set>
  </New>
</Set>
Jetty Code:
During startup, jetty will start acceptors(2=128 in this case) selectors:
org.eclipse.jetty.server.nio.SelectChannelConnector.doStart()
protected void doStart() throws Exception
{
    _manager.setSelectSets(getAcceptors());
    super.doStart();
}
org.eclipse.jetty.io.nio.SelectorManager.doStart()
protected void doStart() throws Exception
{
    _selectSet = new SelectSet[_selectSets];
    for (int i=0;i<_selectset .length="" i="" p="">        _selectSet[i]= new SelectSet(i);

    super.doStart();

    // start a thread to Select
    for (int i=0;i    {
        final int id=i;
        boolean selecting=dispatch(new Runnable()
        {
            public void run()
            {
            // ....
            }

        });
    }
}
Post a Comment

Labels

Java (159) Lucene-Solr (110) All (60) Interview (59) J2SE (53) Algorithm (37) Eclipse (35) Soft Skills (35) Code Example (31) Linux (26) JavaScript (23) Spring (22) Windows (22) Web Development (20) Tools (19) Nutch2 (18) Bugs (17) Debug (15) Defects (14) Text Mining (14) J2EE (13) Network (13) PowerShell (11) Chrome (9) Continuous Integration (9) How to (9) Learning code (9) Performance (9) UIMA (9) html (9) Design (8) Dynamic Languages (8) Http Client (8) Maven (8) Security (8) Trouble Shooting (8) bat (8) blogger (8) Big Data (7) Google (7) Guava (7) JSON (7) Problem Solving (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) IDE (5) Lesson Learned (5) Miscs (5) Programmer Skills (5) System Design (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) OpenNLP (4) Project Managment (4) Python (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Firefox (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Google Drive (2) Gson (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Bit Operation (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Troubleshooting (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts