Http Server No Response under overload in linux

Jul 11, 2015 at 3:35 PM
Edited Jul 11, 2015 at 3:49 PM
Hi!
 I wrote a Http Server with Httplistener(rest sdk 2.6), it worked ok,but when I did a load test  with loadrunner, a little time later it didn't work, and never work again. I found all threads blocked in                wait(unsigned int timeout)       function in pplxlinux.h. Did anyone have the same problem?Thank you! 
Jul 14, 2015 at 12:47 AM
Hi wuxiao,

Can you share a bit or post code showing what you are doing in your http_listener request handlers? It sounds like you have them all busy waiting? On some platforms like Linux all of the tasks and callbacks in the library run on a pretty basic threadpool. It sounds like you have all the threads in the pool blocked busy waiting on something. What platform are you running on?

Steve
Jul 14, 2015 at 3:44 AM
Edited Jul 14, 2015 at 3:45 AM
Hi Steve,

The platform is Centos,the handle function like this:

void CHttpRealSvr::handle_Method(http_request message, HTTP_METHOD_ENUM MethodType)
{
HTTP_DATA_RETURN_ROOT outPutData;
string strUrl;
web::uri RUrl = message.relative_uri();     
strUrl = string_t2string(RUrl.to_string());
string str = string_t2string(message.extract_string().get());
m_pHttpSvr->Reply(strUrl, MethodType, &str, &outPutData);
DealData(message, outPutData);
}

All Threads blocked in this function, more precisely to say in message.exact_string().get(),wait the task complete.
When I debuged the program with GDB, I found there had a epoll_wait() in one thead, when all threads blocked, I
found no epoll_wait() in any thread and the syn backlog would increase.No matter how long I waited,nothing changed.
Thank you for helping.
Jul 15, 2015 at 5:50 PM
Hi wuxiao,

Ok what I bet is happening is the underlying threadpool is getting overloaded with tasks and you have many that are performing blocking waits. It is possible that there are no threads available to service the request body read and signal extract_string. Our threadpool on Linux only creates 40 threads.

My recommendation would be to not perform a blocking wait for the rest of the HTTP request body to arrive. Calling get()/wait() on the task returned from the extract_string() function will block, and another task might be waiting to run to read the rest of the underlying request body and signal. Instead try hooking up a continuation to the task returned from extract_string(). This will allow the thread to not be blocked while waiting. For resources on how to use tasks take a look here and here.

Steve
Jul 16, 2015 at 1:12 PM
Hi Steve,
  Thank you for you advices. It works good.I changed the function like  this:
void CHttpRealSvr::handle_Method(http_request message, HTTP_METHOD_ENUM MethodType)
{
    string strUrl;
    web::uri RUrl = message.relative_uri();
    strUrl = string_t2string(RUrl.to_string());


    message.extract_string().then([=](pplx::task<utility::string_t> extractTask){
            string str ;
            HTTP_DATA_RETURN_ROOT outPutData;
            str = string_t2string(extractTask.get());
            m_pHttpSvr->Reply(strUrl, MethodType, &str, &outPutData);
            DealData(message, outPutData);
    });
}