Tuesday, August 11, 2015

How to timeout tests in gtest

If you want to limit the time to run the test, you can use the following macro: 

#include <future>
#define TEST_TIMEOUT_BEGIN   std::promise<bool> promisedFinished; \
                              auto futureResult = promisedFinished.get_future(); \
                              std::thread([](std::promise<bool>& finished) {
 
#define TEST_TIMEOUT_FAIL_END(X)  finished.set_value(true); \
                                   }, std::ref(promisedFinished)).detach(); \
                                   EXPECT_TRUE(futureResult.wait_for(std::chrono::milliseconds(X)) != std::future_status::timeout);
 
#define TEST_TIMEOUT_SUCCESS_END(X)  finished.set_value(true); \
                                      }, std::ref(promisedFinished)).detach(); \
                                      EXPECT_FALSE(futureResult.wait_for(std::chrono::milliseconds(X)) != std::future_status::timeout);

 

Example:

 

TEST(some_unit, LongCalculationTimeout)
{
  TEST_TIMEOUT_BEGIN
    EXPECT_EQ(10, long_calculation_function());
  TEST_TIMEOUT_FAIL_END(1000)
}

Test will not pass and will be aborted, If execution of long_calculation_function() takes more time than 1000 ms.

In rare cases, we want to test that the function will take at least some time. In that case you can use TEST_TIMEOUT_SUCCESS_END instead of TEST_TIMEOUT_FAIL_END. Test will run for a specified time and if the code is still running after this time, test will pass and program starts next test.

Posted by at
Labels: , , ,

9 comments:

  1. Geoffrey HunterJune 12, 2017 at 2:29 PM

    If you add a "&" to the lambda in the TEST_TIMEOUT_BEGIN macro, you can capture (and use) local variables such as "this" in-between the BEGIN and END macros.

    e.g.
    #define TEST_TIMEOUT_BEGIN std::promise promisedFinished; \
    auto futureResult = promisedFinished.get_future(); \
    std::thread([&](std::promise& finished) {

    ReplyDelete
  2. AnonymousJune 25, 2017 at 4:44 PM

    Please don't use old C-style preprocessor macros in C++. You're already using C++11 so why not leverage it a little more. See:
    https://gist.github.com/compomega/4f59aee81bb03e2376d5421ca6282b67

    ReplyDelete
    Replies
    1. AnonymousJuly 12, 2019 at 3:32 AM

      Why not? It's much shorter and cleaner than having 80 lines of code with templates and lambda expressions to get function that should've been part of gtest in the first place. Also, gtest relies heavily on macros so it seems fitting to use those here

      Delete
  3. AnonymousAugust 1, 2017 at 2:04 AM

    2017, still no official gtest utility for timeout, using your macro, thx man!

    ReplyDelete
  4. chandanJuly 10, 2018 at 10:28 AM

    Thanks Anton. It saved us a lot of time.

    ReplyDelete
  5. UnknownOctober 21, 2018 at 4:59 PM

    ok

    ReplyDelete
  6. X YFebruary 8, 2019 at 1:24 AM

    Thanks, was a great tip!

    ReplyDelete
  7. vextorspaceApril 1, 2020 at 7:43 PM

    the TEST_TIMEOUT_SUCCESS_END seems to kill the test run if it succeeds. I get:

    [1] + Done "/usr/bin/gdb" --interpreter=mi --tty=${DbgTerm} 0<"/tmp/Microsoft-MIEngine-In-v786t2mc.mv2" 1>"/tmp/Microsoft-MIEngine-Out-lt0qxlb8.r3u"


    in vscode with googletest. In the console it just doesn't give me the test summary.

    Otherwise, working great!

    ReplyDelete
  8. MarkOctober 20, 2021 at 9:45 PM

    Warning, this code doesn't handle exceptions well. If the test body (between the TEST_TIMEOUT_BEGIN() and TEST_TIMEOUT_FAIL_END()) throws an exception, then the test process terminates and does not gracefully shutdown. For example, the TearDownTestSuite() is never called.
    That said, this might be the best hoped behavior. If the test times-out, the body of the test continues to run in its own thread and it is not killed unless the entire process terminates. So the only correct thing to do after the timeout is terminate the process immediately.

    ReplyDelete

Subscribe to: Post Comments (Atom)