1 % Testing the JDK 2 3 ## Using "make test" (the run-test framework) 4 5 This new way of running tests is developer-centric. It assumes that you have 6 built a JDK locally and want to test it. Running common test targets is simple, 7 and more complex ad-hoc combination of tests is possible. The user interface is 8 forgiving, and clearly report errors it cannot resolve. 9 10 The main target `test` uses the jdk-image as the tested product. There is 11 also an alternate target `exploded-test` that uses the exploded image 12 instead. Not all tests will run successfully on the exploded image, but using 13 this target can greatly improve rebuild times for certain workflows. 14 15 Previously, `make test` was used invoke an old system for running test, and 16 `make run-test` was used for the new test framework. For backward compatibility 17 with scripts and muscle memory, `run-test` (and variants like 18 `exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system 19 can still be accessed for some time using `cd test && make`. 20 21 Some example command-lines: 22 23 $ make test-tier1 24 $ make test-jdk_lang JTREG="JOBS=8" 25 $ make test TEST=jdk_lang 26 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" 27 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" 28 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" 29 $ make exploded-test TEST=tier2 30 31 ### Configuration 32 33 To be able to run JTReg tests, `configure` needs to know where to find the 34 JTReg test framework. If it is not picked up automatically by configure, use 35 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework. 36 Note that this option should point to the JTReg home, i.e. the top directory, 37 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME` 38 environment variable to point to the JTReg home before running `configure`.) 39 40 ## Test selection 41 42 All functionality is available using the `test` make target. In this use case, 43 the test or tests to be executed is controlled using the `TEST` variable. To 44 speed up subsequent test runs with no source code changes, `test-only` can be 45 used instead, which do not depend on the source and test image build. 46 47 For some common top-level tests, direct make targets have been generated. This 48 includes all JTReg test groups, the hotspot gtest, and custom tests (if 49 present). This means that `make test-tier1` is equivalent to `make test 50 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex 51 test runs, the `test TEST="x"` solution needs to be used. 52 53 The test specifications given in `TEST` is parsed into fully qualified test 54 descriptors, which clearly and unambigously show which tests will be run. As an 55 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 56 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 57 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can 58 always submit a list of fully qualified test descriptors in the `TEST` variable 59 if you want to shortcut the parser. 60 61 ### JTReg 62 63 JTReg tests can be selected either by picking a JTReg test group, or a selection 64 of files or directories containing JTReg tests. 65 66 JTReg test groups can be specified either without a test root, e.g. `:tier1` 67 (or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`, 68 `test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test 69 root can be specified either as an absolute path, or a path relative to the 70 JDK top directory, or the `test` directory. For simplicity, the hotspot 71 JTReg test root, which really is `hotspot/jtreg` can be abbreviated as 72 just `hotspot`. 73 74 When specified without a test root, all matching groups from all test roots 75 will be added. Otherwise, only the group from the specified test root will be 76 added. 77 78 Individual JTReg tests or directories containing JTReg tests can also be 79 specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or 80 `hotspot/jtreg/native_sanity`. Just like for test root selection, you can 81 either specify an absolute path (which can even point to JTReg tests outside 82 the source tree), or a path relative to either the JDK top directory or the 83 `test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as 84 well. 85 86 As long as the test groups or test paths can be uniquely resolved, you do not 87 need to enter the `jtreg:` prefix. If this is not possible, or if you want to 88 use a fully qualified test descriptor, add `jtreg:`, e.g. 89 `jtreg:test/hotspot/jtreg/native_sanity`. 90 91 ### Gtest 92 93 Since the Hotspot Gtest suite is so quick, the default is to run all tests. 94 This is specified by just `gtest`, or as a fully qualified test descriptor 95 `gtest:all`. 96 97 If you want, you can single out an individual test or a group of tests, for 98 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This 99 can be particularly useful if you want to run a shaky test repeatedly. 100 101 For Gtest, there is a separate test suite for each JVM variant. The JVM variant 102 is defined by adding `/<variant>` to the test descriptor, e.g. 103 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM 104 variant present (e.g. server, client). So if you only have the server JVM 105 present, then `gtest:all` will be equivalent to `gtest:all/server`. 106 107 ### Special tests 108 109 A handful of odd tests that are not covered by any other testing framework are 110 accessible using the `special:` test descriptor. Currently, this includes 111 `hotspot-internal`, `failure-handler` and `make`. 112 113 * Hotspot legacy internal testing (run using `-XX:+ExecuteInternalVMTests`) 114 is run using `special:hotspot-internal` or just `hotspot-internal` as test 115 descriptor, and will only work on a debug JVM. 116 117 * Failure handler testing is run using `special:failure-handler` or just 118 `failure-handler` as test descriptor. 119 120 * Tests for the build system, including both makefiles and related 121 functionality, is run using `special:make` or just `make` as test 122 descriptor. This is equivalent to `special:make:all`. 123 124 A specific make test can be run by supplying it as argument, e.g. 125 `special:make:idea`. As a special syntax, this can also be expressed as 126 `make-idea`, which allows for command lines as `make test-make-idea`. 127 128 ## Test results and summary 129 130 At the end of the test run, a summary of all tests run will be presented. This 131 will have a consistent look, regardless of what test suites were used. This is 132 a sample summary: 133 134 ============================== 135 Test summary 136 ============================== 137 TEST TOTAL PASS FAIL ERROR 138 >> jtreg:jdk/test:tier1 1867 1865 2 0 << 139 jtreg:langtools/test:tier1 4711 4711 0 0 140 jtreg:nashorn/test:tier1 133 133 0 0 141 ============================== 142 TEST FAILURE 143 144 Tests where the number of TOTAL tests does not equal the number of PASSed tests 145 will be considered a test failure. These are marked with the `>> ... <<` marker 146 for easy identification. 147 148 The classification of non-passed tests differs a bit between test suites. In 149 the summary, ERROR is used as a catch-all for tests that neither passed nor are 150 classified as failed by the framework. This might indicate test framework 151 error, timeout or other problems. 152 153 In case of test failures, `make test` will exit with a non-zero exit value. 154 155 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`, 156 where TEST_ID is a path-safe conversion from the fully qualified test 157 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is 158 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the 159 test run. 160 161 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For 162 some frameworks, this directory might contain information that is useful in 163 determining the cause of a failed test. 164 165 ## Test suite control 166 167 It is possible to control various aspects of the test suites using make control 168 variables. 169 170 These variables use a keyword=value approach to allow multiple values to be 171 set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg 172 concurrency level to 1 and the timeout factor to 8. This is equivalent to 173 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that 174 the `JTREG` variable is parsed and verified for correctness, so 175 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just 176 pass unnoticed. 177 178 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell 179 normally eats `;`, the recommended usage is to write the assignment inside 180 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved, 181 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`. 182 183 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`. 184 Also, as a special technique, the string `%20` will be replaced with space for 185 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. 186 This can be useful if you have layers of scripts and have trouble getting 187 proper quoting of command line arguments through.) 188 189 As far as possible, the names of the keywords have been standardized between 190 test suites. 191 192 ### JTReg keywords 193 194 #### JOBS 195 The test concurrency (`-concurrency`). 196 197 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to 198 JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but 199 never more than 12. 200 201 #### TIMEOUT 202 The timeout factor (`-timeoutFactor`). 203 204 Defaults to 4. 205 206 #### TEST_MODE 207 The test mode (`-agentvm`, `-samevm` or `-othervm`). 208 209 Defaults to `-agentvm`. 210 211 #### ASSERT 212 Enable asserts (`-ea -esa`, or none). 213 214 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except 215 for hotspot. 216 217 #### VERBOSE 218 The verbosity level (`-verbose`). 219 220 Defaults to `fail,error,summary`. 221 222 #### RETAIN 223 What test data to retain (`-retain`). 224 225 Defaults to `fail,error`. 226 227 #### MAX_MEM 228 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none). 229 230 Limit memory consumption for JTReg test framework and VM under test. Set to 0 231 to disable the limits. 232 233 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit). 234 235 #### OPTIONS 236 Additional options to the JTReg test framework. 237 238 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options. 239 240 #### JAVA_OPTIONS 241 Additional Java options to JTReg (`-javaoption`). 242 243 #### VM_OPTIONS 244 Additional VM options to JTReg (`-vmoption`). 245 246 ### Gtest keywords 247 248 #### REPEAT 249 The number of times to repeat the tests (`--gtest_repeat`). 250 251 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful 252 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent 253 problem. 254 255 #### OPTIONS 256 Additional options to the Gtest test framework. 257 258 Use `GTEST="OPTIONS=--help"` to see all available Gtest options. 259 260 --- 261 # Override some definitions in the global css file that are not optimal for 262 # this document. 263 header-includes: 264 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>' 265 ---