1 % Testing the JDK
2
3 ## Using "make test" (the run-test framework)
4
5 This new way of running tests is developer-centric. It assumes that you have
6 built a JDK locally and want to test it. Running common test targets is simple,
7 and more complex ad-hoc combination of tests is possible. The user interface is
8 forgiving, and clearly report errors it cannot resolve.
9
10 The main target `test` uses the jdk-image as the tested product. There is
11 also an alternate target `exploded-test` that uses the exploded image
12 instead. Not all tests will run successfully on the exploded image, but using
13 this target can greatly improve rebuild times for certain workflows.
14
15 Previously, `make test` was used to invoke an old system for running tests, and
16 `make run-test` was used for the new test framework. For backward compatibility
17 with scripts and muscle memory, `run-test` (and variants like
18 `exploded-run-test` or `run-test-tier1`) are kept as aliases.
19
20 Some example command-lines:
21
22 $ make test-tier1
23 $ make test-jdk_lang JTREG="JOBS=8"
24 $ make test TEST=jdk_lang
25 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
26 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
27 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
28 $ make test TEST="jtreg:test/hotspot/jtreg/runtime/containers/docker" JTREG="JAVA_OPTIONS=-Djdk.test.docker.image.name=ubuntu -Djdk.test.docker.image.version=latest"
29 $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
30 $ make exploded-test TEST=tier2
31
32 ### Configuration
33
34 To be able to run JTReg tests, `configure` needs to know where to find the
35 JTReg test framework. If it is not picked up automatically by configure, use
36 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
37 Note that this option should point to the JTReg home, i.e. the top directory,
38 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
39 environment variable to point to the JTReg home before running `configure`.)
40
41 To be able to run microbenchmarks, `configure` needs to know where to find
42 the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
43 containing the core JMH and transitive dependencies. The recommended dependencies
44 can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which
45 `--with-jmh=build/jmh/jars` should work.
46
47 ## Test selection
48
49 All functionality is available using the `test` make target. In this use case,
50 the test or tests to be executed is controlled using the `TEST` variable. To
51 speed up subsequent test runs with no source code changes, `test-only` can be
52 used instead, which do not depend on the source and test image build.
53
54 For some common top-level tests, direct make targets have been generated. This
55 includes all JTReg test groups, the hotspot gtest, and custom tests (if
56 present). This means that `make test-tier1` is equivalent to `make test
57 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
58 test runs, the `test TEST="x"` solution needs to be used.
59
60 The test specifications given in `TEST` is parsed into fully qualified test
61 descriptors, which clearly and unambigously show which tests will be run. As an
62 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
63 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
64 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can
65 always submit a list of fully qualified test descriptors in the `TEST` variable
66 if you want to shortcut the parser.
67
68 ### JTReg
69
70 JTReg tests can be selected either by picking a JTReg test group, or a selection
71 of files or directories containing JTReg tests.
72
73 JTReg test groups can be specified either without a test root, e.g. `:tier1`
74 (or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`,
75 `test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test
76 root can be specified either as an absolute path, or a path relative to the
77 JDK top directory, or the `test` directory. For simplicity, the hotspot
78 JTReg test root, which really is `hotspot/jtreg` can be abbreviated as
79 just `hotspot`.
80
81 When specified without a test root, all matching groups from all test roots
82 will be added. Otherwise, only the group from the specified test root will be
83 added.
84
85 Individual JTReg tests or directories containing JTReg tests can also be
86 specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or
87 `hotspot/jtreg/native_sanity`. Just like for test root selection, you can
88 either specify an absolute path (which can even point to JTReg tests outside
89 the source tree), or a path relative to either the JDK top directory or the
90 `test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as
91 well.
92
93 As long as the test groups or test paths can be uniquely resolved, you do not
94 need to enter the `jtreg:` prefix. If this is not possible, or if you want to
95 use a fully qualified test descriptor, add `jtreg:`, e.g.
96 `jtreg:test/hotspot/jtreg/native_sanity`.
97
98 ### Gtest
99
100 Since the Hotspot Gtest suite is so quick, the default is to run all tests.
101 This is specified by just `gtest`, or as a fully qualified test descriptor
102 `gtest:all`.
103
104 If you want, you can single out an individual test or a group of tests, for
105 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
106 can be particularly useful if you want to run a shaky test repeatedly.
107
108 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
109 is defined by adding `/<variant>` to the test descriptor, e.g.
110 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
111 variant present (e.g. server, client). So if you only have the server JVM
112 present, then `gtest:all` will be equivalent to `gtest:all/server`.
113
114 ### Microbenchmarks
115
116 Which microbenchmarks to run is selected using a regular expression
117 following the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This
118 delegates the test selection to JMH, meaning package name, class name and even
119 benchmark method names can be used to select tests.
120
121 Using special characters like `|` in the regular expression is possible, but
122 needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`.
123
124 ### Special tests
125
126 A handful of odd tests that are not covered by any other testing framework are
127 accessible using the `special:` test descriptor. Currently, this includes
128 `failure-handler` and `make`.
129
130 * Failure handler testing is run using `special:failure-handler` or just
131 `failure-handler` as test descriptor.
132
133 * Tests for the build system, including both makefiles and related
134 functionality, is run using `special:make` or just `make` as test
135 descriptor. This is equivalent to `special:make:all`.
136
137 A specific make test can be run by supplying it as argument, e.g.
138 `special:make:idea`. As a special syntax, this can also be expressed as
139 `make-idea`, which allows for command lines as `make test-make-idea`.
140
141 ## Test results and summary
142
143 At the end of the test run, a summary of all tests run will be presented. This
144 will have a consistent look, regardless of what test suites were used. This is
145 a sample summary:
146
147 ==============================
148 Test summary
149 ==============================
150 TEST TOTAL PASS FAIL ERROR
151 >> jtreg:jdk/test:tier1 1867 1865 2 0 <<
152 jtreg:langtools/test:tier1 4711 4711 0 0
153 jtreg:nashorn/test:tier1 133 133 0 0
154 ==============================
155 TEST FAILURE
156
157 Tests where the number of TOTAL tests does not equal the number of PASSed tests
158 will be considered a test failure. These are marked with the `>> ... <<` marker
159 for easy identification.
160
161 The classification of non-passed tests differs a bit between test suites. In
162 the summary, ERROR is used as a catch-all for tests that neither passed nor are
163 classified as failed by the framework. This might indicate test framework
164 error, timeout or other problems.
165
166 In case of test failures, `make test` will exit with a non-zero exit value.
167
168 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
169 where TEST_ID is a path-safe conversion from the fully qualified test
170 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
171 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
172 test run.
173
174 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
175 some frameworks, this directory might contain information that is useful in
176 determining the cause of a failed test.
177
178 ## Test suite control
179
180 It is possible to control various aspects of the test suites using make control
181 variables.
182
183 These variables use a keyword=value approach to allow multiple values to be
184 set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg
185 concurrency level to 1 and the timeout factor to 8. This is equivalent to
186 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that
187 the `JTREG` variable is parsed and verified for correctness, so
188 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just
189 pass unnoticed.
190
191 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
192 normally eats `;`, the recommended usage is to write the assignment inside
193 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
194 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
195
196 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`.
197 Also, as a special technique, the string `%20` will be replaced with space for
198 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
199 This can be useful if you have layers of scripts and have trouble getting
200 proper quoting of command line arguments through.)
201
202 As far as possible, the names of the keywords have been standardized between
203 test suites.
204
205 ### General keywords (TEST_OPTS)
206
207 Some keywords are valid across different test suites. If you want to run
208 tests from multiple test suites, or just don't want to care which test suite specific
209 control variable to use, then you can use the general TEST_OPTS control variable.
210
211 There are also some keywords that applies globally to the test runner system,
212 not to any specific test suites. These are also available as TEST_OPTS keywords.
213
214 #### JOBS
215
216 Currently only applies to JTReg.
217
218 #### TIMEOUT_FACTOR
219
220 Currently only applies to JTReg.
221
222 #### VM_OPTIONS
223
224 Applies to JTReg, GTest and Micro.
225
226 #### JAVA_OPTIONS
227
228 Applies to JTReg, GTest and Micro.
229
230 #### AOT_MODULES
231
232 Applies to JTReg and GTest.
233
234 #### JCOV
235
236 This keywords applies globally to the test runner system. If set to `true`, it
237 enables JCov coverage reporting for all tests run. To be useful, the JDK under
238 test must be run with a JDK built with JCov instrumentation (`configure
239 --with-jcov=<path to directory containing lib/jcov.jar>`, `make jcov-image`).
240
241 The simplest way to run tests with JCov coverage report is to use the special
242 target `jcov-test` instead of `test`, e.g. `make jcov-test TEST=jdk_lang`. This
243 will make sure the JCov image is built, and that JCov reporting is enabled.
244
245 The JCov report is stored in `build/$BUILD/test-results/jcov-output`.
246
247 Please note that running with JCov reporting can be very memory intensive.
248
249 ### JTReg keywords
250
251 #### JOBS
252 The test concurrency (`-concurrency`).
253
254 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
255 JOBS, except for Hotspot, where the default is *number of CPU cores/2* (for
256 sparc, if more than 16 cpus, then *number of CPU cores/5*, otherwise *number of
257 CPU cores/4*), but never more than *memory size in GB/2*.
258
259 #### TIMEOUT_FACTOR
260 The timeout factor (`-timeoutFactor`).
261
262 Defaults to 4.
263
264 #### TEST_MODE
265 The test mode (`-agentvm`, `-samevm` or `-othervm`).
266
267 Defaults to `-agentvm`.
268
269 #### ASSERT
270 Enable asserts (`-ea -esa`, or none).
271
272 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
273 for hotspot.
274
275 #### VERBOSE
276 The verbosity level (`-verbose`).
277
278 Defaults to `fail,error,summary`.
279
280 #### RETAIN
281 What test data to retain (`-retain`).
282
283 Defaults to `fail,error`.
284
285 #### MAX_MEM
286 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
287
288 Limit memory consumption for JTReg test framework and VM under test. Set to 0
289 to disable the limits.
290
291 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
292
293 #### KEYWORDS
294
295 JTReg kewords sent to JTReg using `-k`. Please be careful in making sure that
296 spaces and special characters (like `!`) are properly quoted. To avoid some
297 issues, the special value `%20` can be used instead of space.
298
299 #### EXTRA_PROBLEM_LISTS
300
301 Use additional problem lists file or files, in addition to the default
302 ProblemList.txt located at the JTReg test roots.
303
304 If multiple file names are specified, they should be separated by space (or, to
305 help avoid quoting issues, the special value `%20`).
306
307 The file names should be either absolute, or relative to the JTReg test root of
308 the tests to be run.
309
310
311 #### OPTIONS
312 Additional options to the JTReg test framework.
313
314 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
315
316 #### JAVA_OPTIONS
317 Additional Java options to JTReg (`-javaoption`).
318
319 This is useful for tests which may require additional parameters on specific platforms.
320 For example, `make test TEST="jtreg:test/hotspot/jtreg/runtime/containers/docker"`
321 passed on Ubuntu 16.04 but failed on Ubuntu 18.04. To run these tests correctly, the
322 correct docker image parameters are required on Ubuntu 18.04 by using `JAVA_OPTIONS`.
323
324 #### VM_OPTIONS
325 Additional VM options to JTReg (`-vmoption`).
326
327 #### AOT_MODULES
328
329 Generate AOT modules before testing for the specified module, or set of
330 modules. If multiple modules are specified, they should be separated by space
331 (or, to help avoid quoting issues, the special value `%20`).
332
333 ### Gtest keywords
334
335 #### REPEAT
336 The number of times to repeat the tests (`--gtest_repeat`).
337
338 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
339 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
340 problem.
341
342 #### OPTIONS
343 Additional options to the Gtest test framework.
344
345 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
346
347 #### AOT_MODULES
348
349 Generate AOT modules before testing for the specified module, or set of
350 modules. If multiple modules are specified, they should be separated by space
351 (or, to help avoid quoting issues, the special value `%20`).
352
353 ### Microbenchmark keywords
354
355 #### FORK
356 Override the number of benchmark forks to spawn. Same as specifying `-f <num>`.
357
358 #### ITER
359 Number of measurement iterations per fork. Same as specifying `-i <num>`.
360
361 #### TIME
362 Amount of time to spend in each measurement iteration, in seconds. Same as
363 specifying `-r <num>`
364
365 #### WARMUP_ITER
366 Number of warmup iterations to run before the measurement phase in each fork.
367 Same as specifying `-wi <num>`.
368
369 #### WARMUP_TIME
370 Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`.
371
372 #### RESULTS_FORMAT
373 Specify to have the test run save a log of the values. Accepts the same values
374 as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`.
375
376 #### VM_OPTIONS
377 Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>`
378
379 #### OPTIONS
380 Additional arguments to send to JMH.
381
382 ---
383 # Override some definitions in the global css file that are not optimal for
384 # this document.
385 header-includes:
386 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
387 ---