1 % Testing the JDK
2
3 ## Using "make test" (the run-test framework)
4
5 This new way of running tests is developer-centric. It assumes that you have
6 built a JDK locally and want to test it. Running common test targets is simple,
7 and more complex ad-hoc combination of tests is possible. The user interface is
8 forgiving, and clearly report errors it cannot resolve.
9
10 The main target `test` uses the jdk-image as the tested product. There is
11 also an alternate target `exploded-test` that uses the exploded image
12 instead. Not all tests will run successfully on the exploded image, but using
13 this target can greatly improve rebuild times for certain workflows.
14
15 Previously, `make test` was used invoke an old system for running test, and
16 `make run-test` was used for the new test framework. For backward compatibility
17 with scripts and muscle memory, `run-test` (and variants like
18 `exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system
19 can still be accessed for some time using `cd test && make`.
20
21 Some example command-lines:
22
23 $ make test-tier1
24 $ make test-jdk_lang JTREG="JOBS=8"
25 $ make test TEST=jdk_lang
26 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
27 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
28 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
29 $ make exploded-test TEST=tier2
30
31 ### Configuration
32
33 To be able to run JTReg tests, `configure` needs to know where to find the
34 JTReg test framework. If it is not picked up automatically by configure, use
35 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
36 Note that this option should point to the JTReg home, i.e. the top directory,
37 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
38 environment variable to point to the JTReg home before running `configure`.)
39
40 ## Test selection
41
42 All functionality is available using the `test` make target. In this use case,
43 the test or tests to be executed is controlled using the `TEST` variable. To
44 speed up subsequent test runs with no source code changes, `test-only` can be
45 used instead, which do not depend on the source and test image build.
46
47 For some common top-level tests, direct make targets have been generated. This
48 includes all JTReg test groups, the hotspot gtest, and custom tests (if
49 present). This means that `make test-tier1` is equivalent to `make test
50 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
51 test runs, the `test TEST="x"` solution needs to be used.
52
53 The test specifications given in `TEST` is parsed into fully qualified test
54 descriptors, which clearly and unambigously show which tests will be run. As an
55 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
56 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
57 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can
58 always submit a list of fully qualified test descriptors in the `TEST` variable
59 if you want to shortcut the parser.
60
61 ### JTReg
62
63 JTReg tests can be selected either by picking a JTReg test group, or a selection
64 of files or directories containing JTReg tests.
65
66 JTReg test groups can be specified either without a test root, e.g. `:tier1`
67 (or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`,
68 `test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test
69 root can be specified either as an absolute path, or a path relative to the
70 JDK top directory, or the `test` directory. For simplicity, the hotspot
71 JTReg test root, which really is `hotspot/jtreg` can be abbreviated as
72 just `hotspot`.
73
74 When specified without a test root, all matching groups from all test roots
75 will be added. Otherwise, only the group from the specified test root will be
76 added.
77
78 Individual JTReg tests or directories containing JTReg tests can also be
79 specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or
80 `hotspot/jtreg/native_sanity`. Just like for test root selection, you can
81 either specify an absolute path (which can even point to JTReg tests outside
82 the source tree), or a path relative to either the JDK top directory or the
83 `test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as
84 well.
85
86 As long as the test groups or test paths can be uniquely resolved, you do not
87 need to enter the `jtreg:` prefix. If this is not possible, or if you want to
88 use a fully qualified test descriptor, add `jtreg:`, e.g.
89 `jtreg:test/hotspot/jtreg/native_sanity`.
90
91 ### Gtest
92
93 Since the Hotspot Gtest suite is so quick, the default is to run all tests.
94 This is specified by just `gtest`, or as a fully qualified test descriptor
95 `gtest:all`.
96
97 If you want, you can single out an individual test or a group of tests, for
98 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
99 can be particularly useful if you want to run a shaky test repeatedly.
100
101 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
102 is defined by adding `/<variant>` to the test descriptor, e.g.
103 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
104 variant present (e.g. server, client). So if you only have the server JVM
105 present, then `gtest:all` will be equivalent to `gtest:all/server`.
106
107 ### Special tests
108
109 A handful of odd tests that are not covered by any other testing framework are
110 accessible using the `special:` test descriptor. Currently, this includes
111 `failure-handler` and `make`.
112
113 * Failure handler testing is run using `special:failure-handler` or just
114 `failure-handler` as test descriptor.
115
116 * Tests for the build system, including both makefiles and related
117 functionality, is run using `special:make` or just `make` as test
118 descriptor. This is equivalent to `special:make:all`.
119
120 A specific make test can be run by supplying it as argument, e.g.
121 `special:make:idea`. As a special syntax, this can also be expressed as
122 `make-idea`, which allows for command lines as `make test-make-idea`.
123
124 ## Test results and summary
125
126 At the end of the test run, a summary of all tests run will be presented. This
127 will have a consistent look, regardless of what test suites were used. This is
128 a sample summary:
129
130 ==============================
131 Test summary
132 ==============================
133 TEST TOTAL PASS FAIL ERROR
134 >> jtreg:jdk/test:tier1 1867 1865 2 0 <<
135 jtreg:langtools/test:tier1 4711 4711 0 0
136 jtreg:nashorn/test:tier1 133 133 0 0
137 ==============================
138 TEST FAILURE
139
140 Tests where the number of TOTAL tests does not equal the number of PASSed tests
141 will be considered a test failure. These are marked with the `>> ... <<` marker
142 for easy identification.
143
144 The classification of non-passed tests differs a bit between test suites. In
145 the summary, ERROR is used as a catch-all for tests that neither passed nor are
146 classified as failed by the framework. This might indicate test framework
147 error, timeout or other problems.
148
149 In case of test failures, `make test` will exit with a non-zero exit value.
150
151 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
152 where TEST_ID is a path-safe conversion from the fully qualified test
153 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
154 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
155 test run.
156
157 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
158 some frameworks, this directory might contain information that is useful in
159 determining the cause of a failed test.
160
161 ## Test suite control
162
163 It is possible to control various aspects of the test suites using make control
164 variables.
165
166 These variables use a keyword=value approach to allow multiple values to be
167 set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg
168 concurrency level to 1 and the timeout factor to 8. This is equivalent to
169 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that
170 the `JTREG` variable is parsed and verified for correctness, so
171 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just
172 pass unnoticed.
173
174 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
175 normally eats `;`, the recommended usage is to write the assignment inside
176 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
177 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
178
179 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`.
180 Also, as a special technique, the string `%20` will be replaced with space for
181 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
182 This can be useful if you have layers of scripts and have trouble getting
183 proper quoting of command line arguments through.)
184
185 As far as possible, the names of the keywords have been standardized between
186 test suites.
187
188 ### JTReg keywords
189
190 #### JOBS
191 The test concurrency (`-concurrency`).
192
193 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
194 JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but
195 never more than 12.
196
197 #### TIMEOUT
198 The timeout factor (`-timeoutFactor`).
199
200 Defaults to 4.
201
202 #### TEST_MODE
203 The test mode (`-agentvm`, `-samevm` or `-othervm`).
204
205 Defaults to `-agentvm`.
206
207 #### ASSERT
208 Enable asserts (`-ea -esa`, or none).
209
210 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
211 for hotspot.
212
213 #### VERBOSE
214 The verbosity level (`-verbose`).
215
216 Defaults to `fail,error,summary`.
217
218 #### RETAIN
219 What test data to retain (`-retain`).
220
221 Defaults to `fail,error`.
222
223 #### MAX_MEM
224 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
225
226 Limit memory consumption for JTReg test framework and VM under test. Set to 0
227 to disable the limits.
228
229 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
230
231 #### OPTIONS
232 Additional options to the JTReg test framework.
233
234 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
235
236 #### JAVA_OPTIONS
237 Additional Java options to JTReg (`-javaoption`).
238
239 #### VM_OPTIONS
240 Additional VM options to JTReg (`-vmoption`).
241
242 ### Gtest keywords
243
244 #### REPEAT
245 The number of times to repeat the tests (`--gtest_repeat`).
246
247 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
248 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
249 problem.
250
251 #### OPTIONS
252 Additional options to the Gtest test framework.
253
254 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
255
256 ---
257 # Override some definitions in the global css file that are not optimal for
258 # this document.
259 header-includes:
260 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
261 ---