9
10 The main target `test` uses the jdk-image as the tested product. There is
11 also an alternate target `exploded-test` that uses the exploded image
12 instead. Not all tests will run successfully on the exploded image, but using
13 this target can greatly improve rebuild times for certain workflows.
14
15 Previously, `make test` was used invoke an old system for running test, and
16 `make run-test` was used for the new test framework. For backward compatibility
17 with scripts and muscle memory, `run-test` (and variants like
18 `exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system
19 can still be accessed for some time using `cd test && make`.
20
21 Some example command-lines:
22
23 $ make test-tier1
24 $ make test-jdk_lang JTREG="JOBS=8"
25 $ make test TEST=jdk_lang
26 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
27 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
28 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
29 $ make exploded-test TEST=tier2
30
31 ### Configuration
32
33 To be able to run JTReg tests, `configure` needs to know where to find the
34 JTReg test framework. If it is not picked up automatically by configure, use
35 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
36 Note that this option should point to the JTReg home, i.e. the top directory,
37 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
38 environment variable to point to the JTReg home before running `configure`.)
39
40 ## Test selection
41
42 All functionality is available using the `test` make target. In this use case,
43 the test or tests to be executed is controlled using the `TEST` variable. To
44 speed up subsequent test runs with no source code changes, `test-only` can be
45 used instead, which do not depend on the source and test image build.
46
47 For some common top-level tests, direct make targets have been generated. This
48 includes all JTReg test groups, the hotspot gtest, and custom tests (if
49 present). This means that `make test-tier1` is equivalent to `make test
50 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
51 test runs, the `test TEST="x"` solution needs to be used.
52
53 The test specifications given in `TEST` is parsed into fully qualified test
54 descriptors, which clearly and unambigously show which tests will be run. As an
55 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
56 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
57 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can
58 always submit a list of fully qualified test descriptors in the `TEST` variable
59 if you want to shortcut the parser.
87 need to enter the `jtreg:` prefix. If this is not possible, or if you want to
88 use a fully qualified test descriptor, add `jtreg:`, e.g.
89 `jtreg:test/hotspot/jtreg/native_sanity`.
90
91 ### Gtest
92
93 Since the Hotspot Gtest suite is so quick, the default is to run all tests.
94 This is specified by just `gtest`, or as a fully qualified test descriptor
95 `gtest:all`.
96
97 If you want, you can single out an individual test or a group of tests, for
98 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
99 can be particularly useful if you want to run a shaky test repeatedly.
100
101 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
102 is defined by adding `/<variant>` to the test descriptor, e.g.
103 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
104 variant present (e.g. server, client). So if you only have the server JVM
105 present, then `gtest:all` will be equivalent to `gtest:all/server`.
106
107 ### Special tests
108
109 A handful of odd tests that are not covered by any other testing framework are
110 accessible using the `special:` test descriptor. Currently, this includes
111 `failure-handler` and `make`.
112
113 * Failure handler testing is run using `special:failure-handler` or just
114 `failure-handler` as test descriptor.
115
116 * Tests for the build system, including both makefiles and related
117 functionality, is run using `special:make` or just `make` as test
118 descriptor. This is equivalent to `special:make:all`.
119
120 A specific make test can be run by supplying it as argument, e.g.
121 `special:make:idea`. As a special syntax, this can also be expressed as
122 `make-idea`, which allows for command lines as `make test-make-idea`.
123
124 ## Test results and summary
125
126 At the end of the test run, a summary of all tests run will be presented. This
235
236 #### JAVA_OPTIONS
237 Additional Java options to JTReg (`-javaoption`).
238
239 #### VM_OPTIONS
240 Additional VM options to JTReg (`-vmoption`).
241
242 ### Gtest keywords
243
244 #### REPEAT
245 The number of times to repeat the tests (`--gtest_repeat`).
246
247 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
248 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
249 problem.
250
251 #### OPTIONS
252 Additional options to the Gtest test framework.
253
254 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
255
256 ---
257 # Override some definitions in the global css file that are not optimal for
258 # this document.
259 header-includes:
260 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
261 ---
|
9
10 The main target `test` uses the jdk-image as the tested product. There is
11 also an alternate target `exploded-test` that uses the exploded image
12 instead. Not all tests will run successfully on the exploded image, but using
13 this target can greatly improve rebuild times for certain workflows.
14
15 Previously, `make test` was used invoke an old system for running test, and
16 `make run-test` was used for the new test framework. For backward compatibility
17 with scripts and muscle memory, `run-test` (and variants like
18 `exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system
19 can still be accessed for some time using `cd test && make`.
20
21 Some example command-lines:
22
23 $ make test-tier1
24 $ make test-jdk_lang JTREG="JOBS=8"
25 $ make test TEST=jdk_lang
26 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
27 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
28 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
29 $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
30 $ make exploded-test TEST=tier2
31
32 ### Configuration
33
34 To be able to run JTReg tests, `configure` needs to know where to find the
35 JTReg test framework. If it is not picked up automatically by configure, use
36 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
37 Note that this option should point to the JTReg home, i.e. the top directory,
38 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
39 environment variable to point to the JTReg home before running `configure`.)
40
41 To be able to run microbenchmarks, `configure` needs to know where to find
42 the JMH dependency. Use `--with-jmh=<path to JMH jars>` to point to a directory
43 containing the core JMH and transitive dependencies. The recommended dependencies
44 can be retrieved by running `sh make/devkit/createJMHBundle.sh`, after which
45 `--with-jmh=build/jmh/jars` should work.
46
47 ## Test selection
48
49 All functionality is available using the `test` make target. In this use case,
50 the test or tests to be executed is controlled using the `TEST` variable. To
51 speed up subsequent test runs with no source code changes, `test-only` can be
52 used instead, which do not depend on the source and test image build.
53
54 For some common top-level tests, direct make targets have been generated. This
55 includes all JTReg test groups, the hotspot gtest, and custom tests (if
56 present). This means that `make test-tier1` is equivalent to `make test
57 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
58 test runs, the `test TEST="x"` solution needs to be used.
59
60 The test specifications given in `TEST` is parsed into fully qualified test
61 descriptors, which clearly and unambigously show which tests will be run. As an
62 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
63 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
64 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can
65 always submit a list of fully qualified test descriptors in the `TEST` variable
66 if you want to shortcut the parser.
94 need to enter the `jtreg:` prefix. If this is not possible, or if you want to
95 use a fully qualified test descriptor, add `jtreg:`, e.g.
96 `jtreg:test/hotspot/jtreg/native_sanity`.
97
98 ### Gtest
99
100 Since the Hotspot Gtest suite is so quick, the default is to run all tests.
101 This is specified by just `gtest`, or as a fully qualified test descriptor
102 `gtest:all`.
103
104 If you want, you can single out an individual test or a group of tests, for
105 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
106 can be particularly useful if you want to run a shaky test repeatedly.
107
108 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
109 is defined by adding `/<variant>` to the test descriptor, e.g.
110 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
111 variant present (e.g. server, client). So if you only have the server JVM
112 present, then `gtest:all` will be equivalent to `gtest:all/server`.
113
114 ### Microbenchmarks
115
116 Which microbenchmarks to run is selected using a regular expression
117 following the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This
118 delegates the test selection to JMH, meaning package name, class name and even
119 benchmark method names can be used to select tests.
120
121 Using special characters like `|` in the regular expression is possible, but
122 needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`.
123
124 ### Special tests
125
126 A handful of odd tests that are not covered by any other testing framework are
127 accessible using the `special:` test descriptor. Currently, this includes
128 `failure-handler` and `make`.
129
130 * Failure handler testing is run using `special:failure-handler` or just
131 `failure-handler` as test descriptor.
132
133 * Tests for the build system, including both makefiles and related
134 functionality, is run using `special:make` or just `make` as test
135 descriptor. This is equivalent to `special:make:all`.
136
137 A specific make test can be run by supplying it as argument, e.g.
138 `special:make:idea`. As a special syntax, this can also be expressed as
139 `make-idea`, which allows for command lines as `make test-make-idea`.
140
141 ## Test results and summary
142
143 At the end of the test run, a summary of all tests run will be presented. This
252
253 #### JAVA_OPTIONS
254 Additional Java options to JTReg (`-javaoption`).
255
256 #### VM_OPTIONS
257 Additional VM options to JTReg (`-vmoption`).
258
259 ### Gtest keywords
260
261 #### REPEAT
262 The number of times to repeat the tests (`--gtest_repeat`).
263
264 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
265 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
266 problem.
267
268 #### OPTIONS
269 Additional options to the Gtest test framework.
270
271 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
272
273 ### Microbenchmark keywords
274
275 #### FORK
276 Override the number of benchmark forks to spawn. Same as specifying `-f <num>`.
277
278 #### ITER
279 Number of measurement iterations per fork. Same as specifying `-i <num>`.
280
281 #### TIME
282 Amount of time to spend in each measurement iteration, in seconds. Same as
283 specifying `-r <num>`
284
285 #### WARMUP_ITER
286 Number of warmup iterations to run before the measurement phase in each fork.
287 Same as specifying `-wi <num>`.
288
289 #### WARMUP_TIME
290 Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`.
291
292 #### RESULTS_FORMAT
293 Specify to have the test run save a log of the values. Accepts the same values
294 as `-rff`, i.e., `text`, `csv`, `scsv`, `json`, or `latex`.
295
296 #### VM_OPTIONS
297 Additional VM arguments to provide to forked off VMs. Same as `-jvmArgs <args>`
298
299 #### OPTIONS
300 Additional arguments to send to JMH.
301
302 ---
303 # Override some definitions in the global css file that are not optimal for
304 # this document.
305 header-includes:
306 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
307 ---
|