22 <li><a href="#configuration">Configuration</a></li>
23 </ul></li>
24 <li><a href="#test-selection">Test selection</a><ul>
25 <li><a href="#jtreg">JTReg</a></li>
26 <li><a href="#gtest">Gtest</a></li>
27 <li><a href="#microbenchmarks">Microbenchmarks</a></li>
28 <li><a href="#special-tests">Special tests</a></li>
29 </ul></li>
30 <li><a href="#test-results-and-summary">Test results and summary</a></li>
31 <li><a href="#test-suite-control">Test suite control</a><ul>
32 <li><a href="#general-keywords-test_opts">General keywords (TEST_OPTS)</a></li>
33 <li><a href="#jtreg-keywords">JTReg keywords</a></li>
34 <li><a href="#gtest-keywords">Gtest keywords</a></li>
35 <li><a href="#microbenchmark-keywords">Microbenchmark keywords</a></li>
36 </ul></li>
37 </ul>
38 </nav>
39 <h2 id="using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</h2>
40 <p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
41 <p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
42 <p>Previously, <code>make test</code> was used invoke an old system for running test, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases. The old system can still be accessed for some time using <code>cd test && make</code>.</p>
43 <p>Some example command-lines:</p>
44 <pre><code>$ make test-tier1
45 $ make test-jdk_lang JTREG="JOBS=8"
46 $ make test TEST=jdk_lang
47 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
48 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
49 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
50 $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
51 $ make exploded-test TEST=tier2</code></pre>
52 <h3 id="configuration">Configuration</h3>
53 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
54 <p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=<path to JMH jars></code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p>
55 <h2 id="test-selection">Test selection</h2>
56 <p>All functionality is available using the <code>test</code> make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, <code>test-only</code> can be used instead, which do not depend on the source and test image build.</p>
57 <p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make test-tier1</code> is equivalent to <code>make test TEST="tier1"</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>test TEST="x"</code> solution needs to be used.</p>
58 <p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
59 <h3 id="jtreg">JTReg</h3>
60 <p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p>
61 <p>JTReg test groups can be specified either without a test root, e.g. <code>:tier1</code> (or <code>tier1</code>, the initial colon is optional), or with, e.g. <code>hotspot:tier1</code>, <code>test/jdk:jdk_util</code> or <code>$(TOPDIR)/test/hotspot/jtreg:hotspot_all</code>. The test root can be specified either as an absolute path, or a path relative to the JDK top directory, or the <code>test</code> directory. For simplicity, the hotspot JTReg test root, which really is <code>hotspot/jtreg</code> can be abbreviated as just <code>hotspot</code>.</p>
62 <p>When specified without a test root, all matching groups from all test roots will be added. Otherwise, only the group from the specified test root will be added.</p>
|
22 <li><a href="#configuration">Configuration</a></li>
23 </ul></li>
24 <li><a href="#test-selection">Test selection</a><ul>
25 <li><a href="#jtreg">JTReg</a></li>
26 <li><a href="#gtest">Gtest</a></li>
27 <li><a href="#microbenchmarks">Microbenchmarks</a></li>
28 <li><a href="#special-tests">Special tests</a></li>
29 </ul></li>
30 <li><a href="#test-results-and-summary">Test results and summary</a></li>
31 <li><a href="#test-suite-control">Test suite control</a><ul>
32 <li><a href="#general-keywords-test_opts">General keywords (TEST_OPTS)</a></li>
33 <li><a href="#jtreg-keywords">JTReg keywords</a></li>
34 <li><a href="#gtest-keywords">Gtest keywords</a></li>
35 <li><a href="#microbenchmark-keywords">Microbenchmark keywords</a></li>
36 </ul></li>
37 </ul>
38 </nav>
39 <h2 id="using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</h2>
40 <p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
41 <p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
42 <p>Previously, <code>make test</code> was used to invoke an old system for running tests, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases.</p>
43 <p>Some example command-lines:</p>
44 <pre><code>$ make test-tier1
45 $ make test-jdk_lang JTREG="JOBS=8"
46 $ make test TEST=jdk_lang
47 $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
48 $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
49 $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
50 $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
51 $ make exploded-test TEST=tier2</code></pre>
52 <h3 id="configuration">Configuration</h3>
53 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
54 <p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=<path to JMH jars></code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p>
55 <h2 id="test-selection">Test selection</h2>
56 <p>All functionality is available using the <code>test</code> make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, <code>test-only</code> can be used instead, which do not depend on the source and test image build.</p>
57 <p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make test-tier1</code> is equivalent to <code>make test TEST="tier1"</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>test TEST="x"</code> solution needs to be used.</p>
58 <p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
59 <h3 id="jtreg">JTReg</h3>
60 <p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p>
61 <p>JTReg test groups can be specified either without a test root, e.g. <code>:tier1</code> (or <code>tier1</code>, the initial colon is optional), or with, e.g. <code>hotspot:tier1</code>, <code>test/jdk:jdk_util</code> or <code>$(TOPDIR)/test/hotspot/jtreg:hotspot_all</code>. The test root can be specified either as an absolute path, or a path relative to the JDK top directory, or the <code>test</code> directory. For simplicity, the hotspot JTReg test root, which really is <code>hotspot/jtreg</code> can be abbreviated as just <code>hotspot</code>.</p>
62 <p>When specified without a test root, all matching groups from all test roots will be added. Otherwise, only the group from the specified test root will be added.</p>
|