< prev index next >

src/hotspot/share/runtime/synchronizer.cpp

Print this page
rev 56634 : imported patch 8230876.patch
rev 56635 : v2.00 -> v2.05 (CR5/v2.05/8-for-jdk13) patches combined into one; merge with 8229212.patch; merge with jdk-14+11; merge with 8230184.patch; merge with 8230876.patch; merge with jdk-14+15; merge with jdk-14+18.
rev 56636 : renames, comment cleanups and additions, whitespace and indent fixes; add PaddedObjectMonitor typdef to make 'PaddedEnd<ObjectMonitor' cleanups easier; add a couple of missing 'private' decls; delete unused next() function; merge pieces from dcubed.monitor_deflate_conc.v2.06d in dcubed.monitor_deflate_conc.v2.06[ac]; merge with 8229212.patch; merge with jdk-14+11; merge with 8230184.patch.
rev 56637 : Add OM_CACHE_LINE_SIZE so that ObjectMonitor cache line sizes can be experimented with independently of DEFAULT_CACHE_LINE_SIZE; for SPARC and X64 configs that use 128 for DEFAULT_CACHE_LINE_SIZE, we are experimenting with 64; move _previous_owner_tid and _allocation_state fields to share the cache line with ObjectMonitor::_header; put ObjectMonitor::_ref_count on its own cache line after _owner; add 'int* count_p' parameter to deflate_monitor_list() and deflate_monitor_list_using_JT() and push counter updates down to where the ObjectMonitors are actually removed from the in-use lists; monitors_iterate() async deflation check should use negative ref_count; add 'JavaThread* target' param to deflate_per_thread_idle_monitors_using_JT() add deflate_common_idle_monitors_using_JT() to make it clear which JavaThread* is the target of the work and which is the calling JavaThread* (self); g_free_list, g_om_in_use_list and g_om_in_use_count are now static to synchronizer.cpp (reduce scope); add more diagnostic info to some assert()'s; minor code cleanups and code motion; save_om_ptr() should detect a race with a deflating thread that is bailing out and cause a retry when the ref_count field is not positive; merge with jdk-14+11; add special GC support for TestHumongousClassLoader.java; merge with 8230184.patch; merge with jdk-14+14; merge with jdk-14+18.
rev 56638 : Merge the remainder of the lock-free monitor list changes from v2.06 with v2.06a and v2.06b after running the changes through the edit scripts; merge pieces from dcubed.monitor_deflate_conc.v2.06d in dcubed.monitor_deflate_conc.v2.06[ac]; merge pieces from dcubed.monitor_deflate_conc.v2.06e into dcubed.monitor_deflate_conc.v2.06c; merge with jdk-14+11; test work around for test/jdk/tools/jlink/multireleasejar/JLinkMultiReleaseJarTest.java should not been needed anymore; merge with jdk-14+18.
rev 56639 : loosen a couple more counter checks due to races observed in testing; simplify om_release() extraction of mid since list head or cur_mid_in_use is marked; simplify deflate_monitor_list() extraction of mid since there are no parallel deleters due to the safepoint; simplify deflate_monitor_list_using_JT() extraction of mid since list head or cur_mid_in_use is marked; prepend_block_to_lists() - simplify based on David H's comments; does not need load_acquire() or release_store() because of the cmpxchg(); prepend_to_common() - simplify to use mark_next_loop() for m and use mark_list_head() and release_store() for the non-empty list case; add more debugging for "Non-balanced monitor enter/exit" failure mode; fix race in inflate() in the "CASE: neutral" code path; install_displaced_markword_in_object() does not need to clear the header field since that is handled when the ObjectMonitor is moved from the global free list; LSuccess should clear boxReg to set ICC.ZF=1 to avoid depending on existing boxReg contents; update fast_unlock() to detect when object no longer refers to the same ObjectMonitor and take fast path exit instead; clarify fast_lock() code where we detect when object no longer refers to the same ObjectMonitor; add/update comments for movptr() calls where we move a literal into an Address; remove set_owner(); refactor setting of owner field into set_owner_from(2 versions), set_owner_from_BasicLock(), and try_set_owner_from(); the new functions include monitorinflation+owner logging; extract debug code from v2.06 and v2.07 and move to v2.07.debug; change 'jccb' -> 'jcc' and 'jmpb' -> 'jmp' as needed; checkpoint initial version of MacroAssembler::inc_om_ref_count(); update LP64 MacroAssembler::fast_lock() and fast_unlock() to use inc_om_ref_count(); fast_lock() return flag setting logic can use 'testptr(tmpReg, tmpReg)' instead of 'cmpptr(tmpReg, 0)' since that's more efficient; fast_unlock() LSuccess return flag setting logic can use 'testl (boxReg, 0)' instead of 'xorptr(boxReg, boxReg)' since that's more efficient; cleanup "fast-path" vs "fast path" and "slow-path" vs "slow path"; update MacroAssembler::rtm_inflated_locking() to use inc_om_ref_count(); update MacroAssembler::fast_lock() to preserve the flags before decrementing ref_count and restore the flags afterwards; this is more clean than depending on the contents of rax/tmpReg; coleenp CR - refactor async monitor deflation work from ServiceThread::service_thread_entry() to ObjectSynchronizer::deflate_idle_monitors_using_JT(); rehn,eosterlund CR - add support for HandshakeAfterDeflateIdleMonitors for platforms that don't have ObjectMonitor ref_count support implemented in C2 fast_lock() and fast_unlock().


  20  * or visit www.oracle.com if you need additional information or have any
  21  * questions.
  22  *
  23  */
  24 
  25 #include "precompiled.hpp"
  26 #include "classfile/vmSymbols.hpp"
  27 #include "logging/log.hpp"
  28 #include "logging/logStream.hpp"
  29 #include "jfr/jfrEvents.hpp"
  30 #include "memory/allocation.inline.hpp"
  31 #include "memory/metaspaceShared.hpp"
  32 #include "memory/padded.hpp"
  33 #include "memory/resourceArea.hpp"
  34 #include "memory/universe.hpp"
  35 #include "oops/markWord.hpp"
  36 #include "oops/oop.inline.hpp"
  37 #include "runtime/atomic.hpp"
  38 #include "runtime/biasedLocking.hpp"
  39 #include "runtime/handles.inline.hpp"

  40 #include "runtime/interfaceSupport.inline.hpp"
  41 #include "runtime/mutexLocker.hpp"
  42 #include "runtime/objectMonitor.hpp"
  43 #include "runtime/objectMonitor.inline.hpp"
  44 #include "runtime/osThread.hpp"
  45 #include "runtime/safepointVerifiers.hpp"
  46 #include "runtime/sharedRuntime.hpp"
  47 #include "runtime/stubRoutines.hpp"
  48 #include "runtime/synchronizer.hpp"
  49 #include "runtime/thread.inline.hpp"
  50 #include "runtime/timer.hpp"
  51 #include "runtime/vframe.hpp"
  52 #include "runtime/vmThread.hpp"
  53 #include "utilities/align.hpp"
  54 #include "utilities/dtrace.hpp"
  55 #include "utilities/events.hpp"
  56 #include "utilities/preserveException.hpp"
  57 
  58 // The "core" versions of monitor enter and exit reside in this file.
  59 // The interpreter and compilers contain specialized transliterated


 101   }
 102 
 103 #else //  ndef DTRACE_ENABLED
 104 
 105 #define DTRACE_MONITOR_WAIT_PROBE(obj, thread, millis, mon)    {;}
 106 #define DTRACE_MONITOR_PROBE(probe, obj, thread, mon)          {;}
 107 
 108 #endif // ndef DTRACE_ENABLED
 109 
 110 // This exists only as a workaround of dtrace bug 6254741
 111 int dtrace_waited_probe(ObjectMonitor* monitor, Handle obj, Thread* thr) {
 112   DTRACE_MONITOR_PROBE(waited, monitor, obj(), thr);
 113   return 0;
 114 }
 115 
 116 #define NINFLATIONLOCKS 256
 117 static volatile intptr_t gInflationLocks[NINFLATIONLOCKS];
 118 
 119 // global list of blocks of monitors
 120 PaddedObjectMonitor* volatile ObjectSynchronizer::g_block_list = NULL;




 121 // Global ObjectMonitor free list. Newly allocated and deflated
 122 // ObjectMonitors are prepended here.
 123 ObjectMonitor* volatile ObjectSynchronizer::g_free_list = NULL;
 124 // Global ObjectMonitor in-use list. When a JavaThread is exiting,
 125 // ObjectMonitors on its per-thread in-use list are prepended here.
 126 ObjectMonitor* volatile ObjectSynchronizer::g_om_in_use_list = NULL;
 127 int ObjectSynchronizer::g_om_in_use_count = 0;  // # on g_om_in_use_list





 128 
 129 static volatile intptr_t gListLock = 0;   // protects global monitor lists
 130 static volatile int g_om_free_count = 0;  // # on g_free_list

 131 static volatile int g_om_population = 0;  // # Extant -- in circulation

 132 
 133 #define CHAINMARKER (cast_to_oop<intptr_t>(-1))
 134 
 135 














































































































































































































































 136 // =====================> Quick functions
 137 
 138 // The quick_* forms are special fast-path variants used to improve
 139 // performance.  In the simplest case, a "quick_*" implementation could
 140 // simply return false, in which case the caller will perform the necessary
 141 // state transitions and call the slow-path form.
 142 // The fast-path is designed to handle frequently arising cases in an efficient
 143 // manner and is just a degenerate "optimistic" variant of the slow-path.
 144 // returns true  -- to indicate the call was satisfied.
 145 // returns false -- to indicate the call needs the services of the slow-path.
 146 // A no-loitering ordinance is in effect for code in the quick_* family
 147 // operators: safepoints or indefinite blocking (blocking that might span a
 148 // safepoint) are forbidden. Generally the thread_state() is _in_Java upon
 149 // entry.
 150 //
 151 // Consider: An interesting optimization is to have the JIT recognize the
 152 // following common idiom:
 153 //   synchronized (someobj) { .... ; notify(); }
 154 // That is, we find a notify() or notifyAll() call that immediately precedes
 155 // the monitorexit operation.  In that case the JIT could fuse the operations


 194   }
 195 
 196   // biased locking and any other IMS exception states take the slow-path
 197   return false;
 198 }
 199 
 200 
 201 // The LockNode emitted directly at the synchronization site would have
 202 // been too big if it were to have included support for the cases of inflated
 203 // recursive enter and exit, so they go here instead.
 204 // Note that we can't safely call AsyncPrintJavaStack() from within
 205 // quick_enter() as our thread state remains _in_Java.
 206 
 207 bool ObjectSynchronizer::quick_enter(oop obj, Thread* self,
 208                                      BasicLock * lock) {
 209   assert(!SafepointSynchronize::is_at_safepoint(), "invariant");
 210   assert(self->is_Java_thread(), "invariant");
 211   assert(((JavaThread *) self)->thread_state() == _thread_in_Java, "invariant");
 212   NoSafepointVerifier nsv;
 213   if (obj == NULL) return false;       // Need to throw NPE


 214   const markWord mark = obj->mark();
 215 
 216   if (mark.has_monitor()) {
 217     ObjectMonitor* const m = mark.monitor();






 218     assert(m->object() == obj, "invariant");
 219     Thread* const owner = (Thread *) m->_owner;
 220 
 221     // Lock contention and Transactional Lock Elision (TLE) diagnostics
 222     // and observability
 223     // Case: light contention possibly amenable to TLE
 224     // Case: TLE inimical operations such as nested/recursive synchronization
 225 
 226     if (owner == self) {
 227       m->_recursions++;
 228       return true;
 229     }
 230 
 231     // This Java Monitor is inflated so obj's header will never be
 232     // displaced to this thread's BasicLock. Make the displaced header
 233     // non-NULL so this BasicLock is not seen as recursive nor as
 234     // being locked. We do this unconditionally so that this thread's
 235     // BasicLock cannot be mis-interpreted by any stack walkers. For
 236     // performance reasons, stack walkers generally first check for
 237     // Biased Locking in the object's header, the second check is for
 238     // stack-locking in the object's header, the third check is for
 239     // recursive stack-locking in the displaced header in the BasicLock,
 240     // and last are the inflated Java Monitor (ObjectMonitor) checks.
 241     lock->set_displaced_header(markWord::unused_mark());
 242 
 243     if (owner == NULL && Atomic::replace_if_null(self, &(m->_owner))) {










 244       assert(m->_recursions == 0, "invariant");
 245       return true;
 246     }
 247   }


 248 
 249   // Note that we could inflate in quick_enter.
 250   // This is likely a useful optimization
 251   // Critically, in quick_enter() we must not:
 252   // -- perform bias revocation, or
 253   // -- block indefinitely, or
 254   // -- reach a safepoint
 255 
 256   return false;        // revert to slow-path
 257 }
 258 
 259 // -----------------------------------------------------------------------------
 260 // Monitor Enter/Exit
 261 // The interpreter and compiler assembly code tries to lock using the fast path
 262 // of this algorithm. Make sure to update that code if the following function is
 263 // changed. The implementation is extremely sensitive to race condition. Be careful.
 264 
 265 void ObjectSynchronizer::enter(Handle obj, BasicLock* lock, TRAPS) {
 266   if (UseBiasedLocking) {
 267     if (!SafepointSynchronize::is_at_safepoint()) {


 278     // Anticipate successful CAS -- the ST of the displaced mark must
 279     // be visible <= the ST performed by the CAS.
 280     lock->set_displaced_header(mark);
 281     if (mark == obj()->cas_set_mark(markWord::from_pointer(lock), mark)) {
 282       return;
 283     }
 284     // Fall through to inflate() ...
 285   } else if (mark.has_locker() &&
 286              THREAD->is_lock_owned((address)mark.locker())) {
 287     assert(lock != mark.locker(), "must not re-lock the same lock");
 288     assert(lock != (BasicLock*)obj->mark().value(), "don't relock with same BasicLock");
 289     lock->set_displaced_header(markWord::from_pointer(NULL));
 290     return;
 291   }
 292 
 293   // The object header will never be displaced to this lock,
 294   // so it does not matter what the value is, except that it
 295   // must be non-zero to avoid looking like a re-entrant lock,
 296   // and must not look locked either.
 297   lock->set_displaced_header(markWord::unused_mark());
 298   inflate(THREAD, obj(), inflate_cause_monitor_enter)->enter(THREAD);


 299 }
 300 
 301 void ObjectSynchronizer::exit(oop object, BasicLock* lock, TRAPS) {
 302   markWord mark = object->mark();
 303   // We cannot check for Biased Locking if we are racing an inflation.
 304   assert(mark == markWord::INFLATING() ||
 305          !mark.has_bias_pattern(), "should not see bias pattern here");
 306 
 307   markWord dhw = lock->displaced_header();
 308   if (dhw.value() == 0) {
 309     // If the displaced header is NULL, then this exit matches up with
 310     // a recursive enter. No real work to do here except for diagnostics.
 311 #ifndef PRODUCT
 312     if (mark != markWord::INFLATING()) {
 313       // Only do diagnostics if we are not racing an inflation. Simply
 314       // exiting a recursive enter of a Java Monitor that is being
 315       // inflated is safe; see the has_monitor() comment below.
 316       assert(!mark.is_neutral(), "invariant");
 317       assert(!mark.has_locker() ||
 318              THREAD->is_lock_owned((address)mark.locker()), "invariant");


 327         // does not own the Java Monitor.
 328         ObjectMonitor* m = mark.monitor();
 329         assert(((oop)(m->object()))->mark() == mark, "invariant");
 330         assert(m->is_entered(THREAD), "invariant");
 331       }
 332     }
 333 #endif
 334     return;
 335   }
 336 
 337   if (mark == markWord::from_pointer(lock)) {
 338     // If the object is stack-locked by the current thread, try to
 339     // swing the displaced header from the BasicLock back to the mark.
 340     assert(dhw.is_neutral(), "invariant");
 341     if (object->cas_set_mark(dhw, mark) == mark) {
 342       return;
 343     }
 344   }
 345 
 346   // We have to take the slow-path of possible inflation and then exit.
 347   inflate(THREAD, object, inflate_cause_vm_internal)->exit(true, THREAD);


 348 }
 349 
 350 // -----------------------------------------------------------------------------
 351 // Class Loader  support to workaround deadlocks on the class loader lock objects
 352 // Also used by GC
 353 // complete_exit()/reenter() are used to wait on a nested lock
 354 // i.e. to give up an outer lock completely and then re-enter
 355 // Used when holding nested locks - lock acquisition order: lock1 then lock2
 356 //  1) complete_exit lock1 - saving recursion count
 357 //  2) wait on lock2
 358 //  3) when notified on lock2, unlock lock2
 359 //  4) reenter lock1 with original recursion count
 360 //  5) lock lock2
 361 // NOTE: must use heavy weight monitor to handle complete_exit/reenter()
 362 intptr_t ObjectSynchronizer::complete_exit(Handle obj, TRAPS) {
 363   if (UseBiasedLocking) {
 364     BiasedLocking::revoke(obj, THREAD);
 365     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 366   }
 367 
 368   ObjectMonitor* monitor = inflate(THREAD, obj(), inflate_cause_vm_internal);
 369 
 370   return monitor->complete_exit(THREAD);

 371 }
 372 
 373 // NOTE: must use heavy weight monitor to handle complete_exit/reenter()
 374 void ObjectSynchronizer::reenter(Handle obj, intptr_t recursion, TRAPS) {
 375   if (UseBiasedLocking) {
 376     BiasedLocking::revoke(obj, THREAD);
 377     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 378   }
 379 
 380   ObjectMonitor* monitor = inflate(THREAD, obj(), inflate_cause_vm_internal);
 381 
 382   monitor->reenter(recursion, THREAD);
 383 }
 384 // -----------------------------------------------------------------------------
 385 // JNI locks on java objects
 386 // NOTE: must use heavy weight monitor to handle jni monitor enter
 387 void ObjectSynchronizer::jni_enter(Handle obj, TRAPS) {
 388   // the current locking is from JNI instead of Java code
 389   if (UseBiasedLocking) {
 390     BiasedLocking::revoke(obj, THREAD);
 391     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 392   }
 393   THREAD->set_current_pending_monitor_is_from_java(false);
 394   inflate(THREAD, obj(), inflate_cause_jni_enter)->enter(THREAD);


 395   THREAD->set_current_pending_monitor_is_from_java(true);
 396 }
 397 
 398 // NOTE: must use heavy weight monitor to handle jni monitor exit
 399 void ObjectSynchronizer::jni_exit(oop obj, Thread* THREAD) {
 400   if (UseBiasedLocking) {
 401     Handle h_obj(THREAD, obj);
 402     BiasedLocking::revoke(h_obj, THREAD);
 403     obj = h_obj();
 404   }
 405   assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 406 
 407   ObjectMonitor* monitor = inflate(THREAD, obj, inflate_cause_jni_exit);


 408   // If this thread has locked the object, exit the monitor. We
 409   // intentionally do not use CHECK here because we must exit the
 410   // monitor even if an exception is pending.
 411   if (monitor->check_owner(THREAD)) {
 412     monitor->exit(true, THREAD);
 413   }
 414 }
 415 
 416 // -----------------------------------------------------------------------------
 417 // Internal VM locks on java objects
 418 // standard constructor, allows locking failures
 419 ObjectLocker::ObjectLocker(Handle obj, Thread* thread, bool do_lock) {
 420   _dolock = do_lock;
 421   _thread = thread;
 422   _thread->check_for_valid_safepoint_state();
 423   _obj = obj;
 424 
 425   if (_dolock) {
 426     ObjectSynchronizer::enter(_obj, &_lock, _thread);
 427   }
 428 }
 429 
 430 ObjectLocker::~ObjectLocker() {
 431   if (_dolock) {
 432     ObjectSynchronizer::exit(_obj(), &_lock, _thread);
 433   }
 434 }
 435 
 436 
 437 // -----------------------------------------------------------------------------
 438 //  Wait/Notify/NotifyAll
 439 // NOTE: must use heavy weight monitor to handle wait()
 440 int ObjectSynchronizer::wait(Handle obj, jlong millis, TRAPS) {
 441   if (UseBiasedLocking) {
 442     BiasedLocking::revoke(obj, THREAD);
 443     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 444   }
 445   if (millis < 0) {
 446     THROW_MSG_0(vmSymbols::java_lang_IllegalArgumentException(), "timeout value is negative");
 447   }
 448   ObjectMonitor* monitor = inflate(THREAD, obj(), inflate_cause_wait);


 449 
 450   DTRACE_MONITOR_WAIT_PROBE(monitor, obj(), THREAD, millis);
 451   monitor->wait(millis, true, THREAD);
 452 
 453   // This dummy call is in place to get around dtrace bug 6254741.  Once
 454   // that's fixed we can uncomment the following line, remove the call
 455   // and change this function back into a "void" func.
 456   // DTRACE_MONITOR_PROBE(waited, monitor, obj(), THREAD);
 457   return dtrace_waited_probe(monitor, obj, THREAD);

 458 }
 459 
 460 void ObjectSynchronizer::wait_uninterruptibly(Handle obj, jlong millis, TRAPS) {
 461   if (UseBiasedLocking) {
 462     BiasedLocking::revoke(obj, THREAD);
 463     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 464   }
 465   if (millis < 0) {
 466     THROW_MSG(vmSymbols::java_lang_IllegalArgumentException(), "timeout value is negative");
 467   }
 468   inflate(THREAD, obj(), inflate_cause_wait)->wait(millis, false, THREAD);


 469 }
 470 
 471 void ObjectSynchronizer::notify(Handle obj, TRAPS) {
 472   if (UseBiasedLocking) {
 473     BiasedLocking::revoke(obj, THREAD);
 474     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 475   }
 476 
 477   markWord mark = obj->mark();
 478   if (mark.has_locker() && THREAD->is_lock_owned((address)mark.locker())) {
 479     return;
 480   }
 481   inflate(THREAD, obj(), inflate_cause_notify)->notify(THREAD);


 482 }
 483 
 484 // NOTE: see comment of notify()
 485 void ObjectSynchronizer::notifyall(Handle obj, TRAPS) {
 486   if (UseBiasedLocking) {
 487     BiasedLocking::revoke(obj, THREAD);
 488     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 489   }
 490 
 491   markWord mark = obj->mark();
 492   if (mark.has_locker() && THREAD->is_lock_owned((address)mark.locker())) {
 493     return;
 494   }
 495   inflate(THREAD, obj(), inflate_cause_notify)->notifyAll(THREAD);


 496 }
 497 
 498 // -----------------------------------------------------------------------------
 499 // Hash Code handling
 500 //
 501 // Performance concern:
 502 // OrderAccess::storestore() calls release() which at one time stored 0
 503 // into the global volatile OrderAccess::dummy variable. This store was
 504 // unnecessary for correctness. Many threads storing into a common location
 505 // causes considerable cache migration or "sloshing" on large SMP systems.
 506 // As such, I avoided using OrderAccess::storestore(). In some cases
 507 // OrderAccess::fence() -- which incurs local latency on the executing
 508 // processor -- is a better choice as it scales on SMP systems.
 509 //
 510 // See http://blogs.oracle.com/dave/entry/biased_locking_in_hotspot for
 511 // a discussion of coherency costs. Note that all our current reference
 512 // platforms provide strong ST-ST order, so the issue is moot on IA32,
 513 // x64, and SPARC.
 514 //
 515 // As a general policy we use "volatile" to control compiler-based reordering
 516 // and explicit fences (barriers) to control for architectural reordering
 517 // performed by the CPU(s) or platform.
 518 
 519 struct SharedGlobals {
 520   char         _pad_prefix[DEFAULT_CACHE_LINE_SIZE];
 521   // These are highly shared mostly-read variables.
 522   // To avoid false-sharing they need to be the sole occupants of a cache line.
 523   volatile int stw_random;
 524   volatile int stw_cycle;
 525   DEFINE_PAD_MINUS_SIZE(1, DEFAULT_CACHE_LINE_SIZE, sizeof(volatile int) * 2);
 526   // Hot RW variable -- Sequester to avoid false-sharing
 527   volatile int hc_sequence;
 528   DEFINE_PAD_MINUS_SIZE(2, DEFAULT_CACHE_LINE_SIZE, sizeof(volatile int));
 529 };
 530 
 531 static SharedGlobals GVars;
 532 static int MonitorScavengeThreshold = 1000000;
 533 static volatile int ForceMonitorScavenge = 0; // Scavenge required and pending
 534 
 535 static markWord read_stable_mark(oop obj) {
 536   markWord mark = obj->mark();
 537   if (!mark.is_being_inflated()) {
 538     return mark;       // normal fast-path return
 539   }
 540 
 541   int its = 0;
 542   for (;;) {
 543     markWord mark = obj->mark();
 544     if (!mark.is_being_inflated()) {
 545       return mark;    // normal fast-path return
 546     }
 547 
 548     // The object is being inflated by some other thread.


 669       Handle hobj(self, obj);
 670       // Relaxing assertion for bug 6320749.
 671       assert(Universe::verify_in_progress() ||
 672              !SafepointSynchronize::is_at_safepoint(),
 673              "biases should not be seen by VM thread here");
 674       BiasedLocking::revoke(hobj, JavaThread::current());
 675       obj = hobj();
 676       assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 677     }
 678   }
 679 
 680   // hashCode() is a heap mutator ...
 681   // Relaxing assertion for bug 6320749.
 682   assert(Universe::verify_in_progress() || DumpSharedSpaces ||
 683          !SafepointSynchronize::is_at_safepoint(), "invariant");
 684   assert(Universe::verify_in_progress() || DumpSharedSpaces ||
 685          self->is_Java_thread() , "invariant");
 686   assert(Universe::verify_in_progress() || DumpSharedSpaces ||
 687          ((JavaThread *)self)->thread_state() != _thread_blocked, "invariant");
 688 

 689   ObjectMonitor* monitor = NULL;
 690   markWord temp, test;
 691   intptr_t hash;
 692   markWord mark = read_stable_mark(obj);
 693 
 694   // object should remain ineligible for biased locking
 695   assert(!mark.has_bias_pattern(), "invariant");
 696 
 697   if (mark.is_neutral()) {
 698     hash = mark.hash();               // this is a normal header
 699     if (hash != 0) {                  // if it has hash, just return it
 700       return hash;
 701     }
 702     hash = get_next_hash(self, obj);  // allocate a new hash code
 703     temp = mark.copy_set_hash(hash);  // merge the hash code into header
 704     // use (machine word version) atomic operation to install the hash
 705     test = obj->cas_set_mark(temp, mark);
 706     if (test == mark) {
 707       return hash;
 708     }
 709     // If atomic operation failed, we must inflate the header
 710     // into heavy weight monitor. We could add more code here
 711     // for fast path, but it does not worth the complexity.
 712   } else if (mark.has_monitor()) {
 713     monitor = mark.monitor();






 714     temp = monitor->header();
 715     assert(temp.is_neutral(), "invariant: header=" INTPTR_FORMAT, temp.value());
 716     hash = temp.hash();
 717     if (hash != 0) {
 718       return hash;
 719     }
 720     // Skip to the following code to reduce code size
 721   } else if (self->is_lock_owned((address)mark.locker())) {
 722     temp = mark.displaced_mark_helper(); // this is a lightweight monitor owned
 723     assert(temp.is_neutral(), "invariant: header=" INTPTR_FORMAT, temp.value());
 724     hash = temp.hash();                  // by current thread, check if the displaced
 725     if (hash != 0) {                     // header contains hash code
 726       return hash;
 727     }
 728     // WARNING:
 729     // The displaced header in the BasicLock on a thread's stack
 730     // is strictly immutable. It CANNOT be changed in ANY cases.
 731     // So we have to inflate the stack lock into an ObjectMonitor
 732     // even if the current thread owns the lock. The BasicLock on
 733     // a thread's stack can be asynchronously read by other threads
 734     // during an inflate() call so any change to that stack memory
 735     // may not propagate to other threads correctly.
 736   }
 737 
 738   // Inflate the monitor to set hash code
 739   monitor = inflate(self, obj, inflate_cause_hash_code);


 740   // Load displaced header and check it has hash code
 741   mark = monitor->header();
 742   assert(mark.is_neutral(), "invariant: header=" INTPTR_FORMAT, mark.value());
 743   hash = mark.hash();
 744   if (hash == 0) {
 745     hash = get_next_hash(self, obj);
 746     temp = mark.copy_set_hash(hash); // merge hash code into header
 747     assert(temp.is_neutral(), "invariant: header=" INTPTR_FORMAT, temp.value());
 748     uintptr_t v = Atomic::cmpxchg(temp.value(), (volatile uintptr_t*)monitor->header_addr(), mark.value());
 749     test = markWord(v);
 750     if (test != mark) {
 751       // The only non-deflation update to the ObjectMonitor's
 752       // header/dmw field is to merge in the hash code. If someone
 753       // adds a new usage of the header/dmw field, please update
 754       // this code.




 755       hash = test.hash();
 756       assert(test.is_neutral(), "invariant: header=" INTPTR_FORMAT, test.value());
 757       assert(hash != 0, "Trivial unexpected object/monitor header usage.");
 758     }
 759   }
 760   // We finally get the hash
 761   return hash;

 762 }
 763 
 764 // Deprecated -- use FastHashCode() instead.
 765 
 766 intptr_t ObjectSynchronizer::identity_hash_value_for(Handle obj) {
 767   return FastHashCode(Thread::current(), obj());
 768 }
 769 
 770 
 771 bool ObjectSynchronizer::current_thread_holds_lock(JavaThread* thread,
 772                                                    Handle h_obj) {
 773   if (UseBiasedLocking) {
 774     BiasedLocking::revoke(h_obj, thread);
 775     assert(!h_obj->mark().has_bias_pattern(), "biases should be revoked by now");
 776   }
 777 
 778   assert(thread == JavaThread::current(), "Can only be called on current thread");
 779   oop obj = h_obj();
 780 

 781   markWord mark = read_stable_mark(obj);
 782 
 783   // Uncontended case, header points to stack
 784   if (mark.has_locker()) {
 785     return thread->is_lock_owned((address)mark.locker());
 786   }
 787   // Contended case, header points to ObjectMonitor (tagged pointer)
 788   if (mark.has_monitor()) {
 789     ObjectMonitor* monitor = mark.monitor();
 790     return monitor->is_entered(thread) != 0;






 791   }
 792   // Unlocked case, header in place
 793   assert(mark.is_neutral(), "sanity check");
 794   return false;

 795 }
 796 
 797 // Be aware of this method could revoke bias of the lock object.
 798 // This method queries the ownership of the lock handle specified by 'h_obj'.
 799 // If the current thread owns the lock, it returns owner_self. If no
 800 // thread owns the lock, it returns owner_none. Otherwise, it will return
 801 // owner_other.
 802 ObjectSynchronizer::LockOwnership ObjectSynchronizer::query_lock_ownership
 803 (JavaThread *self, Handle h_obj) {
 804   // The caller must beware this method can revoke bias, and
 805   // revocation can result in a safepoint.
 806   assert(!SafepointSynchronize::is_at_safepoint(), "invariant");
 807   assert(self->thread_state() != _thread_blocked, "invariant");
 808 
 809   // Possible mark states: neutral, biased, stack-locked, inflated
 810 
 811   if (UseBiasedLocking && h_obj()->mark().has_bias_pattern()) {
 812     // CASE: biased
 813     BiasedLocking::revoke(h_obj, self);
 814     assert(!h_obj->mark().has_bias_pattern(),
 815            "biases should be revoked by now");
 816   }
 817 
 818   assert(self == JavaThread::current(), "Can only be called on current thread");
 819   oop obj = h_obj();


 820   markWord mark = read_stable_mark(obj);
 821 
 822   // CASE: stack-locked.  Mark points to a BasicLock on the owner's stack.
 823   if (mark.has_locker()) {
 824     return self->is_lock_owned((address)mark.locker()) ?
 825       owner_self : owner_other;
 826   }
 827 
 828   // CASE: inflated. Mark (tagged pointer) points to an ObjectMonitor.
 829   // The Object:ObjectMonitor relationship is stable as long as we're
 830   // not at a safepoint.
 831   if (mark.has_monitor()) {
 832     void* owner = mark.monitor()->_owner;







 833     if (owner == NULL) return owner_none;
 834     return (owner == self ||
 835             self->is_lock_owned((address)owner)) ? owner_self : owner_other;
 836   }
 837 
 838   // CASE: neutral
 839   assert(mark.is_neutral(), "sanity check");
 840   return owner_none;           // it's unlocked

 841 }
 842 
 843 // FIXME: jvmti should call this
 844 JavaThread* ObjectSynchronizer::get_lock_owner(ThreadsList * t_list, Handle h_obj) {
 845   if (UseBiasedLocking) {
 846     if (SafepointSynchronize::is_at_safepoint()) {
 847       BiasedLocking::revoke_at_safepoint(h_obj);
 848     } else {
 849       BiasedLocking::revoke(h_obj, JavaThread::current());
 850     }
 851     assert(!h_obj->mark().has_bias_pattern(), "biases should be revoked by now");
 852   }
 853 
 854   oop obj = h_obj();
 855   address owner = NULL;
 856 


 857   markWord mark = read_stable_mark(obj);
 858 
 859   // Uncontended case, header points to stack
 860   if (mark.has_locker()) {
 861     owner = (address) mark.locker();
 862   }
 863 
 864   // Contended case, header points to ObjectMonitor (tagged pointer)
 865   else if (mark.has_monitor()) {
 866     ObjectMonitor* monitor = mark.monitor();






 867     assert(monitor != NULL, "monitor should be non-null");
 868     owner = (address) monitor->owner();
 869   }
 870 
 871   if (owner != NULL) {
 872     // owning_thread_from_monitor_owner() may also return NULL here
 873     return Threads::owning_thread_from_monitor_owner(t_list, owner);
 874   }
 875 
 876   // Unlocked case, header in place
 877   // Cannot have assertion since this object may have been
 878   // locked by another thread when reaching here.
 879   // assert(mark.is_neutral(), "sanity check");
 880 
 881   return NULL;

 882 }
 883 
 884 // Visitors ...
 885 
 886 void ObjectSynchronizer::monitors_iterate(MonitorClosure* closure) {
 887   PaddedObjectMonitor* block = OrderAccess::load_acquire(&g_block_list);
 888   while (block != NULL) {
 889     assert(block->object() == CHAINMARKER, "must be a block header");
 890     for (int i = _BLOCKSIZE - 1; i > 0; i--) {
 891       ObjectMonitor* mid = (ObjectMonitor *)(block + i);
 892       oop object = (oop)mid->object();
 893       if (object != NULL) {



 894         // Only process with closure if the object is set.







 895         closure->do_monitor(mid);
 896       }
 897     }
 898     block = (PaddedObjectMonitor*)block->_next_om;

 899   }
 900 }
 901 
 902 static bool monitors_used_above_threshold() {
 903   if (g_om_population == 0) {
 904     return false;
 905   }
 906   int monitors_used = g_om_population - g_om_free_count;
 907   int monitor_usage = (monitors_used * 100LL) / g_om_population;






 908   return monitor_usage > MonitorUsedDeflationThreshold;


 909 }
 910 
 911 bool ObjectSynchronizer::is_cleanup_needed() {
 912   if (MonitorUsedDeflationThreshold > 0) {
 913     return monitors_used_above_threshold();






























 914   }
 915   return false;
 916 }
 917 




















 918 void ObjectSynchronizer::oops_do(OopClosure* f) {
 919   // We only scan the global used list here (for moribund threads), and
 920   // the thread-local monitors in Thread::oops_do().
 921   global_used_oops_do(f);
 922 }
 923 
 924 void ObjectSynchronizer::global_used_oops_do(OopClosure* f) {
 925   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
 926   list_oops_do(g_om_in_use_list, f);
 927 }
 928 
 929 void ObjectSynchronizer::thread_local_used_oops_do(Thread* thread, OopClosure* f) {
 930   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
 931   list_oops_do(thread->om_in_use_list, f);
 932 }
 933 
 934 void ObjectSynchronizer::list_oops_do(ObjectMonitor* list, OopClosure* f) {
 935   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
 936   ObjectMonitor* mid;
 937   for (mid = list; mid != NULL; mid = mid->_next_om) {


 938     if (mid->object() != NULL) {
 939       f->do_oop((oop*)mid->object_addr());
 940     }
 941   }
 942 }
 943 
 944 
 945 // -----------------------------------------------------------------------------
 946 // ObjectMonitor Lifecycle
 947 // -----------------------
 948 // Inflation unlinks monitors from the global g_free_list and
 949 // associates them with objects.  Deflation -- which occurs at
 950 // STW-time -- disassociates idle monitors from objects.  Such
 951 // scavenged monitors are returned to the g_free_list.
 952 //
 953 // The global list is protected by gListLock.  All the critical sections
 954 // are short and operate in constant-time.
 955 //
 956 // ObjectMonitors reside in type-stable memory (TSM) and are immortal.
 957 //
 958 // Lifecycle:
 959 // --   unassigned and on the global free list
 960 // --   unassigned and on a thread's private om_free_list
 961 // --   assigned to an object.  The object is inflated and the mark refers
 962 //      to the objectmonitor.
 963 
 964 
 965 // Constraining monitor pool growth via MonitorBound ...
 966 //
 967 // If MonitorBound is not set (<= 0), MonitorBound checks are disabled.
 968 //

 969 // The monitor pool is grow-only.  We scavenge at STW safepoint-time, but the
 970 // the rate of scavenging is driven primarily by GC.  As such,  we can find
 971 // an inordinate number of monitors in circulation.
 972 // To avoid that scenario we can artificially induce a STW safepoint
 973 // if the pool appears to be growing past some reasonable bound.
 974 // Generally we favor time in space-time tradeoffs, but as there's no
 975 // natural back-pressure on the # of extant monitors we need to impose some
 976 // type of limit.  Beware that if MonitorBound is set to too low a value
 977 // we could just loop. In addition, if MonitorBound is set to a low value
 978 // we'll incur more safepoints, which are harmful to performance.
 979 // See also: GuaranteedSafepointInterval
 980 //
 981 // The current implementation uses asynchronous VM operations.
 982 //
 983 // If MonitorBound is set, the boundry applies to

 984 //     (g_om_population - g_om_free_count)
 985 // i.e., if there are not enough ObjectMonitors on the global free list,
 986 // then a safepoint deflation is induced. Picking a good MonitorBound value
 987 // is non-trivial.










 988 
 989 static void InduceScavenge(Thread* self, const char * Whence) {


 990   // Induce STW safepoint to trim monitors
 991   // Ultimately, this results in a call to deflate_idle_monitors() in the near future.
 992   // More precisely, trigger an asynchronous STW safepoint as the number
 993   // of active monitors passes the specified threshold.
 994   // TODO: assert thread state is reasonable
 995 
 996   if (ForceMonitorScavenge == 0 && Atomic::xchg (1, &ForceMonitorScavenge) == 0) {
 997     // Induce a 'null' safepoint to scavenge monitors
 998     // Must VM_Operation instance be heap allocated as the op will be enqueue and posted
 999     // to the VMthread and have a lifespan longer than that of this activation record.
1000     // The VMThread will delete the op when completed.
1001     VMThread::execute(new VM_ScavengeMonitors());
1002   }
1003 }
1004 
1005 ObjectMonitor* ObjectSynchronizer::om_alloc(Thread* self) {

1006   // A large MAXPRIVATE value reduces both list lock contention
1007   // and list coherency traffic, but also tends to increase the
1008   // number of ObjectMonitors in circulation as well as the STW
1009   // scavenge costs.  As usual, we lean toward time in space-time
1010   // tradeoffs.
1011   const int MAXPRIVATE = 1024;

1012   stringStream ss;
1013   for (;;) {
1014     ObjectMonitor* m;
1015 
1016     // 1: try to allocate from the thread's local om_free_list.
1017     // Threads will attempt to allocate first from their local list, then
1018     // from the global list, and only after those attempts fail will the thread
1019     // attempt to instantiate new monitors.   Thread-local free lists take
1020     // heat off the gListLock and improve allocation latency, as well as reducing
1021     // coherency traffic on the shared global list.
1022     m = self->om_free_list;
1023     if (m != NULL) {
1024       self->om_free_list = m->_next_om;
1025       self->om_free_count--;
1026       guarantee(m->object() == NULL, "invariant");
1027       m->_next_om = self->om_in_use_list;
1028       self->om_in_use_list = m;
1029       self->om_in_use_count++;
1030       return m;
1031     }
1032 
1033     // 2: try to allocate from the global g_free_list
1034     // CONSIDER: use muxTry() instead of muxAcquire().
1035     // If the muxTry() fails then drop immediately into case 3.
1036     // If we're using thread-local free lists then try
1037     // to reprovision the caller's free list.
1038     if (g_free_list != NULL) {
1039       // Reprovision the thread's om_free_list.
1040       // Use bulk transfers to reduce the allocation rate and heat
1041       // on various locks.
1042       Thread::muxAcquire(&gListLock, "om_alloc(1)");
1043       for (int i = self->om_free_provision; --i >= 0 && g_free_list != NULL;) {
1044         g_om_free_count--;
1045         ObjectMonitor* take = g_free_list;
1046         g_free_list = take->_next_om;
1047         guarantee(take->object() == NULL, "invariant");
















1048         take->Recycle();




1049         om_release(self, take, false);
1050       }
1051       Thread::muxRelease(&gListLock);
1052       self->om_free_provision += 1 + (self->om_free_provision/2);
1053       if (self->om_free_provision > MAXPRIVATE) self->om_free_provision = MAXPRIVATE;
1054 
1055       const int mx = MonitorBound;
1056       if (mx > 0 && (g_om_population-g_om_free_count) > mx) {

1057         // Not enough ObjectMonitors on the global free list.
1058         // We can't safely induce a STW safepoint from om_alloc() as our thread
1059         // state may not be appropriate for such activities and callers may hold
1060         // naked oops, so instead we defer the action.
1061         InduceScavenge(self, "om_alloc");
1062       }
1063       continue;
1064     }
1065 
1066     // 3: allocate a block of new ObjectMonitors
1067     // Both the local and global free lists are empty -- resort to malloc().
1068     // In the current implementation ObjectMonitors are TSM - immortal.
1069     // Ideally, we'd write "new ObjectMonitor[_BLOCKSIZE], but we want
1070     // each ObjectMonitor to start at the beginning of a cache line,
1071     // so we use align_up().
1072     // A better solution would be to use C++ placement-new.
1073     // BEWARE: As it stands currently, we don't run the ctors!
1074     assert(_BLOCKSIZE > 1, "invariant");
1075     size_t neededsize = sizeof(PaddedObjectMonitor) * _BLOCKSIZE;
1076     PaddedObjectMonitor* temp;
1077     size_t aligned_size = neededsize + (DEFAULT_CACHE_LINE_SIZE - 1);
1078     void* real_malloc_addr = NEW_C_HEAP_ARRAY(char, aligned_size, mtInternal);
1079     temp = (PaddedObjectMonitor*)align_up(real_malloc_addr, DEFAULT_CACHE_LINE_SIZE);
1080     (void)memset((void *) temp, 0, neededsize);
1081 
1082     // Format the block.
1083     // initialize the linked list, each monitor points to its next
1084     // forming the single linked free list, the very first monitor
1085     // will points to next block, which forms the block list.
1086     // The trick of using the 1st element in the block as g_block_list
1087     // linkage should be reconsidered.  A better implementation would
1088     // look like: class Block { Block * next; int N; ObjectMonitor Body [N] ; }
1089 
1090     for (int i = 1; i < _BLOCKSIZE; i++) {
1091       temp[i]._next_om = (ObjectMonitor *)&temp[i+1];

1092     }
1093 
1094     // terminate the last monitor as the end of list
1095     temp[_BLOCKSIZE - 1]._next_om = NULL;
1096 
1097     // Element [0] is reserved for global list linkage
1098     temp[0].set_object(CHAINMARKER);
1099 
1100     // Consider carving out this thread's current request from the
1101     // block in hand.  This avoids some lock traffic and redundant
1102     // list activity.
1103 
1104     // Acquire the gListLock to manipulate g_block_list and g_free_list.
1105     // An Oyama-Taura-Yonezawa scheme might be more efficient.
1106     Thread::muxAcquire(&gListLock, "om_alloc(2)");
1107     g_om_population += _BLOCKSIZE-1;
1108     g_om_free_count += _BLOCKSIZE-1;
1109 
1110     // Add the new block to the list of extant blocks (g_block_list).
1111     // The very first ObjectMonitor in a block is reserved and dedicated.
1112     // It serves as blocklist "next" linkage.
1113     temp[0]._next_om = g_block_list;
1114     // There are lock-free uses of g_block_list so make sure that
1115     // the previous stores happen before we update g_block_list.
1116     OrderAccess::release_store(&g_block_list, temp);
1117 
1118     // Add the new string of ObjectMonitors to the global free list
1119     temp[_BLOCKSIZE - 1]._next_om = g_free_list;
1120     g_free_list = temp + 1;
1121     Thread::muxRelease(&gListLock);
1122   }
1123 }
1124 
1125 // Place "m" on the caller's private per-thread om_free_list.
1126 // In practice there's no need to clamp or limit the number of
1127 // monitors on a thread's om_free_list as the only non-allocation time
1128 // we'll call om_release() is to return a monitor to the free list after
1129 // a CAS attempt failed. This doesn't allow unbounded #s of monitors to
1130 // accumulate on a thread's free list.
1131 //
1132 // Key constraint: all ObjectMonitors on a thread's free list and the global
1133 // free list must have their object field set to null. This prevents the
1134 // scavenger -- deflate_monitor_list() -- from reclaiming them while we
1135 // are trying to release them.
1136 
1137 void ObjectSynchronizer::om_release(Thread* self, ObjectMonitor* m,
1138                                     bool from_per_thread_alloc) {
1139   guarantee(m->header().value() == 0, "invariant");
1140   guarantee(m->object() == NULL, "invariant");
1141   stringStream ss;
1142   guarantee((m->is_busy() | m->_recursions) == 0, "freeing in-use monitor: "
1143             "%s, recursions=" INTPTR_FORMAT, m->is_busy_to_string(&ss),
1144             m->_recursions);

1145   // _next_om is used for both per-thread in-use and free lists so
1146   // we have to remove 'm' from the in-use list first (as needed).
1147   if (from_per_thread_alloc) {
1148     // Need to remove 'm' from om_in_use_list.


1149     ObjectMonitor* cur_mid_in_use = NULL;


1150     bool extracted = false;
1151     for (ObjectMonitor* mid = self->om_in_use_list; mid != NULL; cur_mid_in_use = mid, mid = mid->_next_om) {




1152       if (m == mid) {
1153         // extract from per-thread in-use list
1154         if (mid == self->om_in_use_list) {
1155           self->om_in_use_list = mid->_next_om;
1156         } else if (cur_mid_in_use != NULL) {
1157           cur_mid_in_use->_next_om = mid->_next_om; // maintain the current thread in-use list





1158         }
1159         extracted = true;
1160         self->om_in_use_count--;




1161         break;
1162       }















1163     }
1164     assert(extracted, "Should have extracted from in-use list");
1165   }
1166 
1167   m->_next_om = self->om_free_list;
1168   self->om_free_list = m;
1169   self->om_free_count++;
1170 }
1171 
1172 // Return ObjectMonitors on a moribund thread's free and in-use
1173 // lists to the appropriate global lists. The ObjectMonitors on the
1174 // per-thread in-use list may still be in use by other threads.
1175 //
1176 // We currently call om_flush() from Threads::remove() before the
1177 // thread has been excised from the thread list and is no longer a
1178 // mutator. This means that om_flush() cannot run concurrently with
1179 // a safepoint and interleave with deflate_idle_monitors(). In
1180 // particular, this ensures that the thread's in-use monitors are
1181 // scanned by a GC safepoint, either via Thread::oops_do() (before
1182 // om_flush() is called) or via ObjectSynchronizer::oops_do() (after
1183 // om_flush() is called).





1184 
1185 void ObjectSynchronizer::om_flush(Thread* self) {
1186   ObjectMonitor* free_list = self->om_free_list;
1187   ObjectMonitor* free_tail = NULL;































































1188   int free_count = 0;


1189   if (free_list != NULL) {
1190     ObjectMonitor* s;
1191     // The thread is going away. Set 'free_tail' to the last per-thread free
1192     // monitor which will be linked to g_free_list below under the gListLock.
1193     stringStream ss;
1194     for (s = free_list; s != NULL; s = s->_next_om) {
1195       free_count++;
1196       free_tail = s;
1197       guarantee(s->object() == NULL, "invariant");
1198       guarantee(!s->is_busy(), "must be !is_busy: %s", s->is_busy_to_string(&ss));
1199     }
1200     guarantee(free_tail != NULL, "invariant");
1201     assert(self->om_free_count == free_count, "free-count off");
1202     self->om_free_list = NULL;
1203     self->om_free_count = 0;



1204   }
1205 
1206   ObjectMonitor* in_use_list = self->om_in_use_list;
1207   ObjectMonitor* in_use_tail = NULL;
1208   int in_use_count = 0;
1209   if (in_use_list != NULL) {
1210     // The thread is going away, however the ObjectMonitors on the
1211     // om_in_use_list may still be in-use by other threads. Link
1212     // them to in_use_tail, which will be linked into the global
1213     // in-use list g_om_in_use_list below, under the gListLock.
1214     ObjectMonitor *cur_om;
1215     for (cur_om = in_use_list; cur_om != NULL; cur_om = cur_om->_next_om) {
1216       in_use_tail = cur_om;
1217       in_use_count++;
1218     }
1219     guarantee(in_use_tail != NULL, "invariant");
1220     assert(self->om_in_use_count == in_use_count, "in-use count off");
1221     self->om_in_use_list = NULL;
1222     self->om_in_use_count = 0;
1223   }
1224 
1225   Thread::muxAcquire(&gListLock, "om_flush");
1226   if (free_tail != NULL) {
1227     free_tail->_next_om = g_free_list;
1228     g_free_list = free_list;
1229     g_om_free_count += free_count;
1230   }
1231 
1232   if (in_use_tail != NULL) {
1233     in_use_tail->_next_om = g_om_in_use_list;
1234     g_om_in_use_list = in_use_list;
1235     g_om_in_use_count += in_use_count;
1236   }
1237 
1238   Thread::muxRelease(&gListLock);
1239 
1240   LogStreamHandle(Debug, monitorinflation) lsh_debug;
1241   LogStreamHandle(Info, monitorinflation) lsh_info;
1242   LogStream* ls = NULL;
1243   if (log_is_enabled(Debug, monitorinflation)) {
1244     ls = &lsh_debug;
1245   } else if ((free_count != 0 || in_use_count != 0) &&
1246              log_is_enabled(Info, monitorinflation)) {
1247     ls = &lsh_info;
1248   }
1249   if (ls != NULL) {
1250     ls->print_cr("om_flush: jt=" INTPTR_FORMAT ", free_count=%d"
1251                  ", in_use_count=%d" ", om_free_provision=%d",
1252                  p2i(self), free_count, in_use_count, self->om_free_provision);
1253   }
1254 }
1255 
1256 static void post_monitor_inflate_event(EventJavaMonitorInflate* event,
1257                                        const oop obj,
1258                                        ObjectSynchronizer::InflateCause cause) {
1259   assert(event != NULL, "invariant");
1260   assert(event->should_commit(), "invariant");
1261   event->set_monitorClass(obj->klass());
1262   event->set_address((uintptr_t)(void*)obj);
1263   event->set_cause((u1)cause);
1264   event->commit();
1265 }
1266 
1267 // Fast path code shared by multiple functions
1268 void ObjectSynchronizer::inflate_helper(oop obj) {

1269   markWord mark = obj->mark();
1270   if (mark.has_monitor()) {
1271     assert(ObjectSynchronizer::verify_objmon_isinpool(mark.monitor()), "monitor is invalid");
1272     assert(mark.monitor()->header().is_neutral(), "monitor must record a good object header");










1273     return;
1274   }
1275   inflate(Thread::current(), obj, inflate_cause_vm_internal);
1276 }
1277 
1278 ObjectMonitor* ObjectSynchronizer::inflate(Thread* self,
1279                                            oop object,
1280                                            const InflateCause cause) {
1281   // Inflate mutates the heap ...
1282   // Relaxing assertion for bug 6320749.
1283   assert(Universe::verify_in_progress() ||
1284          !SafepointSynchronize::is_at_safepoint(), "invariant");
1285 
1286   EventJavaMonitorInflate event;
1287 
1288   for (;;) {
1289     const markWord mark = object->mark();
1290     assert(!mark.has_bias_pattern(), "invariant");
1291 
1292     // The mark can be in one of the following states:
1293     // *  Inflated     - just return
1294     // *  Stack-locked - coerce it to inflated
1295     // *  INFLATING    - busy wait for conversion to complete
1296     // *  Neutral      - aggressively inflate the object.
1297     // *  BIASED       - Illegal.  We should never see this
1298 
1299     // CASE: inflated
1300     if (mark.has_monitor()) {
1301       ObjectMonitor* inf = mark.monitor();





1302       markWord dmw = inf->header();
1303       assert(dmw.is_neutral(), "invariant: header=" INTPTR_FORMAT, dmw.value());
1304       assert(inf->object() == object, "invariant");
1305       assert(ObjectSynchronizer::verify_objmon_isinpool(inf), "monitor is invalid");
1306       return inf;
1307     }
1308 
1309     // CASE: inflation in progress - inflating over a stack-lock.
1310     // Some other thread is converting from stack-locked to inflated.
1311     // Only that thread can complete inflation -- other threads must wait.
1312     // The INFLATING value is transient.
1313     // Currently, we spin/yield/park and poll the markword, waiting for inflation to finish.
1314     // We could always eliminate polling by parking the thread on some auxiliary list.
1315     if (mark == markWord::INFLATING()) {
1316       read_stable_mark(object);
1317       continue;
1318     }
1319 
1320     // CASE: stack-locked
1321     // Could be stack-locked either by this thread or by some other thread.
1322     //
1323     // Note that we allocate the objectmonitor speculatively, _before_ attempting
1324     // to install INFLATING into the mark word.  We originally installed INFLATING,
1325     // allocated the objectmonitor, and then finally STed the address of the
1326     // objectmonitor into the mark.  This was correct, but artificially lengthened
1327     // the interval in which INFLATED appeared in the mark, thus increasing
1328     // the odds of inflation contention.
1329     //
1330     // We now use per-thread private objectmonitor free lists.
1331     // These list are reprovisioned from the global free list outside the
1332     // critical INFLATING...ST interval.  A thread can transfer
1333     // multiple objectmonitors en-mass from the global free list to its local free list.
1334     // This reduces coherency traffic and lock contention on the global free list.
1335     // Using such local free lists, it doesn't matter if the om_alloc() call appears
1336     // before or after the CAS(INFLATING) operation.
1337     // See the comments in om_alloc().
1338 
1339     LogStreamHandle(Trace, monitorinflation) lsh;
1340 
1341     if (mark.has_locker()) {
1342       ObjectMonitor* m = om_alloc(self);
1343       // Optimistically prepare the objectmonitor - anticipate successful CAS
1344       // We do this before the CAS in order to minimize the length of time
1345       // in which INFLATING appears in the mark.
1346       m->Recycle();
1347       m->_Responsible  = NULL;
1348       m->_SpinDuration = ObjectMonitor::Knob_SpinLimit;   // Consider: maintain by type/class
1349 
1350       markWord cmp = object->cas_set_mark(markWord::INFLATING(), mark);
1351       if (cmp != mark) {

1352         om_release(self, m, true);
1353         continue;       // Interference -- just retry
1354       }
1355 
1356       // We've successfully installed INFLATING (0) into the mark-word.
1357       // This is the only case where 0 will appear in a mark-word.
1358       // Only the singular thread that successfully swings the mark-word
1359       // to 0 can perform (or more precisely, complete) inflation.
1360       //
1361       // Why do we CAS a 0 into the mark-word instead of just CASing the
1362       // mark-word from the stack-locked value directly to the new inflated state?
1363       // Consider what happens when a thread unlocks a stack-locked object.
1364       // It attempts to use CAS to swing the displaced header value from the
1365       // on-stack BasicLock back into the object header.  Recall also that the
1366       // header value (hash code, etc) can reside in (a) the object header, or
1367       // (b) a displaced header associated with the stack-lock, or (c) a displaced
1368       // header in an ObjectMonitor.  The inflate() routine must copy the header
1369       // value from the BasicLock on the owner's stack to the ObjectMonitor, all
1370       // the while preserving the hashCode stability invariants.  If the owner
1371       // decides to release the lock while the value is 0, the unlock will fail
1372       // and control will eventually pass from slow_exit() to inflate.  The owner
1373       // will then spin, waiting for the 0 value to disappear.   Put another way,
1374       // the 0 causes the owner to stall if the owner happens to try to
1375       // drop the lock (restoring the header from the BasicLock to the object)
1376       // while inflation is in-progress.  This protocol avoids races that might
1377       // would otherwise permit hashCode values to change or "flicker" for an object.
1378       // Critically, while object->mark is 0 mark.displaced_mark_helper() is stable.
1379       // 0 serves as a "BUSY" inflate-in-progress indicator.
1380 
1381 
1382       // fetch the displaced mark from the owner's stack.
1383       // The owner can't die or unwind past the lock while our INFLATING
1384       // object is in the mark.  Furthermore the owner can't complete
1385       // an unlock on the object, either.
1386       markWord dmw = mark.displaced_mark_helper();
1387       // Catch if the object's header is not neutral (not locked and
1388       // not marked is what we care about here).
1389       assert(dmw.is_neutral(), "invariant: header=" INTPTR_FORMAT, dmw.value());
1390 
1391       // Setup monitor fields to proper values -- prepare the monitor
1392       m->set_header(dmw);
1393 
1394       // Optimization: if the mark.locker stack address is associated
1395       // with this thread we could simply set m->_owner = self.
1396       // Note that a thread can inflate an object
1397       // that it has stack-locked -- as might happen in wait() -- directly
1398       // with CAS.  That is, we can avoid the xchg-NULL .... ST idiom.
1399       m->set_owner(mark.locker());




1400       m->set_object(object);
1401       // TODO-FIXME: assert BasicLock->dhw != 0.
1402 


1403       // Must preserve store ordering. The monitor state must
1404       // be stable at the time of publishing the monitor address.
1405       guarantee(object->mark() == markWord::INFLATING(), "invariant");
1406       object->release_set_mark(markWord::encode(m));
1407 





1408       // Hopefully the performance counters are allocated on distinct cache lines
1409       // to avoid false sharing on MP systems ...
1410       OM_PERFDATA_OP(Inflations, inc());
1411       if (log_is_enabled(Trace, monitorinflation)) {
1412         ResourceMark rm(self);
1413         lsh.print_cr("inflate(has_locker): object=" INTPTR_FORMAT ", mark="
1414                      INTPTR_FORMAT ", type='%s'", p2i(object),
1415                      object->mark().value(), object->klass()->external_name());
1416       }
1417       if (event.should_commit()) {
1418         post_monitor_inflate_event(&event, object, cause);
1419       }
1420       return m;

1421     }
1422 
1423     // CASE: neutral
1424     // TODO-FIXME: for entry we currently inflate and then try to CAS _owner.
1425     // If we know we're inflating for entry it's better to inflate by swinging a
1426     // pre-locked ObjectMonitor pointer into the object header.   A successful
1427     // CAS inflates the object *and* confers ownership to the inflating thread.
1428     // In the current implementation we use a 2-step mechanism where we CAS()
1429     // to inflate and then CAS() again to try to swing _owner from NULL to self.
1430     // An inflateTry() method that we could call from enter() would be useful.
1431 
1432     // Catch if the object's header is not neutral (not locked and
1433     // not marked is what we care about here).
1434     assert(mark.is_neutral(), "invariant: header=" INTPTR_FORMAT, mark.value());
1435     ObjectMonitor* m = om_alloc(self);
1436     // prepare m for installation - set monitor to initial state
1437     m->Recycle();
1438     m->set_header(mark);



1439     m->set_object(object);
1440     m->_Responsible  = NULL;
1441     m->_SpinDuration = ObjectMonitor::Knob_SpinLimit;       // consider: keep metastats by type/class
1442 


1443     if (object->cas_set_mark(markWord::encode(m), mark) != mark) {
1444       m->set_header(markWord::zero());
1445       m->set_object(NULL);
1446       m->Recycle();


1447       om_release(self, m, true);
1448       m = NULL;
1449       continue;
1450       // interference - the markword changed - just retry.
1451       // The state-transitions are one-way, so there's no chance of
1452       // live-lock -- "Inflated" is an absorbing state.
1453     }
1454 





1455     // Hopefully the performance counters are allocated on distinct
1456     // cache lines to avoid false sharing on MP systems ...
1457     OM_PERFDATA_OP(Inflations, inc());
1458     if (log_is_enabled(Trace, monitorinflation)) {
1459       ResourceMark rm(self);
1460       lsh.print_cr("inflate(neutral): object=" INTPTR_FORMAT ", mark="
1461                    INTPTR_FORMAT ", type='%s'", p2i(object),
1462                    object->mark().value(), object->klass()->external_name());
1463     }
1464     if (event.should_commit()) {
1465       post_monitor_inflate_event(&event, object, cause);
1466     }
1467     return m;

1468   }
1469 }
1470 
1471 
1472 // We maintain a list of in-use monitors for each thread.
1473 //

1474 // deflate_thread_local_monitors() scans a single thread's in-use list, while
1475 // deflate_idle_monitors() scans only a global list of in-use monitors which
1476 // is populated only as a thread dies (see om_flush()).
1477 //
1478 // These operations are called at all safepoints, immediately after mutators
1479 // are stopped, but before any objects have moved. Collectively they traverse
1480 // the population of in-use monitors, deflating where possible. The scavenged
1481 // monitors are returned to the global monitor free list.
1482 //
1483 // Beware that we scavenge at *every* stop-the-world point. Having a large
1484 // number of monitors in-use could negatively impact performance. We also want
1485 // to minimize the total # of monitors in circulation, as they incur a small
1486 // footprint penalty.
1487 //
1488 // Perversely, the heap size -- and thus the STW safepoint rate --
1489 // typically drives the scavenge rate.  Large heaps can mean infrequent GC,
1490 // which in turn can mean large(r) numbers of ObjectMonitors in circulation.
1491 // This is an unfortunate aspect of this design.

























1492 
1493 // Deflate a single monitor if not in-use
1494 // Return true if deflated, false if in-use
1495 bool ObjectSynchronizer::deflate_monitor(ObjectMonitor* mid, oop obj,
1496                                          ObjectMonitor** free_head_p,
1497                                          ObjectMonitor** free_tail_p) {
1498   bool deflated;
1499   // Normal case ... The monitor is associated with obj.
1500   const markWord mark = obj->mark();
1501   guarantee(mark == markWord::encode(mid), "should match: mark="
1502             INTPTR_FORMAT ", encoded mid=" INTPTR_FORMAT, mark.value(),
1503             markWord::encode(mid).value());
1504   // Make sure that mark.monitor() and markWord::encode() agree:
1505   guarantee(mark.monitor() == mid, "should match: monitor()=" INTPTR_FORMAT
1506             ", mid=" INTPTR_FORMAT, p2i(mark.monitor()), p2i(mid));
1507   const markWord dmw = mid->header();
1508   guarantee(dmw.is_neutral(), "invariant: header=" INTPTR_FORMAT, dmw.value());
1509 
1510   if (mid->is_busy()) {


1511     deflated = false;
1512   } else {
1513     // Deflate the monitor if it is no longer being used
1514     // It's idle - scavenge and return to the global free list
1515     // plain old deflation ...
1516     if (log_is_enabled(Trace, monitorinflation)) {
1517       ResourceMark rm;
1518       log_trace(monitorinflation)("deflate_monitor: "
1519                                   "object=" INTPTR_FORMAT ", mark="
1520                                   INTPTR_FORMAT ", type='%s'", p2i(obj),
1521                                   mark.value(), obj->klass()->external_name());
1522     }
1523 
1524     // Restore the header back to obj
1525     obj->release_set_mark(dmw);







1526     mid->clear();
1527 
1528     assert(mid->object() == NULL, "invariant: object=" INTPTR_FORMAT,
1529            p2i(mid->object()));

1530 
1531     // Move the deflated ObjectMonitor to the working free list
1532     // defined by free_head_p and free_tail_p.

1533     if (*free_head_p == NULL) *free_head_p = mid;
1534     if (*free_tail_p != NULL) {
1535       // We append to the list so the caller can use mid->_next_om
1536       // to fix the linkages in its context.
1537       ObjectMonitor* prevtail = *free_tail_p;
1538       // Should have been cleaned up by the caller:
1539       assert(prevtail->_next_om == NULL, "cleaned up deflated?");
1540       prevtail->_next_om = mid;




1541     }
1542     *free_tail_p = mid;
1543     // At this point, mid->_next_om still refers to its current
1544     // value and another ObjectMonitor's _next_om field still
1545     // refers to this ObjectMonitor. Those linkages have to be
1546     // cleaned up by the caller who has the complete context.
1547     deflated = true;
1548   }
1549   return deflated;
1550 }
1551 
1552 // Walk a given monitor list, and deflate idle monitors
1553 // The given list could be a per-thread list or a global list
1554 // Caller acquires gListLock as needed.













































































































































1555 //
1556 // In the case of parallel processing of thread local monitor lists,
1557 // work is done by Threads::parallel_threads_do() which ensures that
1558 // each Java thread is processed by exactly one worker thread, and
1559 // thus avoid conflicts that would arise when worker threads would
1560 // process the same monitor lists concurrently.
1561 //
1562 // See also ParallelSPCleanupTask and
1563 // SafepointSynchronize::do_cleanup_tasks() in safepoint.cpp and
1564 // Threads::parallel_java_threads_do() in thread.cpp.
1565 int ObjectSynchronizer::deflate_monitor_list(ObjectMonitor** list_p,

1566                                              ObjectMonitor** free_head_p,
1567                                              ObjectMonitor** free_tail_p) {
1568   ObjectMonitor* mid;
1569   ObjectMonitor* next;
1570   ObjectMonitor* cur_mid_in_use = NULL;


1571   int deflated_count = 0;
1572 
1573   for (mid = *list_p; mid != NULL;) {






1574     oop obj = (oop) mid->object();
1575     if (obj != NULL && deflate_monitor(mid, obj, free_head_p, free_tail_p)) {
1576       // Deflation succeeded and already updated free_head_p and
1577       // free_tail_p as needed. Finish the move to the local free list
1578       // by unlinking mid from the global or per-thread in-use list.
1579       if (mid == *list_p) {
1580         *list_p = mid->_next_om;
1581       } else if (cur_mid_in_use != NULL) {
1582         cur_mid_in_use->_next_om = mid->_next_om; // maintain the current thread in-use list





1583       }
1584       next = mid->_next_om;
1585       mid->_next_om = NULL;  // This mid is current tail in the free_head_p list
1586       mid = next;
1587       deflated_count++;







1588     } else {






















































































































1589       cur_mid_in_use = mid;
1590       mid = mid->_next_om;















1591     }



1592   }









1593   return deflated_count;
1594 }
1595 
1596 void ObjectSynchronizer::prepare_deflate_idle_monitors(DeflateMonitorCounters* counters) {
1597   counters->n_in_use = 0;              // currently associated with objects
1598   counters->n_in_circulation = 0;      // extant
1599   counters->n_scavenged = 0;           // reclaimed (global and per-thread)
1600   counters->per_thread_scavenged = 0;  // per-thread scavenge total
1601   counters->per_thread_times = 0.0;    // per-thread scavenge times
1602 }
1603 
1604 void ObjectSynchronizer::deflate_idle_monitors(DeflateMonitorCounters* counters) {
1605   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");









1606   bool deflated = false;
1607 
1608   ObjectMonitor* free_head_p = NULL;  // Local SLL of scavenged monitors
1609   ObjectMonitor* free_tail_p = NULL;
1610   elapsedTimer timer;
1611 
1612   if (log_is_enabled(Info, monitorinflation)) {
1613     timer.start();
1614   }
1615 
1616   // Prevent om_flush from changing mids in Thread dtor's during deflation
1617   // And in case the vm thread is acquiring a lock during a safepoint
1618   // See e.g. 6320749
1619   Thread::muxAcquire(&gListLock, "deflate_idle_monitors");
1620 
1621   // Note: the thread-local monitors lists get deflated in
1622   // a separate pass. See deflate_thread_local_monitors().
1623 
1624   // For moribund threads, scan g_om_in_use_list
1625   int deflated_count = 0;
1626   if (g_om_in_use_list) {
1627     counters->n_in_circulation += g_om_in_use_count;
1628     deflated_count = deflate_monitor_list((ObjectMonitor **)&g_om_in_use_list, &free_head_p, &free_tail_p);
1629     g_om_in_use_count -= deflated_count;
1630     counters->n_scavenged += deflated_count;
1631     counters->n_in_use += g_om_in_use_count;
1632   }
1633 
1634   if (free_head_p != NULL) {
1635     // Move the deflated ObjectMonitors back to the global free list.
1636     guarantee(free_tail_p != NULL && counters->n_scavenged > 0, "invariant");
1637     assert(free_tail_p->_next_om == NULL, "invariant");
1638     // constant-time list splice - prepend scavenged segment to g_free_list
1639     free_tail_p->_next_om = g_free_list;
1640     g_free_list = free_head_p;

1641   }
1642   Thread::muxRelease(&gListLock);
1643   timer.stop();
1644 
1645   LogStreamHandle(Debug, monitorinflation) lsh_debug;
1646   LogStreamHandle(Info, monitorinflation) lsh_info;
1647   LogStream* ls = NULL;
1648   if (log_is_enabled(Debug, monitorinflation)) {
1649     ls = &lsh_debug;
1650   } else if (deflated_count != 0 && log_is_enabled(Info, monitorinflation)) {
1651     ls = &lsh_info;
1652   }
1653   if (ls != NULL) {
1654     ls->print_cr("deflating global idle monitors, %3.7f secs, %d monitors", timer.seconds(), deflated_count);
1655   }
1656 }
1657 













































































































































































1658 void ObjectSynchronizer::finish_deflate_idle_monitors(DeflateMonitorCounters* counters) {
1659   // Report the cumulative time for deflating each thread's idle
1660   // monitors. Note: if the work is split among more than one
1661   // worker thread, then the reported time will likely be more
1662   // than a beginning to end measurement of the phase.
1663   log_info(safepoint, cleanup)("deflating per-thread idle monitors, %3.7f secs, monitors=%d", counters->per_thread_times, counters->per_thread_scavenged);









1664 
1665   g_om_free_count += counters->n_scavenged;


1666 
1667   if (log_is_enabled(Debug, monitorinflation)) {
1668     // exit_globals()'s call to audit_and_print_stats() is done
1669     // at the Info level.
1670     ObjectSynchronizer::audit_and_print_stats(false /* on_exit */);
1671   } else if (log_is_enabled(Info, monitorinflation)) {
1672     Thread::muxAcquire(&gListLock, "finish_deflate_idle_monitors");
1673     log_info(monitorinflation)("g_om_population=%d, g_om_in_use_count=%d, "
1674                                "g_om_free_count=%d", g_om_population,
1675                                g_om_in_use_count, g_om_free_count);
1676     Thread::muxRelease(&gListLock);


1677   }
1678 
1679   ForceMonitorScavenge = 0;    // Reset
1680 
1681   OM_PERFDATA_OP(Deflations, inc(counters->n_scavenged));
1682   OM_PERFDATA_OP(MonExtant, set_value(counters->n_in_circulation));
1683 
1684   GVars.stw_random = os::random();
1685   GVars.stw_cycle++;



1686 }
1687 
1688 void ObjectSynchronizer::deflate_thread_local_monitors(Thread* thread, DeflateMonitorCounters* counters) {
1689   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
1690 





1691   ObjectMonitor* free_head_p = NULL;  // Local SLL of scavenged monitors
1692   ObjectMonitor* free_tail_p = NULL;
1693   elapsedTimer timer;
1694 
1695   if (log_is_enabled(Info, safepoint, cleanup) ||
1696       log_is_enabled(Info, monitorinflation)) {
1697     timer.start();
1698   }
1699 
1700   int deflated_count = deflate_monitor_list(thread->om_in_use_list_addr(), &free_head_p, &free_tail_p);
1701 
1702   Thread::muxAcquire(&gListLock, "deflate_thread_local_monitors");
1703 
1704   // Adjust counters
1705   counters->n_in_circulation += thread->om_in_use_count;
1706   thread->om_in_use_count -= deflated_count;
1707   counters->n_scavenged += deflated_count;
1708   counters->n_in_use += thread->om_in_use_count;
1709   counters->per_thread_scavenged += deflated_count;
1710 
1711   if (free_head_p != NULL) {
1712     // Move the deflated ObjectMonitors back to the global free list.

1713     guarantee(free_tail_p != NULL && deflated_count > 0, "invariant");
1714     assert(free_tail_p->_next_om == NULL, "invariant");
1715 
1716     // constant-time list splice - prepend scavenged segment to g_free_list
1717     free_tail_p->_next_om = g_free_list;
1718     g_free_list = free_head_p;
1719   }
1720 
1721   timer.stop();
1722   // Safepoint logging cares about cumulative per_thread_times and
1723   // we'll capture most of the cost, but not the muxRelease() which
1724   // should be cheap.
1725   counters->per_thread_times += timer.seconds();
1726 
1727   Thread::muxRelease(&gListLock);
1728 
1729   LogStreamHandle(Debug, monitorinflation) lsh_debug;
1730   LogStreamHandle(Info, monitorinflation) lsh_info;
1731   LogStream* ls = NULL;
1732   if (log_is_enabled(Debug, monitorinflation)) {
1733     ls = &lsh_debug;
1734   } else if (deflated_count != 0 && log_is_enabled(Info, monitorinflation)) {
1735     ls = &lsh_info;
1736   }
1737   if (ls != NULL) {
1738     ls->print_cr("jt=" INTPTR_FORMAT ": deflating per-thread idle monitors, %3.7f secs, %d monitors", p2i(thread), timer.seconds(), deflated_count);
1739   }
1740 }
1741 
1742 // Monitor cleanup on JavaThread::exit
1743 
1744 // Iterate through monitor cache and attempt to release thread's monitors
1745 // Gives up on a particular monitor if an exception occurs, but continues
1746 // the overall iteration, swallowing the exception.
1747 class ReleaseJavaMonitorsClosure: public MonitorClosure {
1748  private:


1759 
1760 // Release all inflated monitors owned by THREAD.  Lightweight monitors are
1761 // ignored.  This is meant to be called during JNI thread detach which assumes
1762 // all remaining monitors are heavyweight.  All exceptions are swallowed.
1763 // Scanning the extant monitor list can be time consuming.
1764 // A simple optimization is to add a per-thread flag that indicates a thread
1765 // called jni_monitorenter() during its lifetime.
1766 //
1767 // Instead of No_Savepoint_Verifier it might be cheaper to
1768 // use an idiom of the form:
1769 //   auto int tmp = SafepointSynchronize::_safepoint_counter ;
1770 //   <code that must not run at safepoint>
1771 //   guarantee (((tmp ^ _safepoint_counter) | (tmp & 1)) == 0) ;
1772 // Since the tests are extremely cheap we could leave them enabled
1773 // for normal product builds.
1774 
1775 void ObjectSynchronizer::release_monitors_owned_by_thread(TRAPS) {
1776   assert(THREAD == JavaThread::current(), "must be current Java thread");
1777   NoSafepointVerifier nsv;
1778   ReleaseJavaMonitorsClosure rjmc(THREAD);
1779   Thread::muxAcquire(&gListLock, "release_monitors_owned_by_thread");
1780   ObjectSynchronizer::monitors_iterate(&rjmc);
1781   Thread::muxRelease(&gListLock);
1782   THREAD->clear_pending_exception();
1783 }
1784 
1785 const char* ObjectSynchronizer::inflate_cause_name(const InflateCause cause) {
1786   switch (cause) {
1787     case inflate_cause_vm_internal:    return "VM Internal";
1788     case inflate_cause_monitor_enter:  return "Monitor Enter";
1789     case inflate_cause_wait:           return "Monitor Wait";
1790     case inflate_cause_notify:         return "Monitor Notify";
1791     case inflate_cause_hash_code:      return "Monitor Hash Code";
1792     case inflate_cause_jni_enter:      return "JNI Monitor Enter";
1793     case inflate_cause_jni_exit:       return "JNI Monitor Exit";
1794     default:
1795       ShouldNotReachHere();
1796   }
1797   return "Unknown";
1798 }
1799 
1800 //------------------------------------------------------------------------------
1801 // Debugging code


1815 u_char* ObjectSynchronizer::get_gvars_stw_random_addr() {
1816   return (u_char*)&GVars.stw_random;
1817 }
1818 
1819 void ObjectSynchronizer::audit_and_print_stats(bool on_exit) {
1820   assert(on_exit || SafepointSynchronize::is_at_safepoint(), "invariant");
1821 
1822   LogStreamHandle(Debug, monitorinflation) lsh_debug;
1823   LogStreamHandle(Info, monitorinflation) lsh_info;
1824   LogStreamHandle(Trace, monitorinflation) lsh_trace;
1825   LogStream* ls = NULL;
1826   if (log_is_enabled(Trace, monitorinflation)) {
1827     ls = &lsh_trace;
1828   } else if (log_is_enabled(Debug, monitorinflation)) {
1829     ls = &lsh_debug;
1830   } else if (log_is_enabled(Info, monitorinflation)) {
1831     ls = &lsh_info;
1832   }
1833   assert(ls != NULL, "sanity check");
1834 
1835   if (!on_exit) {
1836     // Not at VM exit so grab the global list lock.
1837     Thread::muxAcquire(&gListLock, "audit_and_print_stats");
1838   }
1839 
1840   // Log counts for the global and per-thread monitor lists:
1841   int chk_om_population = log_monitor_list_counts(ls);
1842   int error_cnt = 0;
1843 
1844   ls->print_cr("Checking global lists:");
1845 
1846   // Check g_om_population:
1847   if (g_om_population == chk_om_population) {
1848     ls->print_cr("g_om_population=%d equals chk_om_population=%d",
1849                  g_om_population, chk_om_population);

1850   } else {
1851     ls->print_cr("ERROR: g_om_population=%d is not equal to "
1852                  "chk_om_population=%d", g_om_population,





1853                  chk_om_population);
1854     error_cnt++;
1855   }
1856 
1857   // Check g_om_in_use_list and g_om_in_use_count:
1858   chk_global_in_use_list_and_count(ls, &error_cnt);
1859 
1860   // Check g_free_list and g_om_free_count:
1861   chk_global_free_list_and_count(ls, &error_cnt);
1862 
1863   if (!on_exit) {
1864     Thread::muxRelease(&gListLock);

1865   }
1866 
1867   ls->print_cr("Checking per-thread lists:");
1868 
1869   for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = jtiwh.next(); ) {
1870     // Check om_in_use_list and om_in_use_count:
1871     chk_per_thread_in_use_list_and_count(jt, ls, &error_cnt);
1872 
1873     // Check om_free_list and om_free_count:
1874     chk_per_thread_free_list_and_count(jt, ls, &error_cnt);
1875   }
1876 
1877   if (error_cnt == 0) {
1878     ls->print_cr("No errors found in monitor list checks.");
1879   } else {
1880     log_error(monitorinflation)("found monitor list errors: error_cnt=%d", error_cnt);
1881   }
1882 
1883   if ((on_exit && log_is_enabled(Info, monitorinflation)) ||
1884       (!on_exit && log_is_enabled(Trace, monitorinflation))) {
1885     // When exiting this log output is at the Info level. When called
1886     // at a safepoint, this log output is at the Trace level since
1887     // there can be a lot of it.
1888     log_in_use_monitor_details(ls, on_exit);
1889   }
1890 
1891   ls->flush();
1892 
1893   guarantee(error_cnt == 0, "ERROR: found monitor list errors: error_cnt=%d", error_cnt);
1894 }
1895 
1896 // Check a free monitor entry; log any errors.
1897 void ObjectSynchronizer::chk_free_entry(JavaThread* jt, ObjectMonitor* n,
1898                                         outputStream * out, int *error_cnt_p) {
1899   stringStream ss;
1900   if (n->is_busy()) {
1901     if (jt != NULL) {
1902       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
1903                     ": free per-thread monitor must not be busy: %s", p2i(jt),
1904                     p2i(n), n->is_busy_to_string(&ss));
1905     } else {
1906       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": free global monitor "
1907                     "must not be busy: %s", p2i(n), n->is_busy_to_string(&ss));
1908     }
1909     *error_cnt_p = *error_cnt_p + 1;
1910   }
1911   if (n->header().value() != 0) {
1912     if (jt != NULL) {
1913       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
1914                     ": free per-thread monitor must have NULL _header "
1915                     "field: _header=" INTPTR_FORMAT, p2i(jt), p2i(n),
1916                     n->header().value());
1917     } else {

1918       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": free global monitor "
1919                     "must have NULL _header field: _header=" INTPTR_FORMAT,
1920                     p2i(n), n->header().value());
1921     }
1922     *error_cnt_p = *error_cnt_p + 1;
1923   }

1924   if (n->object() != NULL) {
1925     if (jt != NULL) {
1926       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
1927                     ": free per-thread monitor must have NULL _object "
1928                     "field: _object=" INTPTR_FORMAT, p2i(jt), p2i(n),
1929                     p2i(n->object()));
1930     } else {
1931       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": free global monitor "
1932                     "must have NULL _object field: _object=" INTPTR_FORMAT,
1933                     p2i(n), p2i(n->object()));
1934     }
1935     *error_cnt_p = *error_cnt_p + 1;
1936   }
1937 }
1938 
1939 // Check the global free list and count; log the results of the checks.
1940 void ObjectSynchronizer::chk_global_free_list_and_count(outputStream * out,
1941                                                         int *error_cnt_p) {
1942   int chk_om_free_count = 0;
1943   for (ObjectMonitor* n = g_free_list; n != NULL; n = n->_next_om) {
1944     chk_free_entry(NULL /* jt */, n, out, error_cnt_p);
1945     chk_om_free_count++;
1946   }
1947   if (g_om_free_count == chk_om_free_count) {
1948     out->print_cr("g_om_free_count=%d equals chk_om_free_count=%d",
1949                   g_om_free_count, chk_om_free_count);

1950   } else {
1951     out->print_cr("ERROR: g_om_free_count=%d is not equal to "
1952                   "chk_om_free_count=%d", g_om_free_count,





1953                   chk_om_free_count);





















1954     *error_cnt_p = *error_cnt_p + 1;
1955   }
1956 }
1957 
1958 // Check the global in-use list and count; log the results of the checks.
1959 void ObjectSynchronizer::chk_global_in_use_list_and_count(outputStream * out,
1960                                                           int *error_cnt_p) {
1961   int chk_om_in_use_count = 0;
1962   for (ObjectMonitor* n = g_om_in_use_list; n != NULL; n = n->_next_om) {
1963     chk_in_use_entry(NULL /* jt */, n, out, error_cnt_p);
1964     chk_om_in_use_count++;
1965   }
1966   if (g_om_in_use_count == chk_om_in_use_count) {
1967     out->print_cr("g_om_in_use_count=%d equals chk_om_in_use_count=%d", g_om_in_use_count,

1968                   chk_om_in_use_count);
1969   } else {
1970     out->print_cr("ERROR: g_om_in_use_count=%d is not equal to chk_om_in_use_count=%d",
1971                   g_om_in_use_count, chk_om_in_use_count);
1972     *error_cnt_p = *error_cnt_p + 1;



1973   }
1974 }
1975 
1976 // Check an in-use monitor entry; log any errors.
1977 void ObjectSynchronizer::chk_in_use_entry(JavaThread* jt, ObjectMonitor* n,
1978                                           outputStream * out, int *error_cnt_p) {
1979   if (n->header().value() == 0) {
1980     if (jt != NULL) {
1981       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
1982                     ": in-use per-thread monitor must have non-NULL _header "
1983                     "field.", p2i(jt), p2i(n));
1984     } else {
1985       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": in-use global monitor "
1986                     "must have non-NULL _header field.", p2i(n));
1987     }
1988     *error_cnt_p = *error_cnt_p + 1;
1989   }
1990   if (n->object() == NULL) {
1991     if (jt != NULL) {
1992       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT


2020       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
2021                     ": in-use per-thread monitor's object does not refer "
2022                     "to the same monitor: obj=" INTPTR_FORMAT ", mark="
2023                     INTPTR_FORMAT ", obj_mon=" INTPTR_FORMAT, p2i(jt),
2024                     p2i(n), p2i(obj), mark.value(), p2i(obj_mon));
2025     } else {
2026       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": in-use global "
2027                     "monitor's object does not refer to the same monitor: obj="
2028                     INTPTR_FORMAT ", mark=" INTPTR_FORMAT ", obj_mon="
2029                     INTPTR_FORMAT, p2i(n), p2i(obj), mark.value(), p2i(obj_mon));
2030     }
2031     *error_cnt_p = *error_cnt_p + 1;
2032   }
2033 }
2034 
2035 // Check the thread's free list and count; log the results of the checks.
2036 void ObjectSynchronizer::chk_per_thread_free_list_and_count(JavaThread *jt,
2037                                                             outputStream * out,
2038                                                             int *error_cnt_p) {
2039   int chk_om_free_count = 0;
2040   for (ObjectMonitor* n = jt->om_free_list; n != NULL; n = n->_next_om) {
2041     chk_free_entry(jt, n, out, error_cnt_p);
2042     chk_om_free_count++;
2043   }
2044   if (jt->om_free_count == chk_om_free_count) {
2045     out->print_cr("jt=" INTPTR_FORMAT ": om_free_count=%d equals "
2046                   "chk_om_free_count=%d", p2i(jt), jt->om_free_count, chk_om_free_count);


2047   } else {
2048     out->print_cr("ERROR: jt=" INTPTR_FORMAT ": om_free_count=%d is not "
2049                   "equal to chk_om_free_count=%d", p2i(jt), jt->om_free_count,

2050                   chk_om_free_count);
2051     *error_cnt_p = *error_cnt_p + 1;
2052   }
2053 }
2054 
2055 // Check the thread's in-use list and count; log the results of the checks.
2056 void ObjectSynchronizer::chk_per_thread_in_use_list_and_count(JavaThread *jt,
2057                                                               outputStream * out,
2058                                                               int *error_cnt_p) {
2059   int chk_om_in_use_count = 0;
2060   for (ObjectMonitor* n = jt->om_in_use_list; n != NULL; n = n->_next_om) {
2061     chk_in_use_entry(jt, n, out, error_cnt_p);
2062     chk_om_in_use_count++;
2063   }
2064   if (jt->om_in_use_count == chk_om_in_use_count) {
2065     out->print_cr("jt=" INTPTR_FORMAT ": om_in_use_count=%d equals "
2066                   "chk_om_in_use_count=%d", p2i(jt), jt->om_in_use_count,

2067                   chk_om_in_use_count);
2068   } else {
2069     out->print_cr("ERROR: jt=" INTPTR_FORMAT ": om_in_use_count=%d is not "
2070                   "equal to chk_om_in_use_count=%d", p2i(jt), jt->om_in_use_count,

2071                   chk_om_in_use_count);
2072     *error_cnt_p = *error_cnt_p + 1;
2073   }
2074 }
2075 
2076 // Log details about ObjectMonitors on the in-use lists. The 'BHL'
2077 // flags indicate why the entry is in-use, 'object' and 'object type'
2078 // indicate the associated object and its type.
2079 void ObjectSynchronizer::log_in_use_monitor_details(outputStream * out,
2080                                                     bool on_exit) {
2081   if (!on_exit) {
2082     // Not at VM exit so grab the global list lock.
2083     Thread::muxAcquire(&gListLock, "log_in_use_monitor_details");
2084   }
2085 
2086   stringStream ss;
2087   if (g_om_in_use_count > 0) {
2088     out->print_cr("In-use global monitor info:");
2089     out->print_cr("(B -> is_busy, H -> has hash code, L -> lock status)");
2090     out->print_cr("%18s  %s  %18s  %18s",
2091                   "monitor", "BHL", "object", "object type");
2092     out->print_cr("==================  ===  ==================  ==================");
2093     for (ObjectMonitor* n = g_om_in_use_list; n != NULL; n = n->_next_om) {
2094       const oop obj = (oop) n->object();
2095       const markWord mark = n->header();
2096       ResourceMark rm;
2097       out->print(INTPTR_FORMAT "  %d%d%d  " INTPTR_FORMAT "  %s", p2i(n),
2098                  n->is_busy() != 0, mark.hash() != 0, n->owner() != NULL,
2099                  p2i(obj), obj->klass()->external_name());

2100       if (n->is_busy() != 0) {
2101         out->print(" (%s)", n->is_busy_to_string(&ss));
2102         ss.reset();
2103       }
2104       out->cr();
2105     }
2106   }
2107 
2108   if (!on_exit) {
2109     Thread::muxRelease(&gListLock);
2110   }
2111 
2112   out->print_cr("In-use per-thread monitor info:");
2113   out->print_cr("(B -> is_busy, H -> has hash code, L -> lock status)");
2114   out->print_cr("%18s  %18s  %s  %18s  %18s",
2115                 "jt", "monitor", "BHL", "object", "object type");
2116   out->print_cr("==================  ==================  ===  ==================  ==================");
2117   for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = jtiwh.next(); ) {
2118     for (ObjectMonitor* n = jt->om_in_use_list; n != NULL; n = n->_next_om) {
2119       const oop obj = (oop) n->object();
2120       const markWord mark = n->header();
2121       ResourceMark rm;
2122       out->print(INTPTR_FORMAT "  " INTPTR_FORMAT "  %d%d%d  " INTPTR_FORMAT
2123                  "  %s", p2i(jt), p2i(n), n->is_busy() != 0,
2124                  mark.hash() != 0, n->owner() != NULL, p2i(obj),
2125                  obj->klass()->external_name());
2126       if (n->is_busy() != 0) {
2127         out->print(" (%s)", n->is_busy_to_string(&ss));
2128         ss.reset();
2129       }
2130       out->cr();
2131     }
2132   }
2133 
2134   out->flush();
2135 }
2136 
2137 // Log counts for the global and per-thread monitor lists and return
2138 // the population count.
2139 int ObjectSynchronizer::log_monitor_list_counts(outputStream * out) {
2140   int pop_count = 0;
2141   out->print_cr("%18s  %10s  %10s  %10s",
2142                 "Global Lists:", "InUse", "Free", "Total");
2143   out->print_cr("==================  ==========  ==========  ==========");
2144   out->print_cr("%18s  %10d  %10d  %10d", "",
2145                 g_om_in_use_count, g_om_free_count, g_om_population);
2146   pop_count += g_om_in_use_count + g_om_free_count;







2147 
2148   out->print_cr("%18s  %10s  %10s  %10s",
2149                 "Per-Thread Lists:", "InUse", "Free", "Provision");
2150   out->print_cr("==================  ==========  ==========  ==========");
2151 
2152   for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = jtiwh.next(); ) {
2153     out->print_cr(INTPTR_FORMAT "  %10d  %10d  %10d", p2i(jt),
2154                   jt->om_in_use_count, jt->om_free_count, jt->om_free_provision);
2155     pop_count += jt->om_in_use_count + jt->om_free_count;



2156   }
2157   return pop_count;
2158 }
2159 
2160 #ifndef PRODUCT
2161 
2162 // Check if monitor belongs to the monitor cache
2163 // The list is grow-only so it's *relatively* safe to traverse
2164 // the list of extant blocks without taking a lock.
2165 
2166 int ObjectSynchronizer::verify_objmon_isinpool(ObjectMonitor *monitor) {
2167   PaddedObjectMonitor* block = OrderAccess::load_acquire(&g_block_list);
2168   while (block != NULL) {
2169     assert(block->object() == CHAINMARKER, "must be a block header");
2170     if (monitor > &block[0] && monitor < &block[_BLOCKSIZE]) {
2171       address mon = (address)monitor;
2172       address blk = (address)block;
2173       size_t diff = mon - blk;
2174       assert((diff % sizeof(PaddedObjectMonitor)) == 0, "must be aligned");
2175       return 1;
2176     }
2177     block = (PaddedObjectMonitor*)block->_next_om;

2178   }
2179   return 0;
2180 }
2181 
2182 #endif


  20  * or visit www.oracle.com if you need additional information or have any
  21  * questions.
  22  *
  23  */
  24 
  25 #include "precompiled.hpp"
  26 #include "classfile/vmSymbols.hpp"
  27 #include "logging/log.hpp"
  28 #include "logging/logStream.hpp"
  29 #include "jfr/jfrEvents.hpp"
  30 #include "memory/allocation.inline.hpp"
  31 #include "memory/metaspaceShared.hpp"
  32 #include "memory/padded.hpp"
  33 #include "memory/resourceArea.hpp"
  34 #include "memory/universe.hpp"
  35 #include "oops/markWord.hpp"
  36 #include "oops/oop.inline.hpp"
  37 #include "runtime/atomic.hpp"
  38 #include "runtime/biasedLocking.hpp"
  39 #include "runtime/handles.inline.hpp"
  40 #include "runtime/handshake.hpp"
  41 #include "runtime/interfaceSupport.inline.hpp"
  42 #include "runtime/mutexLocker.hpp"
  43 #include "runtime/objectMonitor.hpp"
  44 #include "runtime/objectMonitor.inline.hpp"
  45 #include "runtime/osThread.hpp"
  46 #include "runtime/safepointVerifiers.hpp"
  47 #include "runtime/sharedRuntime.hpp"
  48 #include "runtime/stubRoutines.hpp"
  49 #include "runtime/synchronizer.hpp"
  50 #include "runtime/thread.inline.hpp"
  51 #include "runtime/timer.hpp"
  52 #include "runtime/vframe.hpp"
  53 #include "runtime/vmThread.hpp"
  54 #include "utilities/align.hpp"
  55 #include "utilities/dtrace.hpp"
  56 #include "utilities/events.hpp"
  57 #include "utilities/preserveException.hpp"
  58 
  59 // The "core" versions of monitor enter and exit reside in this file.
  60 // The interpreter and compilers contain specialized transliterated


 102   }
 103 
 104 #else //  ndef DTRACE_ENABLED
 105 
 106 #define DTRACE_MONITOR_WAIT_PROBE(obj, thread, millis, mon)    {;}
 107 #define DTRACE_MONITOR_PROBE(probe, obj, thread, mon)          {;}
 108 
 109 #endif // ndef DTRACE_ENABLED
 110 
 111 // This exists only as a workaround of dtrace bug 6254741
 112 int dtrace_waited_probe(ObjectMonitor* monitor, Handle obj, Thread* thr) {
 113   DTRACE_MONITOR_PROBE(waited, monitor, obj(), thr);
 114   return 0;
 115 }
 116 
 117 #define NINFLATIONLOCKS 256
 118 static volatile intptr_t gInflationLocks[NINFLATIONLOCKS];
 119 
 120 // global list of blocks of monitors
 121 PaddedObjectMonitor* volatile ObjectSynchronizer::g_block_list = NULL;
 122 bool volatile ObjectSynchronizer::_is_async_deflation_requested = false;
 123 bool volatile ObjectSynchronizer::_is_special_deflation_requested = false;
 124 jlong ObjectSynchronizer::_last_async_deflation_time_ns = 0;
 125 
 126 // Global ObjectMonitor free list. Newly allocated and deflated
 127 // ObjectMonitors are prepended here.
 128 static ObjectMonitor* volatile g_free_list = NULL;
 129 // Global ObjectMonitor in-use list. When a JavaThread is exiting,
 130 // ObjectMonitors on its per-thread in-use list are prepended here.
 131 static ObjectMonitor* volatile g_om_in_use_list = NULL;
 132 // Global ObjectMonitor wait list. If HandshakeAfterDeflateIdleMonitors
 133 // is true, deflated ObjectMonitors wait on this list until after a
 134 // handshake or a safepoint for platforms that don't support handshakes.
 135 // After the handshake or safepoint, the deflated ObjectMonitors are
 136 // prepended to g_free_list.
 137 static ObjectMonitor* volatile g_wait_list = NULL;
 138 

 139 static volatile int g_om_free_count = 0;    // # on g_free_list
 140 static volatile int g_om_in_use_count = 0;  // # on g_om_in_use_list
 141 static volatile int g_om_population = 0;    // # Extant -- in circulation
 142 static volatile int g_om_wait_count = 0;    // # on g_wait_list
 143 
 144 #define CHAINMARKER (cast_to_oop<intptr_t>(-1))
 145 
 146 
 147 // =====================> List Management functions
 148 
 149 // Return true if the ObjectMonitor's next field is marked.
 150 // Otherwise returns false.
 151 static bool is_next_marked(ObjectMonitor* om) {
 152   return ((intptr_t)OrderAccess::load_acquire(&om->_next_om) & 0x1) != 0;
 153 }
 154 
 155 // Mark an ObjectMonitor* and return it. Note: the om parameter
 156 // may or may not have been marked originally.
 157 static ObjectMonitor* mark_om_ptr(ObjectMonitor* om) {
 158   return (ObjectMonitor*)((intptr_t)om | 0x1);
 159 }
 160 
 161 // Mark the next field in an ObjectMonitor. If marking was successful,
 162 // then the unmarked next field is returned via parameter and true is
 163 // returned. Otherwise false is returned.
 164 static bool mark_next(ObjectMonitor* om, ObjectMonitor** next_p) {
 165   // Get current next field without any marking value.
 166   ObjectMonitor* next = (ObjectMonitor*)
 167       ((intptr_t)OrderAccess::load_acquire(&om->_next_om) & ~0x1);
 168   if (Atomic::cmpxchg(mark_om_ptr(next), &om->_next_om, next) != next) {
 169     return false;  // Could not mark the next field or it was already marked.
 170   }
 171   *next_p = next;
 172   return true;
 173 }
 174 
 175 // Loop until we mark the next field in an ObjectMonitor. The unmarked
 176 // next field is returned.
 177 static ObjectMonitor* mark_next_loop(ObjectMonitor* om) {
 178   ObjectMonitor* next;
 179   while (true) {
 180     if (mark_next(om, &next)) {
 181       // Marked om's next field so return the unmarked value.
 182       return next;
 183     }
 184   }
 185 }
 186 
 187 // Set the next field in an ObjectMonitor to the specified value.
 188 // The caller of set_next() must be the same thread that marked the
 189 // ObjectMonitor.
 190 static void set_next(ObjectMonitor* om, ObjectMonitor* value) {
 191   OrderAccess::release_store(&om->_next_om, value);
 192 }
 193 
 194 // Mark the next field in the list head ObjectMonitor. If marking was
 195 // successful, then the mid and the unmarked next field are returned
 196 // via parameter and true is returned. Otherwise false is returned.
 197 static bool mark_list_head(ObjectMonitor* volatile * list_p,
 198                            ObjectMonitor** mid_p, ObjectMonitor** next_p) {
 199   while (true) {
 200     ObjectMonitor* mid = OrderAccess::load_acquire(list_p);
 201     if (mid == NULL) {
 202       return false;  // The list is empty so nothing to mark.
 203     }
 204     if (mark_next(mid, next_p)) {
 205       if (OrderAccess::load_acquire(list_p) != mid) {
 206         // The list head changed so we have to retry.
 207         set_next(mid, *next_p);  // unmark mid
 208         continue;
 209       }
 210       // We marked next field to guard against races.
 211       *mid_p = mid;
 212       return true;
 213     }
 214   }
 215 }
 216 
 217 // Return the unmarked next field in an ObjectMonitor. Note: the next
 218 // field may or may not have been marked originally.
 219 static ObjectMonitor* unmarked_next(ObjectMonitor* om) {
 220   return (ObjectMonitor*)((intptr_t)OrderAccess::load_acquire(&om->_next_om) & ~0x1);
 221 }
 222 
 223 // Prepend a list of ObjectMonitors to the specified *list_p. 'tail' is
 224 // the last ObjectMonitor in the list and there are 'count' on the list.
 225 // Also updates the specified *count_p.
 226 static void prepend_list_to_common(ObjectMonitor* list, ObjectMonitor* tail,
 227                                    int count, ObjectMonitor* volatile* list_p,
 228                                    volatile int* count_p) {
 229   while (true) {
 230     ObjectMonitor* cur = OrderAccess::load_acquire(list_p);
 231     // Prepend list to *list_p.
 232     ObjectMonitor* next = NULL;
 233     if (!mark_next(tail, &next)) {
 234       continue;  // failed to mark next field so try it all again
 235     }
 236     set_next(tail, cur);  // tail now points to cur (and unmarks tail)
 237     if (cur == NULL) {
 238       // No potential race with takers or other prependers since
 239       // *list_p is empty.
 240       if (Atomic::cmpxchg(list, list_p, cur) == cur) {
 241         // Successfully switched *list_p to the list value.
 242         Atomic::add(count, count_p);
 243         break;
 244       }
 245       // Implied else: try it all again
 246     } else {
 247       // Try to mark next field to guard against races:
 248       if (!mark_next(cur, &next)) {
 249         continue;  // failed to mark next field so try it all again
 250       }
 251       // We marked the next field so try to switch *list_p to the list value.
 252       if (Atomic::cmpxchg(list, list_p, cur) != cur) {
 253         // The list head has changed so unmark the next field and try again:
 254         set_next(cur, next);
 255         continue;
 256       }
 257       Atomic::add(count, count_p);
 258       set_next(cur, next);  // unmark next field
 259       break;
 260     }
 261   }
 262 }
 263 
 264 // Prepend a newly allocated block of ObjectMonitors to g_block_list and
 265 // g_free_list. Also updates g_om_population and g_om_free_count.
 266 void ObjectSynchronizer::prepend_block_to_lists(PaddedObjectMonitor* new_blk) {
 267   // First we handle g_block_list:
 268   while (true) {
 269     PaddedObjectMonitor* cur = g_block_list;
 270     // Prepend new_blk to g_block_list. The first ObjectMonitor in
 271     // a block is reserved for use as linkage to the next block.
 272     new_blk[0]._next_om = cur;
 273     if (Atomic::cmpxchg(new_blk, &g_block_list, cur) == cur) {
 274       // Successfully switched g_block_list to the new_blk value.
 275       Atomic::add(_BLOCKSIZE - 1, &g_om_population);
 276       break;
 277     }
 278     // Implied else: try it all again
 279   }
 280 
 281   // Second we handle g_free_list:
 282   prepend_list_to_common(new_blk + 1, &new_blk[_BLOCKSIZE - 1], _BLOCKSIZE - 1,
 283                          &g_free_list, &g_om_free_count);
 284 }
 285 
 286 // Prepend a list of ObjectMonitors to g_free_list. 'tail' is the last
 287 // ObjectMonitor in the list and there are 'count' on the list. Also
 288 // updates g_om_free_count.
 289 static void prepend_list_to_g_free_list(ObjectMonitor* list,
 290                                         ObjectMonitor* tail, int count) {
 291   prepend_list_to_common(list, tail, count, &g_free_list, &g_om_free_count);
 292 }
 293 
 294 // Prepend a list of ObjectMonitors to g_wait_list. 'tail' is the last
 295 // ObjectMonitor in the list and there are 'count' on the list. Also
 296 // updates g_om_wait_count.
 297 static void prepend_list_to_g_wait_list(ObjectMonitor* list,
 298                                         ObjectMonitor* tail, int count) {
 299   assert(HandshakeAfterDeflateIdleMonitors, "sanity check");
 300   prepend_list_to_common(list, tail, count, &g_wait_list, &g_om_wait_count);
 301 }
 302 
 303 // Prepend a list of ObjectMonitors to g_om_in_use_list. 'tail' is the last
 304 // ObjectMonitor in the list and there are 'count' on the list. Also
 305 // updates g_om_in_use_list.
 306 static void prepend_list_to_g_om_in_use_list(ObjectMonitor* list,
 307                                              ObjectMonitor* tail, int count) {
 308   prepend_list_to_common(list, tail, count, &g_om_in_use_list, &g_om_in_use_count);
 309 }
 310 
 311 // Prepend an ObjectMonitor to the specified list. Also updates
 312 // the specified counter.
 313 static void prepend_to_common(ObjectMonitor* m, ObjectMonitor* volatile * list_p,
 314                               int volatile * count_p) {
 315   while (true) {
 316     (void)mark_next_loop(m);  // mark m so we can safely update its next field
 317     ObjectMonitor* cur = NULL;
 318     ObjectMonitor* next = NULL;
 319     // Mark the list head to guard against A-B-A race:
 320     if (mark_list_head(list_p, &cur, &next)) {
 321       // List head is now marked so we can safely switch it.
 322       set_next(m, cur);  // m now points to cur (and unmarks m)
 323       OrderAccess::release_store(list_p, m);  // Switch list head to unmarked m.
 324       set_next(cur, next);  // Unmark the previous list head.
 325       break;
 326     }
 327     // The list is empty so try to set the list head.
 328     assert(cur == NULL, "cur must be NULL: cur=" INTPTR_FORMAT, p2i(cur));
 329     set_next(m, cur);  // m now points to NULL (and unmarks m)
 330     if (Atomic::cmpxchg(m, list_p, cur) == cur) {
 331       // List head is now unmarked m.
 332       break;
 333     }
 334     // Implied else: try it all again
 335   }
 336   Atomic::inc(count_p);
 337 }
 338 
 339 // Prepend an ObjectMonitor to a per-thread om_free_list.
 340 // Also updates the per-thread om_free_count.
 341 static void prepend_to_om_free_list(Thread* self, ObjectMonitor* m) {
 342   prepend_to_common(m, &self->om_free_list, &self->om_free_count);
 343 }
 344 
 345 // Prepend an ObjectMonitor to a per-thread om_in_use_list.
 346 // Also updates the per-thread om_in_use_count.
 347 static void prepend_to_om_in_use_list(Thread* self, ObjectMonitor* m) {
 348   prepend_to_common(m, &self->om_in_use_list, &self->om_in_use_count);
 349 }
 350 
 351 // Take an ObjectMonitor from the start of the specified list. Also
 352 // decrements the specified counter. Returns NULL if none are available.
 353 static ObjectMonitor* take_from_start_of_common(ObjectMonitor* volatile * list_p,
 354                                                 int volatile * count_p) {
 355   ObjectMonitor* next = NULL;
 356   ObjectMonitor* take = NULL;
 357   // Mark the list head to guard against A-B-A race:
 358   if (!mark_list_head(list_p, &take, &next)) {
 359     return NULL;  // None are available.
 360   }
 361   // Switch marked list head to next (which unmarks the list head, but
 362   // leaves take marked):
 363   OrderAccess::release_store(list_p, next);
 364   Atomic::dec(count_p);
 365   // Unmark take, but leave the next value for any lagging list
 366   // walkers. It will get cleaned up when take is prepended to
 367   // the in-use list:
 368   set_next(take, next);
 369   return take;
 370 }
 371 
 372 // Take an ObjectMonitor from the start of the global free-list. Also
 373 // updates g_om_free_count. Returns NULL if none are available.
 374 static ObjectMonitor* take_from_start_of_g_free_list() {
 375   return take_from_start_of_common(&g_free_list, &g_om_free_count);
 376 }
 377 
 378 // Take an ObjectMonitor from the start of a per-thread free-list.
 379 // Also updates om_free_count. Returns NULL if none are available.
 380 static ObjectMonitor* take_from_start_of_om_free_list(Thread* self) {
 381   return take_from_start_of_common(&self->om_free_list, &self->om_free_count);
 382 }
 383 
 384 
 385 // =====================> Quick functions
 386 
 387 // The quick_* forms are special fast-path variants used to improve
 388 // performance.  In the simplest case, a "quick_*" implementation could
 389 // simply return false, in which case the caller will perform the necessary
 390 // state transitions and call the slow-path form.
 391 // The fast-path is designed to handle frequently arising cases in an efficient
 392 // manner and is just a degenerate "optimistic" variant of the slow-path.
 393 // returns true  -- to indicate the call was satisfied.
 394 // returns false -- to indicate the call needs the services of the slow-path.
 395 // A no-loitering ordinance is in effect for code in the quick_* family
 396 // operators: safepoints or indefinite blocking (blocking that might span a
 397 // safepoint) are forbidden. Generally the thread_state() is _in_Java upon
 398 // entry.
 399 //
 400 // Consider: An interesting optimization is to have the JIT recognize the
 401 // following common idiom:
 402 //   synchronized (someobj) { .... ; notify(); }
 403 // That is, we find a notify() or notifyAll() call that immediately precedes
 404 // the monitorexit operation.  In that case the JIT could fuse the operations


 443   }
 444 
 445   // biased locking and any other IMS exception states take the slow-path
 446   return false;
 447 }
 448 
 449 
 450 // The LockNode emitted directly at the synchronization site would have
 451 // been too big if it were to have included support for the cases of inflated
 452 // recursive enter and exit, so they go here instead.
 453 // Note that we can't safely call AsyncPrintJavaStack() from within
 454 // quick_enter() as our thread state remains _in_Java.
 455 
 456 bool ObjectSynchronizer::quick_enter(oop obj, Thread* self,
 457                                      BasicLock * lock) {
 458   assert(!SafepointSynchronize::is_at_safepoint(), "invariant");
 459   assert(self->is_Java_thread(), "invariant");
 460   assert(((JavaThread *) self)->thread_state() == _thread_in_Java, "invariant");
 461   NoSafepointVerifier nsv;
 462   if (obj == NULL) return false;       // Need to throw NPE
 463 
 464   while (true) {
 465     const markWord mark = obj->mark();
 466 
 467     if (mark.has_monitor()) {
 468       ObjectMonitorHandle omh;
 469       if (!omh.save_om_ptr(obj, mark)) {
 470         // Lost a race with async deflation so try again.
 471         assert(AsyncDeflateIdleMonitors, "sanity check");
 472         continue;
 473       }
 474       ObjectMonitor* const m = omh.om_ptr();
 475       assert(m->object() == obj, "invariant");
 476       Thread* const owner = (Thread *) m->_owner;
 477 
 478       // Lock contention and Transactional Lock Elision (TLE) diagnostics
 479       // and observability
 480       // Case: light contention possibly amenable to TLE
 481       // Case: TLE inimical operations such as nested/recursive synchronization
 482 
 483       if (owner == self) {
 484         m->_recursions++;
 485         return true;
 486       }
 487 
 488       // This Java Monitor is inflated so obj's header will never be
 489       // displaced to this thread's BasicLock. Make the displaced header
 490       // non-NULL so this BasicLock is not seen as recursive nor as
 491       // being locked. We do this unconditionally so that this thread's
 492       // BasicLock cannot be mis-interpreted by any stack walkers. For
 493       // performance reasons, stack walkers generally first check for
 494       // Biased Locking in the object's header, the second check is for
 495       // stack-locking in the object's header, the third check is for
 496       // recursive stack-locking in the displaced header in the BasicLock,
 497       // and last are the inflated Java Monitor (ObjectMonitor) checks.
 498       lock->set_displaced_header(markWord::unused_mark());
 499 
 500       if (owner == NULL && m->try_set_owner_from(self, NULL) == NULL) {
 501         assert(m->_recursions == 0, "invariant");
 502         return true;
 503       }
 504 
 505       if (AsyncDeflateIdleMonitors &&
 506           m->try_set_owner_from(self, DEFLATER_MARKER) == DEFLATER_MARKER) {
 507         // The deflation protocol finished the first part (setting owner),
 508         // but it failed the second part (making ref_count negative) and
 509         // bailed. Or the ObjectMonitor was async deflated and reused.
 510         // Acquired the monitor.
 511         assert(m->_recursions == 0, "invariant");
 512         return true;
 513       }
 514     }
 515     break;
 516   }
 517 
 518   // Note that we could inflate in quick_enter.
 519   // This is likely a useful optimization
 520   // Critically, in quick_enter() we must not:
 521   // -- perform bias revocation, or
 522   // -- block indefinitely, or
 523   // -- reach a safepoint
 524 
 525   return false;        // revert to slow-path
 526 }
 527 
 528 // -----------------------------------------------------------------------------
 529 // Monitor Enter/Exit
 530 // The interpreter and compiler assembly code tries to lock using the fast path
 531 // of this algorithm. Make sure to update that code if the following function is
 532 // changed. The implementation is extremely sensitive to race condition. Be careful.
 533 
 534 void ObjectSynchronizer::enter(Handle obj, BasicLock* lock, TRAPS) {
 535   if (UseBiasedLocking) {
 536     if (!SafepointSynchronize::is_at_safepoint()) {


 547     // Anticipate successful CAS -- the ST of the displaced mark must
 548     // be visible <= the ST performed by the CAS.
 549     lock->set_displaced_header(mark);
 550     if (mark == obj()->cas_set_mark(markWord::from_pointer(lock), mark)) {
 551       return;
 552     }
 553     // Fall through to inflate() ...
 554   } else if (mark.has_locker() &&
 555              THREAD->is_lock_owned((address)mark.locker())) {
 556     assert(lock != mark.locker(), "must not re-lock the same lock");
 557     assert(lock != (BasicLock*)obj->mark().value(), "don't relock with same BasicLock");
 558     lock->set_displaced_header(markWord::from_pointer(NULL));
 559     return;
 560   }
 561 
 562   // The object header will never be displaced to this lock,
 563   // so it does not matter what the value is, except that it
 564   // must be non-zero to avoid looking like a re-entrant lock,
 565   // and must not look locked either.
 566   lock->set_displaced_header(markWord::unused_mark());
 567   ObjectMonitorHandle omh;
 568   inflate(&omh, THREAD, obj(), inflate_cause_monitor_enter);
 569   omh.om_ptr()->enter(THREAD);
 570 }
 571 
 572 void ObjectSynchronizer::exit(oop object, BasicLock* lock, TRAPS) {
 573   markWord mark = object->mark();
 574   // We cannot check for Biased Locking if we are racing an inflation.
 575   assert(mark == markWord::INFLATING() ||
 576          !mark.has_bias_pattern(), "should not see bias pattern here");
 577 
 578   markWord dhw = lock->displaced_header();
 579   if (dhw.value() == 0) {
 580     // If the displaced header is NULL, then this exit matches up with
 581     // a recursive enter. No real work to do here except for diagnostics.
 582 #ifndef PRODUCT
 583     if (mark != markWord::INFLATING()) {
 584       // Only do diagnostics if we are not racing an inflation. Simply
 585       // exiting a recursive enter of a Java Monitor that is being
 586       // inflated is safe; see the has_monitor() comment below.
 587       assert(!mark.is_neutral(), "invariant");
 588       assert(!mark.has_locker() ||
 589              THREAD->is_lock_owned((address)mark.locker()), "invariant");


 598         // does not own the Java Monitor.
 599         ObjectMonitor* m = mark.monitor();
 600         assert(((oop)(m->object()))->mark() == mark, "invariant");
 601         assert(m->is_entered(THREAD), "invariant");
 602       }
 603     }
 604 #endif
 605     return;
 606   }
 607 
 608   if (mark == markWord::from_pointer(lock)) {
 609     // If the object is stack-locked by the current thread, try to
 610     // swing the displaced header from the BasicLock back to the mark.
 611     assert(dhw.is_neutral(), "invariant");
 612     if (object->cas_set_mark(dhw, mark) == mark) {
 613       return;
 614     }
 615   }
 616 
 617   // We have to take the slow-path of possible inflation and then exit.
 618   ObjectMonitorHandle omh;
 619   inflate(&omh, THREAD, object, inflate_cause_vm_internal);
 620   omh.om_ptr()->exit(true, THREAD);
 621 }
 622 
 623 // -----------------------------------------------------------------------------
 624 // Class Loader  support to workaround deadlocks on the class loader lock objects
 625 // Also used by GC
 626 // complete_exit()/reenter() are used to wait on a nested lock
 627 // i.e. to give up an outer lock completely and then re-enter
 628 // Used when holding nested locks - lock acquisition order: lock1 then lock2
 629 //  1) complete_exit lock1 - saving recursion count
 630 //  2) wait on lock2
 631 //  3) when notified on lock2, unlock lock2
 632 //  4) reenter lock1 with original recursion count
 633 //  5) lock lock2
 634 // NOTE: must use heavy weight monitor to handle complete_exit/reenter()
 635 intptr_t ObjectSynchronizer::complete_exit(Handle obj, TRAPS) {
 636   if (UseBiasedLocking) {
 637     BiasedLocking::revoke(obj, THREAD);
 638     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 639   }
 640 
 641   ObjectMonitorHandle omh;
 642   inflate(&omh, THREAD, obj(), inflate_cause_vm_internal);
 643   intptr_t ret_code = omh.om_ptr()->complete_exit(THREAD);
 644   return ret_code;
 645 }
 646 
 647 // NOTE: must use heavy weight monitor to handle complete_exit/reenter()
 648 void ObjectSynchronizer::reenter(Handle obj, intptr_t recursion, TRAPS) {
 649   if (UseBiasedLocking) {
 650     BiasedLocking::revoke(obj, THREAD);
 651     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 652   }
 653 
 654   ObjectMonitorHandle omh;
 655   inflate(&omh, THREAD, obj(), inflate_cause_vm_internal);
 656   omh.om_ptr()->reenter(recursion, THREAD);
 657 }
 658 // -----------------------------------------------------------------------------
 659 // JNI locks on java objects
 660 // NOTE: must use heavy weight monitor to handle jni monitor enter
 661 void ObjectSynchronizer::jni_enter(Handle obj, TRAPS) {
 662   // the current locking is from JNI instead of Java code
 663   if (UseBiasedLocking) {
 664     BiasedLocking::revoke(obj, THREAD);
 665     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 666   }
 667   THREAD->set_current_pending_monitor_is_from_java(false);
 668   ObjectMonitorHandle omh;
 669   inflate(&omh, THREAD, obj(), inflate_cause_jni_enter);
 670   omh.om_ptr()->enter(THREAD);
 671   THREAD->set_current_pending_monitor_is_from_java(true);
 672 }
 673 
 674 // NOTE: must use heavy weight monitor to handle jni monitor exit
 675 void ObjectSynchronizer::jni_exit(oop obj, Thread* THREAD) {
 676   if (UseBiasedLocking) {
 677     Handle h_obj(THREAD, obj);
 678     BiasedLocking::revoke(h_obj, THREAD);
 679     obj = h_obj();
 680   }
 681   assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 682 
 683   ObjectMonitorHandle omh;
 684   inflate(&omh, THREAD, obj, inflate_cause_jni_exit);
 685   ObjectMonitor* monitor = omh.om_ptr();
 686   // If this thread has locked the object, exit the monitor. We
 687   // intentionally do not use CHECK here because we must exit the
 688   // monitor even if an exception is pending.
 689   if (monitor->check_owner(THREAD)) {
 690     monitor->exit(true, THREAD);
 691   }
 692 }
 693 
 694 // -----------------------------------------------------------------------------
 695 // Internal VM locks on java objects
 696 // standard constructor, allows locking failures
 697 ObjectLocker::ObjectLocker(Handle obj, Thread* thread, bool do_lock) {
 698   _dolock = do_lock;
 699   _thread = thread;
 700   _thread->check_for_valid_safepoint_state();
 701   _obj = obj;
 702 
 703   if (_dolock) {
 704     ObjectSynchronizer::enter(_obj, &_lock, _thread);
 705   }
 706 }
 707 
 708 ObjectLocker::~ObjectLocker() {
 709   if (_dolock) {
 710     ObjectSynchronizer::exit(_obj(), &_lock, _thread);
 711   }
 712 }
 713 
 714 
 715 // -----------------------------------------------------------------------------
 716 //  Wait/Notify/NotifyAll
 717 // NOTE: must use heavy weight monitor to handle wait()
 718 int ObjectSynchronizer::wait(Handle obj, jlong millis, TRAPS) {
 719   if (UseBiasedLocking) {
 720     BiasedLocking::revoke(obj, THREAD);
 721     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 722   }
 723   if (millis < 0) {
 724     THROW_MSG_0(vmSymbols::java_lang_IllegalArgumentException(), "timeout value is negative");
 725   }
 726   ObjectMonitorHandle omh;
 727   inflate(&omh, THREAD, obj(), inflate_cause_wait);
 728   ObjectMonitor* monitor = omh.om_ptr();
 729 
 730   DTRACE_MONITOR_WAIT_PROBE(monitor, obj(), THREAD, millis);
 731   monitor->wait(millis, true, THREAD);
 732 
 733   // This dummy call is in place to get around dtrace bug 6254741.  Once
 734   // that's fixed we can uncomment the following line, remove the call
 735   // and change this function back into a "void" func.
 736   // DTRACE_MONITOR_PROBE(waited, monitor, obj(), THREAD);
 737   int ret_code = dtrace_waited_probe(monitor, obj, THREAD);
 738   return ret_code;
 739 }
 740 
 741 void ObjectSynchronizer::wait_uninterruptibly(Handle obj, jlong millis, TRAPS) {
 742   if (UseBiasedLocking) {
 743     BiasedLocking::revoke(obj, THREAD);
 744     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 745   }
 746   if (millis < 0) {
 747     THROW_MSG(vmSymbols::java_lang_IllegalArgumentException(), "timeout value is negative");
 748   }
 749   ObjectMonitorHandle omh;
 750   inflate(&omh, THREAD, obj(), inflate_cause_wait);
 751   omh.om_ptr()->wait(millis, false, THREAD);
 752 }
 753 
 754 void ObjectSynchronizer::notify(Handle obj, TRAPS) {
 755   if (UseBiasedLocking) {
 756     BiasedLocking::revoke(obj, THREAD);
 757     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 758   }
 759 
 760   markWord mark = obj->mark();
 761   if (mark.has_locker() && THREAD->is_lock_owned((address)mark.locker())) {
 762     return;
 763   }
 764   ObjectMonitorHandle omh;
 765   inflate(&omh, THREAD, obj(), inflate_cause_notify);
 766   omh.om_ptr()->notify(THREAD);
 767 }
 768 
 769 // NOTE: see comment of notify()
 770 void ObjectSynchronizer::notifyall(Handle obj, TRAPS) {
 771   if (UseBiasedLocking) {
 772     BiasedLocking::revoke(obj, THREAD);
 773     assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 774   }
 775 
 776   markWord mark = obj->mark();
 777   if (mark.has_locker() && THREAD->is_lock_owned((address)mark.locker())) {
 778     return;
 779   }
 780   ObjectMonitorHandle omh;
 781   inflate(&omh, THREAD, obj(), inflate_cause_notify);
 782   omh.om_ptr()->notifyAll(THREAD);
 783 }
 784 
 785 // -----------------------------------------------------------------------------
 786 // Hash Code handling
 787 //
 788 // Performance concern:
 789 // OrderAccess::storestore() calls release() which at one time stored 0
 790 // into the global volatile OrderAccess::dummy variable. This store was
 791 // unnecessary for correctness. Many threads storing into a common location
 792 // causes considerable cache migration or "sloshing" on large SMP systems.
 793 // As such, I avoided using OrderAccess::storestore(). In some cases
 794 // OrderAccess::fence() -- which incurs local latency on the executing
 795 // processor -- is a better choice as it scales on SMP systems.
 796 //
 797 // See http://blogs.oracle.com/dave/entry/biased_locking_in_hotspot for
 798 // a discussion of coherency costs. Note that all our current reference
 799 // platforms provide strong ST-ST order, so the issue is moot on IA32,
 800 // x64, and SPARC.
 801 //
 802 // As a general policy we use "volatile" to control compiler-based reordering
 803 // and explicit fences (barriers) to control for architectural reordering
 804 // performed by the CPU(s) or platform.
 805 
 806 struct SharedGlobals {
 807   char         _pad_prefix[OM_CACHE_LINE_SIZE];
 808   // These are highly shared mostly-read variables.
 809   // To avoid false-sharing they need to be the sole occupants of a cache line.
 810   volatile int stw_random;
 811   volatile int stw_cycle;
 812   DEFINE_PAD_MINUS_SIZE(1, OM_CACHE_LINE_SIZE, sizeof(volatile int) * 2);
 813   // Hot RW variable -- Sequester to avoid false-sharing
 814   volatile int hc_sequence;
 815   DEFINE_PAD_MINUS_SIZE(2, OM_CACHE_LINE_SIZE, sizeof(volatile int));
 816 };
 817 
 818 static SharedGlobals GVars;
 819 static int MonitorScavengeThreshold = 1000000;
 820 static volatile int ForceMonitorScavenge = 0; // Scavenge required and pending
 821 
 822 static markWord read_stable_mark(oop obj) {
 823   markWord mark = obj->mark();
 824   if (!mark.is_being_inflated()) {
 825     return mark;       // normal fast-path return
 826   }
 827 
 828   int its = 0;
 829   for (;;) {
 830     markWord mark = obj->mark();
 831     if (!mark.is_being_inflated()) {
 832       return mark;    // normal fast-path return
 833     }
 834 
 835     // The object is being inflated by some other thread.


 956       Handle hobj(self, obj);
 957       // Relaxing assertion for bug 6320749.
 958       assert(Universe::verify_in_progress() ||
 959              !SafepointSynchronize::is_at_safepoint(),
 960              "biases should not be seen by VM thread here");
 961       BiasedLocking::revoke(hobj, JavaThread::current());
 962       obj = hobj();
 963       assert(!obj->mark().has_bias_pattern(), "biases should be revoked by now");
 964     }
 965   }
 966 
 967   // hashCode() is a heap mutator ...
 968   // Relaxing assertion for bug 6320749.
 969   assert(Universe::verify_in_progress() || DumpSharedSpaces ||
 970          !SafepointSynchronize::is_at_safepoint(), "invariant");
 971   assert(Universe::verify_in_progress() || DumpSharedSpaces ||
 972          self->is_Java_thread() , "invariant");
 973   assert(Universe::verify_in_progress() || DumpSharedSpaces ||
 974          ((JavaThread *)self)->thread_state() != _thread_blocked, "invariant");
 975 
 976   while (true) {
 977     ObjectMonitor* monitor = NULL;
 978     markWord temp, test;
 979     intptr_t hash;
 980     markWord mark = read_stable_mark(obj);
 981 
 982     // object should remain ineligible for biased locking
 983     assert(!mark.has_bias_pattern(), "invariant");
 984 
 985     if (mark.is_neutral()) {
 986       hash = mark.hash();              // this is a normal header
 987       if (hash != 0) {                  // if it has hash, just return it
 988         return hash;
 989       }
 990       hash = get_next_hash(self, obj);  // allocate a new hash code
 991       temp = mark.copy_set_hash(hash); // merge the hash code into header
 992       // use (machine word version) atomic operation to install the hash
 993       test = obj->cas_set_mark(temp, mark);
 994       if (test == mark) {
 995         return hash;
 996       }
 997       // If atomic operation failed, we must inflate the header
 998       // into heavy weight monitor. We could add more code here
 999       // for fast path, but it does not worth the complexity.
1000     } else if (mark.has_monitor()) {
1001       ObjectMonitorHandle omh;
1002       if (!omh.save_om_ptr(obj, mark)) {
1003         // Lost a race with async deflation so try again.
1004         assert(AsyncDeflateIdleMonitors, "sanity check");
1005         continue;
1006       }
1007       monitor = omh.om_ptr();
1008       temp = monitor->header();
1009       assert(temp.is_neutral(), "invariant: header=" INTPTR_FORMAT, temp.value());
1010       hash = temp.hash();
1011       if (hash != 0) {
1012         return hash;
1013       }
1014       // Skip to the following code to reduce code size
1015     } else if (self->is_lock_owned((address)mark.locker())) {
1016       temp = mark.displaced_mark_helper(); // this is a lightweight monitor owned
1017       assert(temp.is_neutral(), "invariant: header=" INTPTR_FORMAT, temp.value());
1018       hash = temp.hash();              // by current thread, check if the displaced
1019       if (hash != 0) {                  // header contains hash code
1020         return hash;
1021       }
1022       // WARNING:
1023       // The displaced header in the BasicLock on a thread's stack
1024       // is strictly immutable. It CANNOT be changed in ANY cases.
1025       // So we have to inflate the stack lock into an ObjectMonitor
1026       // even if the current thread owns the lock. The BasicLock on
1027       // a thread's stack can be asynchronously read by other threads
1028       // during an inflate() call so any change to that stack memory
1029       // may not propagate to other threads correctly.
1030     }
1031 
1032     // Inflate the monitor to set hash code
1033     ObjectMonitorHandle omh;
1034     inflate(&omh, self, obj, inflate_cause_hash_code);
1035     monitor = omh.om_ptr();
1036     // Load displaced header and check it has hash code
1037     mark = monitor->header();
1038     assert(mark.is_neutral(), "invariant: header=" INTPTR_FORMAT, mark.value());
1039     hash = mark.hash();
1040     if (hash == 0) {
1041       hash = get_next_hash(self, obj);
1042       temp = mark.copy_set_hash(hash); // merge hash code into header
1043       assert(temp.is_neutral(), "invariant: header=" INTPTR_FORMAT, temp.value());
1044       uintptr_t v = Atomic::cmpxchg(temp.value(), (volatile uintptr_t*)monitor->header_addr(), mark.value());
1045       test = markWord(v);
1046       if (test != mark) {
1047         // The only non-deflation update to the ObjectMonitor's
1048         // header/dmw field is to merge in the hash code. If someone
1049         // adds a new usage of the header/dmw field, please update
1050         // this code.
1051         // ObjectMonitor::install_displaced_markword_in_object()
1052         // does mark the header/dmw field as part of async deflation,
1053         // but that protocol cannot happen now due to the
1054         // ObjectMonitorHandle above.
1055         hash = test.hash();
1056         assert(test.is_neutral(), "invariant: header=" INTPTR_FORMAT, test.value());
1057         assert(hash != 0, "Trivial unexpected object/monitor header usage.");
1058       }
1059     }
1060     // We finally get the hash
1061     return hash;
1062   }
1063 }
1064 
1065 // Deprecated -- use FastHashCode() instead.
1066 
1067 intptr_t ObjectSynchronizer::identity_hash_value_for(Handle obj) {
1068   return FastHashCode(Thread::current(), obj());
1069 }
1070 
1071 
1072 bool ObjectSynchronizer::current_thread_holds_lock(JavaThread* thread,
1073                                                    Handle h_obj) {
1074   if (UseBiasedLocking) {
1075     BiasedLocking::revoke(h_obj, thread);
1076     assert(!h_obj->mark().has_bias_pattern(), "biases should be revoked by now");
1077   }
1078 
1079   assert(thread == JavaThread::current(), "Can only be called on current thread");
1080   oop obj = h_obj();
1081 
1082   while (true) {
1083     markWord mark = read_stable_mark(obj);
1084 
1085     // Uncontended case, header points to stack
1086     if (mark.has_locker()) {
1087       return thread->is_lock_owned((address)mark.locker());
1088     }
1089     // Contended case, header points to ObjectMonitor (tagged pointer)
1090     if (mark.has_monitor()) {
1091       ObjectMonitorHandle omh;
1092       if (!omh.save_om_ptr(obj, mark)) {
1093         // Lost a race with async deflation so try again.
1094         assert(AsyncDeflateIdleMonitors, "sanity check");
1095         continue;
1096       }
1097       bool ret_code = omh.om_ptr()->is_entered(thread) != 0;
1098       return ret_code;
1099     }
1100     // Unlocked case, header in place
1101     assert(mark.is_neutral(), "sanity check");
1102     return false;
1103   }
1104 }
1105 
1106 // Be aware of this method could revoke bias of the lock object.
1107 // This method queries the ownership of the lock handle specified by 'h_obj'.
1108 // If the current thread owns the lock, it returns owner_self. If no
1109 // thread owns the lock, it returns owner_none. Otherwise, it will return
1110 // owner_other.
1111 ObjectSynchronizer::LockOwnership ObjectSynchronizer::query_lock_ownership
1112 (JavaThread *self, Handle h_obj) {
1113   // The caller must beware this method can revoke bias, and
1114   // revocation can result in a safepoint.
1115   assert(!SafepointSynchronize::is_at_safepoint(), "invariant");
1116   assert(self->thread_state() != _thread_blocked, "invariant");
1117 
1118   // Possible mark states: neutral, biased, stack-locked, inflated
1119 
1120   if (UseBiasedLocking && h_obj()->mark().has_bias_pattern()) {
1121     // CASE: biased
1122     BiasedLocking::revoke(h_obj, self);
1123     assert(!h_obj->mark().has_bias_pattern(),
1124            "biases should be revoked by now");
1125   }
1126 
1127   assert(self == JavaThread::current(), "Can only be called on current thread");
1128   oop obj = h_obj();
1129 
1130   while (true) {
1131     markWord mark = read_stable_mark(obj);
1132 
1133     // CASE: stack-locked.  Mark points to a BasicLock on the owner's stack.
1134     if (mark.has_locker()) {
1135       return self->is_lock_owned((address)mark.locker()) ?
1136         owner_self : owner_other;
1137     }
1138 
1139     // CASE: inflated. Mark (tagged pointer) points to an ObjectMonitor.
1140     // The Object:ObjectMonitor relationship is stable as long as we're
1141     // not at a safepoint and AsyncDeflateIdleMonitors is false.
1142     if (mark.has_monitor()) {
1143       ObjectMonitorHandle omh;
1144       if (!omh.save_om_ptr(obj, mark)) {
1145         // Lost a race with async deflation so try again.
1146         assert(AsyncDeflateIdleMonitors, "sanity check");
1147         continue;
1148       }
1149       ObjectMonitor* monitor = omh.om_ptr();
1150       void* owner = monitor->_owner;
1151       if (owner == NULL) return owner_none;
1152       return (owner == self ||
1153               self->is_lock_owned((address)owner)) ? owner_self : owner_other;
1154     }
1155 
1156     // CASE: neutral
1157     assert(mark.is_neutral(), "sanity check");
1158     return owner_none;           // it's unlocked
1159   }
1160 }
1161 
1162 // FIXME: jvmti should call this
1163 JavaThread* ObjectSynchronizer::get_lock_owner(ThreadsList * t_list, Handle h_obj) {
1164   if (UseBiasedLocking) {
1165     if (SafepointSynchronize::is_at_safepoint()) {
1166       BiasedLocking::revoke_at_safepoint(h_obj);
1167     } else {
1168       BiasedLocking::revoke(h_obj, JavaThread::current());
1169     }
1170     assert(!h_obj->mark().has_bias_pattern(), "biases should be revoked by now");
1171   }
1172 
1173   oop obj = h_obj();

1174 
1175   while (true) {
1176     address owner = NULL;
1177     markWord mark = read_stable_mark(obj);
1178 
1179     // Uncontended case, header points to stack
1180     if (mark.has_locker()) {
1181       owner = (address) mark.locker();
1182     }
1183 
1184     // Contended case, header points to ObjectMonitor (tagged pointer)
1185     else if (mark.has_monitor()) {
1186       ObjectMonitorHandle omh;
1187       if (!omh.save_om_ptr(obj, mark)) {
1188         // Lost a race with async deflation so try again.
1189         assert(AsyncDeflateIdleMonitors, "sanity check");
1190         continue;
1191       }
1192       ObjectMonitor* monitor = omh.om_ptr();
1193       assert(monitor != NULL, "monitor should be non-null");
1194       owner = (address) monitor->owner();
1195     }
1196 
1197     if (owner != NULL) {
1198       // owning_thread_from_monitor_owner() may also return NULL here
1199       return Threads::owning_thread_from_monitor_owner(t_list, owner);
1200     }
1201 
1202     // Unlocked case, header in place
1203     // Cannot have assertion since this object may have been
1204     // locked by another thread when reaching here.
1205     // assert(mark.is_neutral(), "sanity check");
1206 
1207     return NULL;
1208   }
1209 }
1210 
1211 // Visitors ...
1212 
1213 void ObjectSynchronizer::monitors_iterate(MonitorClosure* closure) {
1214   PaddedObjectMonitor* block = OrderAccess::load_acquire(&g_block_list);
1215   while (block != NULL) {
1216     assert(block->object() == CHAINMARKER, "must be a block header");
1217     for (int i = _BLOCKSIZE - 1; i > 0; i--) {
1218       ObjectMonitor* mid = (ObjectMonitor *)(block + i);
1219       if (mid->is_active()) {
1220         ObjectMonitorHandle omh(mid);
1221 
1222         if (mid->object() == NULL ||
1223             (AsyncDeflateIdleMonitors && mid->ref_count() < 0)) {
1224           // Only process with closure if the object is set.
1225           // For async deflation, race here if monitor is not owned!
1226           // The above ref_count bump (in ObjectMonitorHandle ctr)
1227           // will cause subsequent async deflation to skip it.
1228           // However, previous or concurrent async deflation is a race
1229           // so skip this ObjectMonitor if it is being async deflated.
1230           continue;
1231         }
1232         closure->do_monitor(mid);
1233       }
1234     }
1235     // unmarked_next() is not needed with g_block_list (no next field marking).
1236     block = (PaddedObjectMonitor*)OrderAccess::load_acquire(&block->_next_om);
1237   }
1238 }
1239 
1240 static bool monitors_used_above_threshold() {
1241   if (OrderAccess::load_acquire(&g_om_population) == 0) {
1242     return false;
1243   }
1244   if (MonitorUsedDeflationThreshold > 0) {
1245     int monitors_used = OrderAccess::load_acquire(&g_om_population) -
1246                         OrderAccess::load_acquire(&g_om_free_count);
1247     if (HandshakeAfterDeflateIdleMonitors) {
1248       monitors_used -= OrderAccess::load_acquire(&g_om_wait_count);
1249     }
1250     int monitor_usage = (monitors_used * 100LL) /
1251                         OrderAccess::load_acquire(&g_om_population);
1252     return monitor_usage > MonitorUsedDeflationThreshold;
1253   }
1254   return false;
1255 }
1256 
1257 // Returns true if MonitorBound is set (> 0) and if the specified
1258 // cnt is > MonitorBound. Otherwise returns false.
1259 static bool is_MonitorBound_exceeded(const int cnt) {
1260   const int mx = MonitorBound;
1261   return mx > 0 && cnt > mx;
1262 }
1263 
1264 bool ObjectSynchronizer::is_async_deflation_needed() {
1265   if (!AsyncDeflateIdleMonitors) {
1266     return false;
1267   }
1268   if (is_async_deflation_requested()) {
1269     // Async deflation request.
1270     return true;
1271   }
1272   if (AsyncDeflationInterval > 0 &&
1273       time_since_last_async_deflation_ms() > AsyncDeflationInterval &&
1274       monitors_used_above_threshold()) {
1275     // It's been longer than our specified deflate interval and there
1276     // are too many monitors in use. We don't deflate more frequently
1277     // than AsyncDeflationInterval (unless is_async_deflation_requested)
1278     // in order to not swamp the ServiceThread.
1279     _last_async_deflation_time_ns = os::javaTimeNanos();
1280     return true;
1281   }
1282   int monitors_used = OrderAccess::load_acquire(&g_om_population) -
1283                       OrderAccess::load_acquire(&g_om_free_count);
1284   if (HandshakeAfterDeflateIdleMonitors) {
1285     monitors_used -= OrderAccess::load_acquire(&g_om_wait_count);
1286   }
1287   if (is_MonitorBound_exceeded(monitors_used)) {
1288     // Not enough ObjectMonitors on the global free list.
1289     return true;
1290   }
1291   return false;
1292 }
1293 
1294 bool ObjectSynchronizer::is_safepoint_deflation_needed() {
1295   if (!AsyncDeflateIdleMonitors) {
1296     if (monitors_used_above_threshold()) {
1297       // Too many monitors in use.
1298       return true;
1299     }
1300     return false;
1301   }
1302   if (is_special_deflation_requested()) {
1303     // For AsyncDeflateIdleMonitors only do a safepoint deflation
1304     // if there is a special deflation request.
1305     return true;
1306   }
1307   return false;
1308 }
1309 
1310 jlong ObjectSynchronizer::time_since_last_async_deflation_ms() {
1311   return (os::javaTimeNanos() - _last_async_deflation_time_ns) / (NANOUNITS / MILLIUNITS);
1312 }
1313 
1314 void ObjectSynchronizer::oops_do(OopClosure* f) {
1315   // We only scan the global used list here (for moribund threads), and
1316   // the thread-local monitors in Thread::oops_do().
1317   global_used_oops_do(f);
1318 }
1319 
1320 void ObjectSynchronizer::global_used_oops_do(OopClosure* f) {
1321   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
1322   list_oops_do(OrderAccess::load_acquire(&g_om_in_use_list), OrderAccess::load_acquire(&g_om_in_use_count), f);
1323 }
1324 
1325 void ObjectSynchronizer::thread_local_used_oops_do(Thread* thread, OopClosure* f) {
1326   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
1327   list_oops_do(OrderAccess::load_acquire(&thread->om_in_use_list), OrderAccess::load_acquire(&thread->om_in_use_count), f);
1328 }
1329 
1330 void ObjectSynchronizer::list_oops_do(ObjectMonitor* list, int count, OopClosure* f) {
1331   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
1332   // The oops_do() phase does not overlap with monitor deflation
1333   // so no need to update the ObjectMonitor's ref_count for this
1334   // ObjectMonitor* use.
1335   for (ObjectMonitor* mid = list; mid != NULL; mid = unmarked_next(mid)) {
1336     if (mid->object() != NULL) {
1337       f->do_oop((oop*)mid->object_addr());
1338     }
1339   }
1340 }
1341 
1342 
1343 // -----------------------------------------------------------------------------
1344 // ObjectMonitor Lifecycle
1345 // -----------------------
1346 // Inflation unlinks monitors from the global g_free_list and
1347 // associates them with objects.  Deflation -- which occurs at
1348 // STW-time -- disassociates idle monitors from objects.  Such
1349 // scavenged monitors are returned to the g_free_list.
1350 //



1351 // ObjectMonitors reside in type-stable memory (TSM) and are immortal.
1352 //
1353 // Lifecycle:
1354 // --   unassigned and on the global free list
1355 // --   unassigned and on a thread's private om_free_list
1356 // --   assigned to an object.  The object is inflated and the mark refers
1357 //      to the objectmonitor.
1358 
1359 
1360 // Constraining monitor pool growth via MonitorBound ...
1361 //
1362 // If MonitorBound is not set (<= 0), MonitorBound checks are disabled.
1363 //
1364 // When safepoint deflation is being used (!AsyncDeflateIdleMonitors):
1365 // The monitor pool is grow-only.  We scavenge at STW safepoint-time, but the
1366 // the rate of scavenging is driven primarily by GC.  As such,  we can find
1367 // an inordinate number of monitors in circulation.
1368 // To avoid that scenario we can artificially induce a STW safepoint
1369 // if the pool appears to be growing past some reasonable bound.
1370 // Generally we favor time in space-time tradeoffs, but as there's no
1371 // natural back-pressure on the # of extant monitors we need to impose some
1372 // type of limit.  Beware that if MonitorBound is set to too low a value
1373 // we could just loop. In addition, if MonitorBound is set to a low value
1374 // we'll incur more safepoints, which are harmful to performance.
1375 // See also: GuaranteedSafepointInterval
1376 //
1377 // The current implementation uses asynchronous VM operations.
1378 //
1379 // When safepoint deflation is being used and MonitorBound is set, the
1380 // boundry applies to
1381 //     (g_om_population - g_om_free_count)
1382 // i.e., if there are not enough ObjectMonitors on the global free list,
1383 // then a safepoint deflation is induced. Picking a good MonitorBound value
1384 // is non-trivial.
1385 //
1386 // When async deflation is being used:
1387 // The monitor pool is still grow-only. Async deflation is requested
1388 // by a safepoint's cleanup phase or by the ServiceThread at periodic
1389 // intervals when is_async_deflation_needed() returns true. In
1390 // addition to other policies that are checked, if there are not
1391 // enough ObjectMonitors on the global free list, then
1392 // is_async_deflation_needed() will return true. The ServiceThread
1393 // calls deflate_global_idle_monitors_using_JT() and also calls
1394 // deflate_per_thread_idle_monitors_using_JT() as needed.
1395 
1396 static void InduceScavenge(Thread* self, const char * Whence) {
1397   assert(!AsyncDeflateIdleMonitors, "is not used by async deflation");
1398 
1399   // Induce STW safepoint to trim monitors
1400   // Ultimately, this results in a call to deflate_idle_monitors() in the near future.
1401   // More precisely, trigger an asynchronous STW safepoint as the number
1402   // of active monitors passes the specified threshold.
1403   // TODO: assert thread state is reasonable
1404 
1405   if (ForceMonitorScavenge == 0 && Atomic::xchg (1, &ForceMonitorScavenge) == 0) {
1406     // Induce a 'null' safepoint to scavenge monitors
1407     // Must VM_Operation instance be heap allocated as the op will be enqueue and posted
1408     // to the VMthread and have a lifespan longer than that of this activation record.
1409     // The VMThread will delete the op when completed.
1410     VMThread::execute(new VM_ScavengeMonitors());
1411   }
1412 }
1413 
1414 ObjectMonitor* ObjectSynchronizer::om_alloc(Thread* self,
1415                                            const InflateCause cause) {
1416   // A large MAXPRIVATE value reduces both list lock contention
1417   // and list coherency traffic, but also tends to increase the
1418   // number of ObjectMonitors in circulation as well as the STW
1419   // scavenge costs.  As usual, we lean toward time in space-time
1420   // tradeoffs.
1421   const int MAXPRIVATE = 1024;
1422 
1423   stringStream ss;
1424   for (;;) {
1425     ObjectMonitor* m;
1426 
1427     // 1: try to allocate from the thread's local om_free_list.
1428     // Threads will attempt to allocate first from their local list, then
1429     // from the global list, and only after those attempts fail will the
1430     // thread attempt to instantiate new monitors. Thread-local free lists
1431     // improve allocation latency, as well as reducing coherency traffic
1432     // on the shared global list.
1433     m = take_from_start_of_om_free_list(self);
1434     if (m != NULL) {


1435       guarantee(m->object() == NULL, "invariant");
1436       m->set_allocation_state(ObjectMonitor::New);
1437       prepend_to_om_in_use_list(self, m);

1438       return m;
1439     }
1440 
1441     // 2: try to allocate from the global g_free_list
1442     // CONSIDER: use muxTry() instead of muxAcquire().
1443     // If the muxTry() fails then drop immediately into case 3.
1444     // If we're using thread-local free lists then try
1445     // to reprovision the caller's free list.
1446     if (OrderAccess::load_acquire(&g_free_list) != NULL) {
1447       // Reprovision the thread's om_free_list.
1448       // Use bulk transfers to reduce the allocation rate and heat
1449       // on various locks.
1450       for (int i = self->om_free_provision; --i >= 0;) {
1451         ObjectMonitor* take = take_from_start_of_g_free_list();
1452         if (take == NULL) {
1453           break;  // No more are available.
1454         }
1455         guarantee(take->object() == NULL, "invariant");
1456         if (AsyncDeflateIdleMonitors) {
1457           // We allowed 3 field values to linger during async deflation.
1458           // We clear header and restore ref_count here, but we leave
1459           // owner == DEFLATER_MARKER so the simple C2 ObjectMonitor
1460           // enter optimization can no longer race with async deflation
1461           // and reuse.
1462           take->set_header(markWord::zero());
1463           if (take->ref_count() < 0) {
1464             // Add back max_jint to restore the ref_count field to its
1465             // proper value.
1466             Atomic::add(max_jint, &take->_ref_count);
1467 
1468             assert(take->ref_count() >= 0, "must not be negative: ref_count=%d",
1469                    take->ref_count());
1470           }
1471         }
1472         take->Recycle();
1473         // Since we're taking from the global free-list, take must be Free.
1474         // om_release() also sets the allocation state to Free because it
1475         // is called from other code paths.
1476         assert(take->is_free(), "invariant");
1477         om_release(self, take, false);
1478       }

1479       self->om_free_provision += 1 + (self->om_free_provision/2);
1480       if (self->om_free_provision > MAXPRIVATE) self->om_free_provision = MAXPRIVATE;
1481 
1482       if (!AsyncDeflateIdleMonitors &&
1483           is_MonitorBound_exceeded(OrderAccess::load_acquire(&g_om_population) -
1484                                    OrderAccess::load_acquire(&g_om_free_count))) {
1485         // Not enough ObjectMonitors on the global free list.
1486         // We can't safely induce a STW safepoint from om_alloc() as our thread
1487         // state may not be appropriate for such activities and callers may hold
1488         // naked oops, so instead we defer the action.
1489         InduceScavenge(self, "om_alloc");
1490       }
1491       continue;
1492     }
1493 
1494     // 3: allocate a block of new ObjectMonitors
1495     // Both the local and global free lists are empty -- resort to malloc().
1496     // In the current implementation ObjectMonitors are TSM - immortal.
1497     // Ideally, we'd write "new ObjectMonitor[_BLOCKSIZE], but we want
1498     // each ObjectMonitor to start at the beginning of a cache line,
1499     // so we use align_up().
1500     // A better solution would be to use C++ placement-new.
1501     // BEWARE: As it stands currently, we don't run the ctors!
1502     assert(_BLOCKSIZE > 1, "invariant");
1503     size_t neededsize = sizeof(PaddedObjectMonitor) * _BLOCKSIZE;
1504     PaddedObjectMonitor* temp;
1505     size_t aligned_size = neededsize + (OM_CACHE_LINE_SIZE - 1);
1506     void* real_malloc_addr = NEW_C_HEAP_ARRAY(char, aligned_size, mtInternal);
1507     temp = (PaddedObjectMonitor*)align_up(real_malloc_addr, OM_CACHE_LINE_SIZE);
1508     (void)memset((void *) temp, 0, neededsize);
1509 
1510     // Format the block.
1511     // initialize the linked list, each monitor points to its next
1512     // forming the single linked free list, the very first monitor
1513     // will points to next block, which forms the block list.
1514     // The trick of using the 1st element in the block as g_block_list
1515     // linkage should be reconsidered.  A better implementation would
1516     // look like: class Block { Block * next; int N; ObjectMonitor Body [N] ; }
1517 
1518     for (int i = 1; i < _BLOCKSIZE; i++) {
1519       OrderAccess::release_store(&temp[i]._next_om, (ObjectMonitor*)&temp[i+1]);
1520       assert(temp[i].is_free(), "invariant");
1521     }
1522 
1523     // terminate the last monitor as the end of list
1524     OrderAccess::release_store(&temp[_BLOCKSIZE - 1]._next_om, (ObjectMonitor*)NULL);
1525 
1526     // Element [0] is reserved for global list linkage
1527     temp[0].set_object(CHAINMARKER);
1528 
1529     // Consider carving out this thread's current request from the
1530     // block in hand.  This avoids some lock traffic and redundant
1531     // list activity.
1532 
1533     prepend_block_to_lists(temp);

















1534   }
1535 }
1536 
1537 // Place "m" on the caller's private per-thread om_free_list.
1538 // In practice there's no need to clamp or limit the number of
1539 // monitors on a thread's om_free_list as the only non-allocation time
1540 // we'll call om_release() is to return a monitor to the free list after
1541 // a CAS attempt failed. This doesn't allow unbounded #s of monitors to
1542 // accumulate on a thread's free list.
1543 //
1544 // Key constraint: all ObjectMonitors on a thread's free list and the global
1545 // free list must have their object field set to null. This prevents the
1546 // scavenger -- deflate_monitor_list() or deflate_monitor_list_using_JT()
1547 // -- from reclaiming them while we are trying to release them.
1548 
1549 void ObjectSynchronizer::om_release(Thread* self, ObjectMonitor* m,
1550                                     bool from_per_thread_alloc) {
1551   guarantee(m->header().value() == 0, "invariant");
1552   guarantee(m->object() == NULL, "invariant");
1553   stringStream ss;
1554   guarantee((m->is_busy() | m->_recursions) == 0, "freeing in-use monitor: "
1555             "%s, recursions=" INTX_FORMAT, m->is_busy_to_string(&ss),
1556             m->_recursions);
1557   m->set_allocation_state(ObjectMonitor::Free);
1558   // _next_om is used for both per-thread in-use and free lists so
1559   // we have to remove 'm' from the in-use list first (as needed).
1560   if (from_per_thread_alloc) {
1561     // Need to remove 'm' from om_in_use_list.
1562     // We use the more complicated mark-cur_mid_in_use-and-mid-as-we-go
1563     // protocol because async deflation can do list deletions in parallel.
1564     ObjectMonitor* cur_mid_in_use = NULL;
1565     ObjectMonitor* mid = NULL;
1566     ObjectMonitor* next = NULL;
1567     bool extracted = false;
1568 
1569     if (!mark_list_head(&self->om_in_use_list, &mid, &next)) {
1570       fatal("thread=" INTPTR_FORMAT " in-use list must not be empty.", p2i(self));
1571     }
1572     while (true) {
1573       if (m == mid) {
1574         // We found 'm' on the per-thread in-use list so try to extract it.
1575         if (cur_mid_in_use == NULL) {
1576           // mid is the list head and it is marked. Switch the list head
1577           // to next which unmarks the list head, but leaves mid marked:
1578           OrderAccess::release_store(&self->om_in_use_list, next);
1579         } else {
1580           // mid and cur_mid_in_use are marked. Switch cur_mid_in_use's
1581           // next field to next which unmarks cur_mid_in_use, but leaves
1582           // mid marked:
1583           OrderAccess::release_store(&cur_mid_in_use->_next_om, next);
1584         }
1585         extracted = true;
1586         Atomic::dec(&self->om_in_use_count);
1587         // Unmark mid, but leave the next value for any lagging list
1588         // walkers. It will get cleaned up when mid is prepended to
1589         // the thread's free list:
1590         set_next(mid, next);
1591         break;
1592       }
1593       if (cur_mid_in_use != NULL) {
1594         set_next(cur_mid_in_use, mid);  // umark cur_mid_in_use
1595       }
1596       // The next cur_mid_in_use keeps mid's marked next field so
1597       // that it is stable for a possible next field change. It
1598       // cannot be deflated while it is marked.
1599       cur_mid_in_use = mid;
1600       mid = next;
1601       if (mid == NULL) {
1602         // Reached end of the list and didn't find m so:
1603         fatal("must find m=" INTPTR_FORMAT "on om_in_use_list=" INTPTR_FORMAT,
1604               p2i(m), p2i(self->om_in_use_list));
1605       }
1606       // Mark mid's next field so we can possibly extract it:
1607       next = mark_next_loop(mid);
1608     }

1609   }
1610 
1611   prepend_to_om_free_list(self, m);
1612   guarantee(m->is_free(), "invariant");

1613 }
1614 
1615 // Return ObjectMonitors on a moribund thread's free and in-use
1616 // lists to the appropriate global lists. The ObjectMonitors on the
1617 // per-thread in-use list may still be in use by other threads.
1618 //
1619 // We currently call om_flush() from Threads::remove() before the
1620 // thread has been excised from the thread list and is no longer a
1621 // mutator. This means that om_flush() cannot run concurrently with
1622 // a safepoint and interleave with deflate_idle_monitors(). In
1623 // particular, this ensures that the thread's in-use monitors are
1624 // scanned by a GC safepoint, either via Thread::oops_do() (before
1625 // om_flush() is called) or via ObjectSynchronizer::oops_do() (after
1626 // om_flush() is called).
1627 //
1628 // With AsyncDeflateIdleMonitors, deflate_global_idle_monitors_using_JT()
1629 // and deflate_per_thread_idle_monitors_using_JT() (in another thread) can
1630 // run at the same time as om_flush() so we have to follow a careful
1631 // protocol to prevent list corruption.
1632 
1633 void ObjectSynchronizer::om_flush(Thread* self) {
1634   // This function can race with an async deflater thread. Since
1635   // deflation has to process the per-thread in-use list before
1636   // prepending the deflated ObjectMonitors to the global free list,
1637   // we process the per-thread lists in the same order to prevent
1638   // ordering races.
1639   int in_use_count = 0;
1640   ObjectMonitor* in_use_list = NULL;
1641   ObjectMonitor* in_use_tail = NULL;
1642   ObjectMonitor* next = NULL;
1643 
1644   // An async deflation thread checks to see if the target thread
1645   // is exiting, but if it has made it past that check before we
1646   // started exiting, then it is racing to get to the in-use list.
1647   if (mark_list_head(&self->om_in_use_list, &in_use_list, &next)) {
1648     // At this point, we have marked the in-use list head so an
1649     // async deflation thread cannot come in after us. If an async
1650     // deflation thread is ahead of us, then we'll detect that and
1651     // wait for it to finish its work.
1652     //
1653     // The thread is going away, however the ObjectMonitors on the
1654     // om_in_use_list may still be in-use by other threads. Link
1655     // them to in_use_tail, which will be linked into the global
1656     // in-use list g_om_in_use_list below.
1657     //
1658     // Account for the in-use list head before the loop since it is
1659     // already marked (by this thread):
1660     in_use_tail = in_use_list;
1661     in_use_count++;
1662     for (ObjectMonitor* cur_om = unmarked_next(in_use_list); cur_om != NULL;) {
1663       if (is_next_marked(cur_om)) {
1664         // This next field is marked so there must be an async deflater
1665         // thread ahead of us so we'll give it a chance to finish.
1666         while (is_next_marked(cur_om)) {
1667           os::naked_short_sleep(1);
1668         }
1669         // Refetch the possibly changed next field and try again.
1670         cur_om = unmarked_next(in_use_tail);
1671         continue;
1672       }
1673       if (!cur_om->is_active()) {
1674         // cur_om was deflated and the allocation state was changed
1675         // to Free while it was marked. We happened to see it just
1676         // after it was unmarked (and added to the free list).
1677         // Refetch the possibly changed next field and try again.
1678         cur_om = unmarked_next(in_use_tail);
1679         continue;
1680       }
1681       in_use_tail = cur_om;
1682       in_use_count++;
1683       cur_om = unmarked_next(cur_om);
1684     }
1685     guarantee(in_use_tail != NULL, "invariant");
1686     int l_om_in_use_count = OrderAccess::load_acquire(&self->om_in_use_count);
1687     ADIM_guarantee(l_om_in_use_count == in_use_count, "in-use counts don't "
1688                    "match: l_om_in_use_count=%d, in_use_count=%d",
1689                    l_om_in_use_count, in_use_count);
1690     // Clear the in-use count before unmarking the in-use list head
1691     // to avoid races:
1692     OrderAccess::release_store(&self->om_in_use_count, 0);
1693     // Clear the in-use list head (which also unmarks it):
1694     OrderAccess::release_store(&self->om_in_use_list, (ObjectMonitor*)NULL);
1695     // Unmark the disconnected list head:
1696     set_next(in_use_list, next);
1697   }
1698 
1699   int free_count = 0;
1700   ObjectMonitor* free_list = OrderAccess::load_acquire(&self->om_free_list);
1701   ObjectMonitor* free_tail = NULL;
1702   if (free_list != NULL) {

1703     // The thread is going away. Set 'free_tail' to the last per-thread free
1704     // monitor which will be linked to g_free_list below.
1705     stringStream ss;
1706     for (ObjectMonitor* s = free_list; s != NULL; s = unmarked_next(s)) {
1707       free_count++;
1708       free_tail = s;
1709       guarantee(s->object() == NULL, "invariant");
1710       guarantee(!s->is_busy(), "must be !is_busy: %s", s->is_busy_to_string(&ss));
1711     }
1712     guarantee(free_tail != NULL, "invariant");
1713     int l_om_free_count = OrderAccess::load_acquire(&self->om_free_count);
1714     ADIM_guarantee(l_om_free_count == free_count, "free counts don't match: "
1715                    "l_om_free_count=%d, free_count=%d", l_om_free_count,
1716                    free_count);
1717     OrderAccess::release_store(&self->om_free_list, (ObjectMonitor*)NULL);
1718     OrderAccess::release_store(&self->om_free_count, 0);
1719   }
1720 




















1721   if (free_tail != NULL) {
1722     prepend_list_to_g_free_list(free_list, free_tail, free_count);


1723   }
1724 
1725   if (in_use_tail != NULL) {
1726     prepend_list_to_g_om_in_use_list(in_use_list, in_use_tail, in_use_count);


1727   }
1728 


1729   LogStreamHandle(Debug, monitorinflation) lsh_debug;
1730   LogStreamHandle(Info, monitorinflation) lsh_info;
1731   LogStream* ls = NULL;
1732   if (log_is_enabled(Debug, monitorinflation)) {
1733     ls = &lsh_debug;
1734   } else if ((free_count != 0 || in_use_count != 0) &&
1735              log_is_enabled(Info, monitorinflation)) {
1736     ls = &lsh_info;
1737   }
1738   if (ls != NULL) {
1739     ls->print_cr("om_flush: jt=" INTPTR_FORMAT ", free_count=%d"
1740                  ", in_use_count=%d" ", om_free_provision=%d",
1741                  p2i(self), free_count, in_use_count, self->om_free_provision);
1742   }
1743 }
1744 
1745 static void post_monitor_inflate_event(EventJavaMonitorInflate* event,
1746                                        const oop obj,
1747                                        ObjectSynchronizer::InflateCause cause) {
1748   assert(event != NULL, "invariant");
1749   assert(event->should_commit(), "invariant");
1750   event->set_monitorClass(obj->klass());
1751   event->set_address((uintptr_t)(void*)obj);
1752   event->set_cause((u1)cause);
1753   event->commit();
1754 }
1755 
1756 // Fast path code shared by multiple functions
1757 void ObjectSynchronizer::inflate_helper(ObjectMonitorHandle* omh_p, oop obj) {
1758   while (true) {
1759     markWord mark = obj->mark();
1760     if (mark.has_monitor()) {
1761       if (!omh_p->save_om_ptr(obj, mark)) {
1762         // Lost a race with async deflation so try again.
1763         assert(AsyncDeflateIdleMonitors, "sanity check");
1764         continue;
1765       }
1766       ObjectMonitor* monitor = omh_p->om_ptr();
1767       assert(ObjectSynchronizer::verify_objmon_isinpool(monitor), "monitor is invalid");
1768       markWord dmw = monitor->header();
1769       assert(dmw.is_neutral(), "sanity check: header=" INTPTR_FORMAT, dmw.value());
1770       return;
1771     }
1772     inflate(omh_p, Thread::current(), obj, inflate_cause_vm_internal);
1773     return;
1774   }

1775 }
1776 
1777 void ObjectSynchronizer::inflate(ObjectMonitorHandle* omh_p, Thread* self,
1778                                  oop object, const InflateCause cause) {

1779   // Inflate mutates the heap ...
1780   // Relaxing assertion for bug 6320749.
1781   assert(Universe::verify_in_progress() ||
1782          !SafepointSynchronize::is_at_safepoint(), "invariant");
1783 
1784   EventJavaMonitorInflate event;
1785 
1786   for (;;) {
1787     const markWord mark = object->mark();
1788     assert(!mark.has_bias_pattern(), "invariant");
1789 
1790     // The mark can be in one of the following states:
1791     // *  Inflated     - just return
1792     // *  Stack-locked - coerce it to inflated
1793     // *  INFLATING    - busy wait for conversion to complete
1794     // *  Neutral      - aggressively inflate the object.
1795     // *  BIASED       - Illegal.  We should never see this
1796 
1797     // CASE: inflated
1798     if (mark.has_monitor()) {
1799       if (!omh_p->save_om_ptr(object, mark)) {
1800         // Lost a race with async deflation so try again.
1801         assert(AsyncDeflateIdleMonitors, "sanity check");
1802         continue;
1803       }
1804       ObjectMonitor* inf = omh_p->om_ptr();
1805       markWord dmw = inf->header();
1806       assert(dmw.is_neutral(), "invariant: header=" INTPTR_FORMAT, dmw.value());
1807       assert(inf->object() == object, "invariant");
1808       assert(ObjectSynchronizer::verify_objmon_isinpool(inf), "monitor is invalid");
1809       return;
1810     }
1811 
1812     // CASE: inflation in progress - inflating over a stack-lock.
1813     // Some other thread is converting from stack-locked to inflated.
1814     // Only that thread can complete inflation -- other threads must wait.
1815     // The INFLATING value is transient.
1816     // Currently, we spin/yield/park and poll the markword, waiting for inflation to finish.
1817     // We could always eliminate polling by parking the thread on some auxiliary list.
1818     if (mark == markWord::INFLATING()) {
1819       read_stable_mark(object);
1820       continue;
1821     }
1822 
1823     // CASE: stack-locked
1824     // Could be stack-locked either by this thread or by some other thread.
1825     //
1826     // Note that we allocate the objectmonitor speculatively, _before_ attempting
1827     // to install INFLATING into the mark word.  We originally installed INFLATING,
1828     // allocated the objectmonitor, and then finally STed the address of the
1829     // objectmonitor into the mark.  This was correct, but artificially lengthened
1830     // the interval in which INFLATED appeared in the mark, thus increasing
1831     // the odds of inflation contention.
1832     //
1833     // We now use per-thread private objectmonitor free lists.
1834     // These list are reprovisioned from the global free list outside the
1835     // critical INFLATING...ST interval.  A thread can transfer
1836     // multiple objectmonitors en-mass from the global free list to its local free list.
1837     // This reduces coherency traffic and lock contention on the global free list.
1838     // Using such local free lists, it doesn't matter if the om_alloc() call appears
1839     // before or after the CAS(INFLATING) operation.
1840     // See the comments in om_alloc().
1841 
1842     LogStreamHandle(Trace, monitorinflation) lsh;
1843 
1844     if (mark.has_locker()) {
1845       ObjectMonitor* m = om_alloc(self, cause);
1846       // Optimistically prepare the objectmonitor - anticipate successful CAS
1847       // We do this before the CAS in order to minimize the length of time
1848       // in which INFLATING appears in the mark.
1849       m->Recycle();
1850       m->_Responsible  = NULL;
1851       m->_SpinDuration = ObjectMonitor::Knob_SpinLimit;   // Consider: maintain by type/class
1852 
1853       markWord cmp = object->cas_set_mark(markWord::INFLATING(), mark);
1854       if (cmp != mark) {
1855         // om_release() will reset the allocation state from New to Free.
1856         om_release(self, m, true);
1857         continue;       // Interference -- just retry
1858       }
1859 
1860       // We've successfully installed INFLATING (0) into the mark-word.
1861       // This is the only case where 0 will appear in a mark-word.
1862       // Only the singular thread that successfully swings the mark-word
1863       // to 0 can perform (or more precisely, complete) inflation.
1864       //
1865       // Why do we CAS a 0 into the mark-word instead of just CASing the
1866       // mark-word from the stack-locked value directly to the new inflated state?
1867       // Consider what happens when a thread unlocks a stack-locked object.
1868       // It attempts to use CAS to swing the displaced header value from the
1869       // on-stack BasicLock back into the object header.  Recall also that the
1870       // header value (hash code, etc) can reside in (a) the object header, or
1871       // (b) a displaced header associated with the stack-lock, or (c) a displaced
1872       // header in an ObjectMonitor.  The inflate() routine must copy the header
1873       // value from the BasicLock on the owner's stack to the ObjectMonitor, all
1874       // the while preserving the hashCode stability invariants.  If the owner
1875       // decides to release the lock while the value is 0, the unlock will fail
1876       // and control will eventually pass from slow_exit() to inflate.  The owner
1877       // will then spin, waiting for the 0 value to disappear.   Put another way,
1878       // the 0 causes the owner to stall if the owner happens to try to
1879       // drop the lock (restoring the header from the BasicLock to the object)
1880       // while inflation is in-progress.  This protocol avoids races that might
1881       // would otherwise permit hashCode values to change or "flicker" for an object.
1882       // Critically, while object->mark is 0 mark.displaced_mark_helper() is stable.
1883       // 0 serves as a "BUSY" inflate-in-progress indicator.
1884 
1885 
1886       // fetch the displaced mark from the owner's stack.
1887       // The owner can't die or unwind past the lock while our INFLATING
1888       // object is in the mark.  Furthermore the owner can't complete
1889       // an unlock on the object, either.
1890       markWord dmw = mark.displaced_mark_helper();
1891       // Catch if the object's header is not neutral (not locked and
1892       // not marked is what we care about here).
1893       ADIM_guarantee(dmw.is_neutral(), "invariant: header=" INTPTR_FORMAT, dmw.value());
1894 
1895       // Setup monitor fields to proper values -- prepare the monitor
1896       m->set_header(dmw);
1897 
1898       // Optimization: if the mark.locker stack address is associated
1899       // with this thread we could simply set m->_owner = self.
1900       // Note that a thread can inflate an object
1901       // that it has stack-locked -- as might happen in wait() -- directly
1902       // with CAS.  That is, we can avoid the xchg-NULL .... ST idiom.
1903       if (AsyncDeflateIdleMonitors) {
1904         m->set_owner_from(mark.locker(), NULL, DEFLATER_MARKER);
1905       } else {
1906         m->set_owner_from(mark.locker(), NULL);
1907       }
1908       m->set_object(object);
1909       // TODO-FIXME: assert BasicLock->dhw != 0.
1910 
1911       omh_p->set_om_ptr(m);
1912 
1913       // Must preserve store ordering. The monitor state must
1914       // be stable at the time of publishing the monitor address.
1915       guarantee(object->mark() == markWord::INFLATING(), "invariant");
1916       object->release_set_mark(markWord::encode(m));
1917 
1918       // Once ObjectMonitor is configured and the object is associated
1919       // with the ObjectMonitor, it is safe to allow async deflation:
1920       assert(m->is_new(), "freshly allocated monitor must be new");
1921       m->set_allocation_state(ObjectMonitor::Old);
1922 
1923       // Hopefully the performance counters are allocated on distinct cache lines
1924       // to avoid false sharing on MP systems ...
1925       OM_PERFDATA_OP(Inflations, inc());
1926       if (log_is_enabled(Trace, monitorinflation)) {
1927         ResourceMark rm(self);
1928         lsh.print_cr("inflate(has_locker): object=" INTPTR_FORMAT ", mark="
1929                      INTPTR_FORMAT ", type='%s'", p2i(object),
1930                      object->mark().value(), object->klass()->external_name());
1931       }
1932       if (event.should_commit()) {
1933         post_monitor_inflate_event(&event, object, cause);
1934       }
1935       ADIM_guarantee(!m->is_free(), "inflated monitor to be returned cannot be free");
1936       return;
1937     }
1938 
1939     // CASE: neutral
1940     // TODO-FIXME: for entry we currently inflate and then try to CAS _owner.
1941     // If we know we're inflating for entry it's better to inflate by swinging a
1942     // pre-locked ObjectMonitor pointer into the object header.   A successful
1943     // CAS inflates the object *and* confers ownership to the inflating thread.
1944     // In the current implementation we use a 2-step mechanism where we CAS()
1945     // to inflate and then CAS() again to try to swing _owner from NULL to self.
1946     // An inflateTry() method that we could call from enter() would be useful.
1947 
1948     // Catch if the object's header is not neutral (not locked and
1949     // not marked is what we care about here).
1950     ADIM_guarantee(mark.is_neutral(), "invariant: header=" INTPTR_FORMAT,mark.value());
1951     ObjectMonitor* m = om_alloc(self, cause);
1952     // prepare m for installation - set monitor to initial state
1953     m->Recycle();
1954     m->set_header(mark);
1955     // If we leave _owner == DEFLATER_MARKER here, then the simple C2
1956     // ObjectMonitor enter optimization can no longer race with async
1957     // deflation and reuse.
1958     m->set_object(object);
1959     m->_Responsible  = NULL;
1960     m->_SpinDuration = ObjectMonitor::Knob_SpinLimit;       // consider: keep metastats by type/class
1961 
1962     omh_p->set_om_ptr(m);
1963 
1964     if (object->cas_set_mark(markWord::encode(m), mark) != mark) {
1965       m->set_header(markWord::zero());
1966       m->set_object(NULL);
1967       m->Recycle();
1968       omh_p->set_om_ptr(NULL);
1969       // om_release() will reset the allocation state from New to Free.
1970       om_release(self, m, true);
1971       m = NULL;
1972       continue;
1973       // interference - the markword changed - just retry.
1974       // The state-transitions are one-way, so there's no chance of
1975       // live-lock -- "Inflated" is an absorbing state.
1976     }
1977 
1978     // Once the ObjectMonitor is configured and object is associated
1979     // with the ObjectMonitor, it is safe to allow async deflation:
1980     assert(m->is_new(), "freshly allocated monitor must be new");
1981     m->set_allocation_state(ObjectMonitor::Old);
1982 
1983     // Hopefully the performance counters are allocated on distinct
1984     // cache lines to avoid false sharing on MP systems ...
1985     OM_PERFDATA_OP(Inflations, inc());
1986     if (log_is_enabled(Trace, monitorinflation)) {
1987       ResourceMark rm(self);
1988       lsh.print_cr("inflate(neutral): object=" INTPTR_FORMAT ", mark="
1989                    INTPTR_FORMAT ", type='%s'", p2i(object),
1990                    object->mark().value(), object->klass()->external_name());
1991     }
1992     if (event.should_commit()) {
1993       post_monitor_inflate_event(&event, object, cause);
1994     }
1995     ADIM_guarantee(!m->is_free(), "inflated monitor to be returned cannot be free");
1996     return;
1997   }
1998 }
1999 
2000 
2001 // We maintain a list of in-use monitors for each thread.
2002 //
2003 // For safepoint based deflation:
2004 // deflate_thread_local_monitors() scans a single thread's in-use list, while
2005 // deflate_idle_monitors() scans only a global list of in-use monitors which
2006 // is populated only as a thread dies (see om_flush()).
2007 //
2008 // These operations are called at all safepoints, immediately after mutators
2009 // are stopped, but before any objects have moved. Collectively they traverse
2010 // the population of in-use monitors, deflating where possible. The scavenged
2011 // monitors are returned to the global monitor free list.
2012 //
2013 // Beware that we scavenge at *every* stop-the-world point. Having a large
2014 // number of monitors in-use could negatively impact performance. We also want
2015 // to minimize the total # of monitors in circulation, as they incur a small
2016 // footprint penalty.
2017 //
2018 // Perversely, the heap size -- and thus the STW safepoint rate --
2019 // typically drives the scavenge rate.  Large heaps can mean infrequent GC,
2020 // which in turn can mean large(r) numbers of ObjectMonitors in circulation.
2021 // This is an unfortunate aspect of this design.
2022 //
2023 // For async deflation:
2024 // If a special deflation request is made, then the safepoint based
2025 // deflation mechanism is used. Otherwise, an async deflation request
2026 // is registered with the ServiceThread and it is notified.
2027 
2028 void ObjectSynchronizer::do_safepoint_work(DeflateMonitorCounters* counters) {
2029   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
2030 
2031   // The per-thread in-use lists are handled in
2032   // ParallelSPCleanupThreadClosure::do_thread().
2033 
2034   if (!AsyncDeflateIdleMonitors || is_special_deflation_requested()) {
2035     // Use the older mechanism for the global in-use list or if a
2036     // special deflation has been requested before the safepoint.
2037     ObjectSynchronizer::deflate_idle_monitors(counters);
2038     return;
2039   }
2040 
2041   log_debug(monitorinflation)("requesting async deflation of idle monitors.");
2042   // Request deflation of idle monitors by the ServiceThread:
2043   set_is_async_deflation_requested(true);
2044   MonitorLocker ml(Service_lock, Mutex::_no_safepoint_check_flag);
2045   ml.notify_all();
2046 }
2047 
2048 // Deflate a single monitor if not in-use
2049 // Return true if deflated, false if in-use
2050 bool ObjectSynchronizer::deflate_monitor(ObjectMonitor* mid, oop obj,
2051                                          ObjectMonitor** free_head_p,
2052                                          ObjectMonitor** free_tail_p) {
2053   bool deflated;
2054   // Normal case ... The monitor is associated with obj.
2055   const markWord mark = obj->mark();
2056   guarantee(mark == markWord::encode(mid), "should match: mark="
2057             INTPTR_FORMAT ", encoded mid=" INTPTR_FORMAT, mark.value(),
2058             markWord::encode(mid).value());
2059   // Make sure that mark.monitor() and markWord::encode() agree:
2060   guarantee(mark.monitor() == mid, "should match: monitor()=" INTPTR_FORMAT
2061             ", mid=" INTPTR_FORMAT, p2i(mark.monitor()), p2i(mid));
2062   const markWord dmw = mid->header();
2063   guarantee(dmw.is_neutral(), "invariant: header=" INTPTR_FORMAT, dmw.value());
2064 
2065   if (mid->is_busy() || mid->ref_count() != 0) {
2066     // Easy checks are first - the ObjectMonitor is busy or ObjectMonitor*
2067     // is in use so no deflation.
2068     deflated = false;
2069   } else {
2070     // Deflate the monitor if it is no longer being used
2071     // It's idle - scavenge and return to the global free list
2072     // plain old deflation ...
2073     if (log_is_enabled(Trace, monitorinflation)) {
2074       ResourceMark rm;
2075       log_trace(monitorinflation)("deflate_monitor: "
2076                                   "object=" INTPTR_FORMAT ", mark="
2077                                   INTPTR_FORMAT ", type='%s'", p2i(obj),
2078                                   mark.value(), obj->klass()->external_name());
2079     }
2080 
2081     // Restore the header back to obj
2082     obj->release_set_mark(dmw);
2083     if (AsyncDeflateIdleMonitors) {
2084       // clear() expects the owner field to be NULL and we won't race
2085       // with the simple C2 ObjectMonitor enter optimization since
2086       // we're at a safepoint. DEFLATER_MARKER is the only non-NULL
2087       // value we should see here.
2088       mid->try_set_owner_from(NULL, DEFLATER_MARKER);
2089     }
2090     mid->clear();
2091 
2092     assert(mid->object() == NULL, "invariant: object=" INTPTR_FORMAT,
2093            p2i(mid->object()));
2094     assert(mid->is_free(), "invariant");
2095 
2096     // Move the deflated ObjectMonitor to the working free list
2097     // defined by free_head_p and free_tail_p. No races on this list
2098     // so no need for load_acquire() or store_release().
2099     if (*free_head_p == NULL) *free_head_p = mid;
2100     if (*free_tail_p != NULL) {
2101       // We append to the list so the caller can use mid->_next_om
2102       // to fix the linkages in its context.
2103       ObjectMonitor* prevtail = *free_tail_p;
2104       // Should have been cleaned up by the caller:
2105       // Note: Should not have to mark prevtail here since we're at a
2106       // safepoint and ObjectMonitors on the local free list should
2107       // not be accessed in parallel.
2108       assert(prevtail->_next_om == NULL, "must be NULL: _next_om="
2109              INTPTR_FORMAT, p2i(prevtail->_next_om));
2110       set_next(prevtail, mid);
2111     }
2112     *free_tail_p = mid;
2113     // At this point, mid->_next_om still refers to its current
2114     // value and another ObjectMonitor's _next_om field still
2115     // refers to this ObjectMonitor. Those linkages have to be
2116     // cleaned up by the caller who has the complete context.
2117     deflated = true;
2118   }
2119   return deflated;
2120 }
2121 
2122 // Deflate the specified ObjectMonitor if not in-use using a JavaThread.
2123 // Returns true if it was deflated and false otherwise.
2124 //
2125 // The async deflation protocol sets owner to DEFLATER_MARKER and
2126 // makes ref_count negative as signals to contending threads that
2127 // an async deflation is in progress. There are a number of checks
2128 // as part of the protocol to make sure that the calling thread has
2129 // not lost the race to a contending thread or to a thread that just
2130 // wants to use the ObjectMonitor*.
2131 //
2132 // The ObjectMonitor has been successfully async deflated when:
2133 // (owner == DEFLATER_MARKER && ref_count < 0)
2134 // Contending threads or ObjectMonitor* using threads that see those
2135 // values know to retry their operation.
2136 //
2137 bool ObjectSynchronizer::deflate_monitor_using_JT(ObjectMonitor* mid,
2138                                                   ObjectMonitor** free_head_p,
2139                                                   ObjectMonitor** free_tail_p) {
2140   assert(AsyncDeflateIdleMonitors, "sanity check");
2141   assert(Thread::current()->is_Java_thread(), "precondition");
2142   // A newly allocated ObjectMonitor should not be seen here so we
2143   // avoid an endless inflate/deflate cycle.
2144   assert(mid->is_old(), "must be old: allocation_state=%d",
2145          (int) mid->allocation_state());
2146 
2147   if (mid->is_busy() || mid->ref_count() != 0) {
2148     // Easy checks are first - the ObjectMonitor is busy or ObjectMonitor*
2149     // is in use so no deflation.
2150     return false;
2151   }
2152 
2153   if (mid->try_set_owner_from(DEFLATER_MARKER, NULL) == NULL) {
2154     // ObjectMonitor is not owned by another thread. Our setting
2155     // owner to DEFLATER_MARKER forces any contending thread through
2156     // the slow path. This is just the first part of the async
2157     // deflation dance.
2158 
2159     if (mid->_contentions != 0 || mid->_waiters != 0) {
2160       // Another thread has raced to enter the ObjectMonitor after
2161       // mid->is_busy() above or has already entered and waited on
2162       // it which makes it busy so no deflation. Restore owner to
2163       // NULL if it is still DEFLATER_MARKER.
2164       mid->try_set_owner_from(NULL, DEFLATER_MARKER);
2165       return false;
2166     }
2167 
2168     if (Atomic::cmpxchg(-max_jint, &mid->_ref_count, (jint)0) == 0) {
2169       // Make ref_count negative to force any contending threads or
2170       // ObjectMonitor* using threads to retry. This is the second
2171       // part of the async deflation dance.
2172 
2173       if (mid->owner_is_DEFLATER_MARKER()) {
2174         // If owner is still DEFLATER_MARKER, then we have successfully
2175         // signaled any contending threads to retry. If it is not, then we
2176         // have lost the race to an entering thread and the ObjectMonitor
2177         // is now busy. This is the third and final part of the async
2178         // deflation dance.
2179         // Note: This owner check solves the ABA problem with ref_count
2180         // where another thread acquired the ObjectMonitor, finished
2181         // using it and restored the ref_count to zero.
2182 
2183         // Sanity checks for the races:
2184         guarantee(mid->_contentions == 0, "must be 0: contentions=%d",
2185                   mid->_contentions);
2186         guarantee(mid->_waiters == 0, "must be 0: waiters=%d", mid->_waiters);
2187         guarantee(mid->_cxq == NULL, "must be no contending threads: cxq="
2188                   INTPTR_FORMAT, p2i(mid->_cxq));
2189         guarantee(mid->_EntryList == NULL,
2190                   "must be no entering threads: EntryList=" INTPTR_FORMAT,
2191                   p2i(mid->_EntryList));
2192 
2193         const oop obj = (oop) mid->object();
2194         if (log_is_enabled(Trace, monitorinflation)) {
2195           ResourceMark rm;
2196           log_trace(monitorinflation)("deflate_monitor_using_JT: "
2197                                       "object=" INTPTR_FORMAT ", mark="
2198                                       INTPTR_FORMAT ", type='%s'",
2199                                       p2i(obj), obj->mark().value(),
2200                                       obj->klass()->external_name());
2201         }
2202 
2203         // Install the old mark word if nobody else has already done it.
2204         mid->install_displaced_markword_in_object(obj);
2205         mid->clear_using_JT();
2206 
2207         assert(mid->object() == NULL, "must be NULL: object=" INTPTR_FORMAT,
2208                p2i(mid->object()));
2209         assert(mid->is_free(), "must be free: allocation_state=%d",
2210                (int) mid->allocation_state());
2211 
2212         // Move the deflated ObjectMonitor to the working free list
2213         // defined by free_head_p and free_tail_p. No races on this list
2214         // so no need for load_acquire() or store_release().
2215         if (*free_head_p == NULL) {
2216           // First one on the list.
2217           *free_head_p = mid;
2218         }
2219         if (*free_tail_p != NULL) {
2220           // We append to the list so the caller can use mid->_next_om
2221           // to fix the linkages in its context.
2222           ObjectMonitor* prevtail = *free_tail_p;
2223           // Should have been cleaned up by the caller:
2224           ObjectMonitor* next = mark_next_loop(prevtail);
2225           assert(unmarked_next(prevtail) == NULL, "must be NULL: _next_om="
2226                  INTPTR_FORMAT, p2i(unmarked_next(prevtail)));
2227           set_next(prevtail, mid);  // prevtail now points to mid (and is unmarked)
2228         }
2229         *free_tail_p = mid;
2230 
2231         // At this point, mid->_next_om still refers to its current
2232         // value and another ObjectMonitor's _next_om field still
2233         // refers to this ObjectMonitor. Those linkages have to be
2234         // cleaned up by the caller who has the complete context.
2235 
2236         // We leave owner == DEFLATER_MARKER and ref_count < 0
2237         // to force any racing threads to retry.
2238         return true;  // Success, ObjectMonitor has been deflated.
2239       }
2240 
2241       // The owner was changed from DEFLATER_MARKER so we lost the
2242       // race since the ObjectMonitor is now busy.
2243 
2244       // Add back max_jint to restore the ref_count field to its
2245       // proper value (which may not be what we saw above):
2246       Atomic::add(max_jint, &mid->_ref_count);
2247 
2248       assert(mid->ref_count() >= 0, "must not be negative: ref_count=%d",
2249              mid->ref_count());
2250       return false;
2251     }
2252 
2253     // The ref_count was no longer 0 so we lost the race since the
2254     // ObjectMonitor is now busy or the ObjectMonitor* is now is use.
2255     // Restore owner to NULL if it is still DEFLATER_MARKER:
2256     mid->try_set_owner_from(NULL, DEFLATER_MARKER);
2257   }
2258 
2259   // The owner field is no longer NULL so we lost the race since the
2260   // ObjectMonitor is now busy.
2261   return false;
2262 }
2263 
2264 // Walk a given monitor list, and deflate idle monitors.
2265 // The given list could be a per-thread list or a global list.
2266 //
2267 // In the case of parallel processing of thread local monitor lists,
2268 // work is done by Threads::parallel_threads_do() which ensures that
2269 // each Java thread is processed by exactly one worker thread, and
2270 // thus avoid conflicts that would arise when worker threads would
2271 // process the same monitor lists concurrently.
2272 //
2273 // See also ParallelSPCleanupTask and
2274 // SafepointSynchronize::do_cleanup_tasks() in safepoint.cpp and
2275 // Threads::parallel_java_threads_do() in thread.cpp.
2276 int ObjectSynchronizer::deflate_monitor_list(ObjectMonitor* volatile * list_p,
2277                                              int volatile * count_p,
2278                                              ObjectMonitor** free_head_p,
2279                                              ObjectMonitor** free_tail_p) {


2280   ObjectMonitor* cur_mid_in_use = NULL;
2281   ObjectMonitor* mid = NULL;
2282   ObjectMonitor* next = NULL;
2283   int deflated_count = 0;
2284 
2285   // We use the simpler mark-mid-as-we-go protocol since there are no
2286   // parallel list deletions since we are at a safepoint.
2287   if (!mark_list_head(list_p, &mid, &next)) {
2288     return 0;  // The list is empty so nothing to deflate.
2289   }
2290 
2291   while (true) {
2292     oop obj = (oop) mid->object();
2293     if (obj != NULL && deflate_monitor(mid, obj, free_head_p, free_tail_p)) {
2294       // Deflation succeeded and already updated free_head_p and
2295       // free_tail_p as needed. Finish the move to the local free list
2296       // by unlinking mid from the global or per-thread in-use list.
2297       if (cur_mid_in_use == NULL) {
2298         // mid is the list head and it is marked. Switch the list head
2299         // to next which unmarks the list head, but leaves mid marked:
2300         OrderAccess::release_store(list_p, next);
2301       } else {
2302         // mid is marked. Switch cur_mid_in_use's next field to next
2303         // which is safe because we have no parallel list deletions,
2304         // but we leave mid marked:
2305         OrderAccess::release_store(&cur_mid_in_use->_next_om, next);
2306       }
2307       // At this point mid is disconnected from the in-use list so
2308       // its marked next field no longer has any effects.

2309       deflated_count++;
2310       Atomic::dec(count_p);
2311       // mid is current tail in the free_head_p list so NULL terminate it
2312       // (which also unmarks it):
2313       set_next(mid, NULL);
2314 
2315       // All the list management is done so move on to the next one:
2316       mid = next;
2317     } else {
2318       set_next(mid, next);  // unmark next field
2319 
2320       // All the list management is done so move on to the next one:
2321       cur_mid_in_use = mid;
2322       mid = next;
2323     }
2324     if (mid == NULL) {
2325       break;  // Reached end of the list so nothing more to deflate.
2326     }
2327     // Mark mid's next field so we can possibly deflate it:
2328     next = mark_next_loop(mid);
2329   }
2330   return deflated_count;
2331 }
2332 
2333 // Walk a given ObjectMonitor list and deflate idle ObjectMonitors using
2334 // a JavaThread. Returns the number of deflated ObjectMonitors. The given
2335 // list could be a per-thread in-use list or the global in-use list.
2336 // If a safepoint has started, then we save state via saved_mid_in_use_p
2337 // and return to the caller to honor the safepoint.
2338 //
2339 int ObjectSynchronizer::deflate_monitor_list_using_JT(ObjectMonitor* volatile * list_p,
2340                                                       int volatile * count_p,
2341                                                       ObjectMonitor** free_head_p,
2342                                                       ObjectMonitor** free_tail_p,
2343                                                       ObjectMonitor** saved_mid_in_use_p) {
2344   assert(AsyncDeflateIdleMonitors, "sanity check");
2345   assert(Thread::current()->is_Java_thread(), "precondition");
2346 
2347   ObjectMonitor* cur_mid_in_use = NULL;
2348   ObjectMonitor* mid = NULL;
2349   ObjectMonitor* next = NULL;
2350   ObjectMonitor* next_next = NULL;
2351   int deflated_count = 0;
2352 
2353   // We use the more complicated mark-cur_mid_in_use-and-mid-as-we-go
2354   // protocol because om_release() can do list deletions in parallel.
2355   // We also mark-next-next-as-we-go to prevent an om_flush() that is
2356   // behind this thread from passing us.
2357   if (*saved_mid_in_use_p == NULL) {
2358     // No saved state so start at the beginning.
2359     // Mark the list head's next field so we can possibly deflate it:
2360     if (!mark_list_head(list_p, &mid, &next)) {
2361       return 0;  // The list is empty so nothing to deflate.
2362     }
2363   } else {
2364     // We're restarting after a safepoint so restore the necessary state
2365     // before we resume.
2366     cur_mid_in_use = *saved_mid_in_use_p;
2367     // Mark cur_mid_in_use's next field so we can possibly update its
2368     // next field to extract a deflated ObjectMonitor.
2369     mid = mark_next_loop(cur_mid_in_use);
2370     if (mid == NULL) {
2371       set_next(cur_mid_in_use, NULL);  // unmark next field
2372       *saved_mid_in_use_p = NULL;
2373       return 0;  // The remainder is empty so nothing more to deflate.
2374     }
2375     // Mark mid's next field so we can possibly deflate it:
2376     next = mark_next_loop(mid);
2377   }
2378 
2379   while (true) {
2380     // The current mid's next field is marked at this point. If we have
2381     // a cur_mid_in_use, then its next field is also marked at this point.
2382 
2383     if (next != NULL) {
2384       // We mark next's next field so that an om_flush()
2385       // thread that is behind us cannot pass us when we
2386       // unmark the current mid's next field.
2387       next_next = mark_next_loop(next);
2388     }
2389 
2390     // Only try to deflate if there is an associated Java object and if
2391     // mid is old (is not newly allocated and is not newly freed).
2392     if (mid->object() != NULL && mid->is_old() &&
2393         deflate_monitor_using_JT(mid, free_head_p, free_tail_p)) {
2394       // Deflation succeeded and already updated free_head_p and
2395       // free_tail_p as needed. Finish the move to the local free list
2396       // by unlinking mid from the global or per-thread in-use list.
2397       if (cur_mid_in_use == NULL) {
2398         // mid is the list head and it is marked. Switch the list head
2399         // to next which is also marked (if not NULL) and also leave
2400         // mid marked:
2401         OrderAccess::release_store(list_p, next);
2402       } else {
2403         ObjectMonitor* marked_next = mark_om_ptr(next);
2404         // mid and cur_mid_in_use are marked. Switch cur_mid_in_use's
2405         // next field to marked_next and also leave mid marked:
2406         OrderAccess::release_store(&cur_mid_in_use->_next_om, marked_next);
2407       }
2408       // At this point mid is disconnected from the in-use list so
2409       // its marked next field no longer has any effects.
2410       deflated_count++;
2411       Atomic::dec(count_p);
2412       // mid is current tail in the free_head_p list so NULL terminate it
2413       // (which also unmarks it):
2414       set_next(mid, NULL);
2415 
2416       // All the list management is done so move on to the next one:
2417       mid = next;  // mid keeps non-NULL next's marked next field
2418       next = next_next;
2419     } else {
2420       // mid is considered in-use if it does not have an associated
2421       // Java object or mid is not old or deflation did not succeed.
2422       // A mid->is_new() node can be seen here when it is freshly
2423       // returned by om_alloc() (and skips the deflation code path).
2424       // A mid->is_old() node can be seen here when deflation failed.
2425       // A mid->is_free() node can be seen here when a fresh node from
2426       // om_alloc() is released by om_release() due to losing the race
2427       // in inflate().
2428 
2429       // All the list management is done so move on to the next one:
2430       if (cur_mid_in_use != NULL) {
2431         set_next(cur_mid_in_use, mid);  // umark cur_mid_in_use
2432       }
2433       // The next cur_mid_in_use keeps mid's marked next field so
2434       // that it is stable for a possible next field change. It
2435       // cannot be modified by om_release() while it is marked.
2436       cur_mid_in_use = mid;
2437       mid = next;  // mid keeps non-NULL next's marked next field
2438       next = next_next;
2439 
2440       if (SafepointSynchronize::is_synchronizing() &&
2441           cur_mid_in_use != OrderAccess::load_acquire(list_p) &&
2442           cur_mid_in_use->is_old()) {
2443         // If a safepoint has started and cur_mid_in_use is not the list
2444         // head and is old, then it is safe to use as saved state. Return
2445         // to the caller before blocking.
2446         *saved_mid_in_use_p = cur_mid_in_use;
2447         set_next(cur_mid_in_use, mid);  // umark cur_mid_in_use
2448         if (mid != NULL) {
2449           set_next(mid, next);  // umark mid
2450         }
2451         return deflated_count;
2452       }
2453     }
2454     if (mid == NULL) {
2455       if (cur_mid_in_use != NULL) {
2456         set_next(cur_mid_in_use, mid);  // umark cur_mid_in_use
2457       }
2458       break;  // Reached end of the list so nothing more to deflate.
2459     }
2460 
2461     // The current mid's next field is marked at this point. If we have
2462     // a cur_mid_in_use, then its next field is also marked at this point.
2463   }
2464   // We finished the list without a safepoint starting so there's
2465   // no need to save state.
2466   *saved_mid_in_use_p = NULL;
2467   return deflated_count;
2468 }
2469 
2470 void ObjectSynchronizer::prepare_deflate_idle_monitors(DeflateMonitorCounters* counters) {
2471   OrderAccess::release_store(&counters->n_in_use, 0);              // currently associated with objects
2472   OrderAccess::release_store(&counters->n_in_circulation, 0);      // extant
2473   OrderAccess::release_store(&counters->n_scavenged, 0);           // reclaimed (global and per-thread)
2474   OrderAccess::release_store(&counters->per_thread_scavenged, 0);  // per-thread scavenge total
2475   counters->per_thread_times = 0.0;                                // per-thread scavenge times
2476 }
2477 
2478 void ObjectSynchronizer::deflate_idle_monitors(DeflateMonitorCounters* counters) {
2479   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
2480 
2481   if (AsyncDeflateIdleMonitors) {
2482     // Nothing to do when global idle ObjectMonitors are deflated using
2483     // a JavaThread unless a special deflation has been requested.
2484     if (!is_special_deflation_requested()) {
2485       return;
2486     }
2487   }
2488 
2489   bool deflated = false;
2490 
2491   ObjectMonitor* free_head_p = NULL;  // Local SLL of scavenged monitors
2492   ObjectMonitor* free_tail_p = NULL;
2493   elapsedTimer timer;
2494 
2495   if (log_is_enabled(Info, monitorinflation)) {
2496     timer.start();
2497   }
2498 





2499   // Note: the thread-local monitors lists get deflated in
2500   // a separate pass. See deflate_thread_local_monitors().
2501 
2502   // For moribund threads, scan g_om_in_use_list
2503   int deflated_count = 0;
2504   if (OrderAccess::load_acquire(&g_om_in_use_list) != NULL) {
2505     // Update n_in_circulation before g_om_in_use_count is updated by deflation.
2506     Atomic::add(OrderAccess::load_acquire(&g_om_in_use_count), &counters->n_in_circulation);
2507 
2508     deflated_count = deflate_monitor_list(&g_om_in_use_list, &g_om_in_use_count, &free_head_p, &free_tail_p);
2509     Atomic::add(OrderAccess::load_acquire(&g_om_in_use_count), &counters->n_in_use);
2510   }
2511 
2512   if (free_head_p != NULL) {
2513     // Move the deflated ObjectMonitors back to the global free list.
2514     // No races on the working free list so no need for load_acquire().
2515     guarantee(free_tail_p != NULL && deflated_count > 0, "invariant");
2516     assert(free_tail_p->_next_om == NULL, "must be NULL: _next_om="
2517            INTPTR_FORMAT, p2i(free_tail_p->_next_om));
2518     prepend_list_to_g_free_list(free_head_p, free_tail_p, deflated_count);
2519     Atomic::add(deflated_count, &counters->n_scavenged);
2520   }

2521   timer.stop();
2522 
2523   LogStreamHandle(Debug, monitorinflation) lsh_debug;
2524   LogStreamHandle(Info, monitorinflation) lsh_info;
2525   LogStream* ls = NULL;
2526   if (log_is_enabled(Debug, monitorinflation)) {
2527     ls = &lsh_debug;
2528   } else if (deflated_count != 0 && log_is_enabled(Info, monitorinflation)) {
2529     ls = &lsh_info;
2530   }
2531   if (ls != NULL) {
2532     ls->print_cr("deflating global idle monitors, %3.7f secs, %d monitors", timer.seconds(), deflated_count);
2533   }
2534 }
2535 
2536 class HandshakeForDeflation : public ThreadClosure {
2537  public:
2538   void do_thread(Thread* thread) {
2539     log_trace(monitorinflation)("HandshakeForDeflation::do_thread: thread="
2540                                 INTPTR_FORMAT, p2i(thread));
2541   }
2542 };
2543 
2544 void ObjectSynchronizer::deflate_idle_monitors_using_JT() {
2545   assert(AsyncDeflateIdleMonitors, "sanity check");
2546 
2547   // Deflate any global idle monitors.
2548   deflate_global_idle_monitors_using_JT();
2549 
2550   int count = 0;
2551   for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = jtiwh.next(); ) {
2552     if (jt->om_in_use_count > 0 && !jt->is_exiting()) {
2553       // This JavaThread is using ObjectMonitors so deflate any that
2554       // are idle unless this JavaThread is exiting; do not race with
2555       // ObjectSynchronizer::om_flush().
2556       deflate_per_thread_idle_monitors_using_JT(jt);
2557       count++;
2558     }
2559   }
2560   if (count > 0) {
2561     log_debug(monitorinflation)("did async deflation of idle monitors for %d thread(s).", count);
2562   }
2563   // The ServiceThread's async deflation request has been processed.
2564   set_is_async_deflation_requested(false);
2565 
2566   if (HandshakeAfterDeflateIdleMonitors && g_om_wait_count > 0) {
2567     // There are deflated ObjectMonitors waiting for a handshake
2568     // (or a safepoint) for safety.
2569 
2570     // g_wait_list and g_om_wait_count are only updated by the calling
2571     // thread so no need for load_acquire() or release_store().
2572     ObjectMonitor* list = g_wait_list;
2573     ADIM_guarantee(list != NULL, "g_wait_list must not be NULL");
2574     int count = g_om_wait_count;
2575     g_wait_list = NULL;
2576     g_om_wait_count = 0;
2577 
2578     // Find the tail for prepend_list_to_common().
2579     int l_count = 0;
2580     ObjectMonitor* tail = NULL;
2581     for (ObjectMonitor* n = list; n != NULL; n = unmarked_next(n)) {
2582       tail = n;
2583       l_count++;
2584     }
2585     ADIM_guarantee(count == l_count, "count=%d != l_count=%d", count, l_count);
2586 
2587     // Will execute a safepoint if !ThreadLocalHandshakes:
2588     HandshakeForDeflation hfd_tc;
2589     Handshake::execute(&hfd_tc);
2590 
2591     prepend_list_to_common(list, tail, count, &g_free_list, &g_om_free_count);
2592 
2593     log_info(monitorinflation)("moved %d idle monitors from global waiting list to global free list", count);
2594   }
2595 }
2596 
2597 // Deflate global idle ObjectMonitors using a JavaThread.
2598 //
2599 void ObjectSynchronizer::deflate_global_idle_monitors_using_JT() {
2600   assert(AsyncDeflateIdleMonitors, "sanity check");
2601   assert(Thread::current()->is_Java_thread(), "precondition");
2602   JavaThread* self = JavaThread::current();
2603 
2604   deflate_common_idle_monitors_using_JT(true /* is_global */, self);
2605 }
2606 
2607 // Deflate the specified JavaThread's idle ObjectMonitors using a JavaThread.
2608 //
2609 void ObjectSynchronizer::deflate_per_thread_idle_monitors_using_JT(JavaThread* target) {
2610   assert(AsyncDeflateIdleMonitors, "sanity check");
2611   assert(Thread::current()->is_Java_thread(), "precondition");
2612 
2613   deflate_common_idle_monitors_using_JT(false /* !is_global */, target);
2614 }
2615 
2616 // Deflate global or per-thread idle ObjectMonitors using a JavaThread.
2617 //
2618 void ObjectSynchronizer::deflate_common_idle_monitors_using_JT(bool is_global, JavaThread* target) {
2619   JavaThread* self = JavaThread::current();
2620 
2621   int deflated_count = 0;
2622   ObjectMonitor* free_head_p = NULL;  // Local SLL of scavenged ObjectMonitors
2623   ObjectMonitor* free_tail_p = NULL;
2624   ObjectMonitor* saved_mid_in_use_p = NULL;
2625   elapsedTimer timer;
2626 
2627   if (log_is_enabled(Info, monitorinflation)) {
2628     timer.start();
2629   }
2630 
2631   if (is_global) {
2632     OM_PERFDATA_OP(MonExtant, set_value(OrderAccess::load_acquire(&g_om_in_use_count)));
2633   } else {
2634     OM_PERFDATA_OP(MonExtant, inc(OrderAccess::load_acquire(&target->om_in_use_count)));
2635   }
2636 
2637   do {
2638     int local_deflated_count;
2639     if (is_global) {
2640       local_deflated_count = deflate_monitor_list_using_JT(&g_om_in_use_list, &g_om_in_use_count, &free_head_p, &free_tail_p, &saved_mid_in_use_p);
2641     } else {
2642       local_deflated_count = deflate_monitor_list_using_JT(&target->om_in_use_list, &target->om_in_use_count, &free_head_p, &free_tail_p, &saved_mid_in_use_p);
2643     }
2644     deflated_count += local_deflated_count;
2645 
2646     if (free_head_p != NULL) {
2647       // Move the deflated ObjectMonitors to the global free list.
2648       // No races on the working list so no need for load_acquire().
2649       guarantee(free_tail_p != NULL && local_deflated_count > 0, "free_tail_p=" INTPTR_FORMAT ", local_deflated_count=%d", p2i(free_tail_p), local_deflated_count);
2650       // Note: The target thread can be doing an om_alloc() that
2651       // is trying to prepend an ObjectMonitor on its in-use list
2652       // at the same time that we have deflated the current in-use
2653       // list head and put it on the local free list. prepend_to_common()
2654       // will detect the race and retry which avoids list corruption,
2655       // but the next field in free_tail_p can flicker to marked
2656       // and then unmarked while prepend_to_common() is sorting it
2657       // all out.
2658       assert(unmarked_next(free_tail_p) == NULL, "must be NULL: _next_om="
2659              INTPTR_FORMAT, p2i(unmarked_next(free_tail_p)));
2660 
2661       if (HandshakeAfterDeflateIdleMonitors) {
2662         prepend_list_to_g_wait_list(free_head_p, free_tail_p, local_deflated_count);
2663       } else {
2664         prepend_list_to_g_free_list(free_head_p, free_tail_p, local_deflated_count);
2665       }
2666 
2667       OM_PERFDATA_OP(Deflations, inc(local_deflated_count));
2668     }
2669 
2670     if (saved_mid_in_use_p != NULL) {
2671       // deflate_monitor_list_using_JT() detected a safepoint starting.
2672       timer.stop();
2673       {
2674         if (is_global) {
2675           log_debug(monitorinflation)("pausing deflation of global idle monitors for a safepoint.");
2676         } else {
2677           log_debug(monitorinflation)("jt=" INTPTR_FORMAT ": pausing deflation of per-thread idle monitors for a safepoint.", p2i(target));
2678         }
2679         assert(SafepointSynchronize::is_synchronizing(), "sanity check");
2680         ThreadBlockInVM blocker(self);
2681       }
2682       // Prepare for another loop after the safepoint.
2683       free_head_p = NULL;
2684       free_tail_p = NULL;
2685       if (log_is_enabled(Info, monitorinflation)) {
2686         timer.start();
2687       }
2688     }
2689   } while (saved_mid_in_use_p != NULL);
2690   timer.stop();
2691 
2692   LogStreamHandle(Debug, monitorinflation) lsh_debug;
2693   LogStreamHandle(Info, monitorinflation) lsh_info;
2694   LogStream* ls = NULL;
2695   if (log_is_enabled(Debug, monitorinflation)) {
2696     ls = &lsh_debug;
2697   } else if (deflated_count != 0 && log_is_enabled(Info, monitorinflation)) {
2698     ls = &lsh_info;
2699   }
2700   if (ls != NULL) {
2701     if (is_global) {
2702       ls->print_cr("async-deflating global idle monitors, %3.7f secs, %d monitors", timer.seconds(), deflated_count);
2703     } else {
2704       ls->print_cr("jt=" INTPTR_FORMAT ": async-deflating per-thread idle monitors, %3.7f secs, %d monitors", p2i(target), timer.seconds(), deflated_count);
2705     }
2706   }
2707 }
2708 
2709 void ObjectSynchronizer::finish_deflate_idle_monitors(DeflateMonitorCounters* counters) {
2710   // Report the cumulative time for deflating each thread's idle
2711   // monitors. Note: if the work is split among more than one
2712   // worker thread, then the reported time will likely be more
2713   // than a beginning to end measurement of the phase.
2714   // Note: AsyncDeflateIdleMonitors only deflates per-thread idle
2715   // monitors at a safepoint when a special deflation has been requested.
2716   log_info(safepoint, cleanup)("deflating per-thread idle monitors, %3.7f secs, monitors=%d",
2717                                counters->per_thread_times,
2718                                OrderAccess::load_acquire(&counters->per_thread_scavenged));
2719 
2720   bool needs_special_deflation = is_special_deflation_requested();
2721   if (!AsyncDeflateIdleMonitors || needs_special_deflation) {
2722     // AsyncDeflateIdleMonitors does not use these counters unless
2723     // there is a special deflation request.
2724 
2725     OM_PERFDATA_OP(Deflations, inc(counters->n_scavenged));
2726     OM_PERFDATA_OP(MonExtant, set_value(counters->n_in_circulation));
2727   }
2728 
2729   if (log_is_enabled(Debug, monitorinflation)) {
2730     // exit_globals()'s call to audit_and_print_stats() is done
2731     // at the Info level.
2732     ObjectSynchronizer::audit_and_print_stats(false /* on_exit */);
2733   } else if (log_is_enabled(Info, monitorinflation)) {

2734     log_info(monitorinflation)("g_om_population=%d, g_om_in_use_count=%d, "
2735                                "g_om_free_count=%d, g_om_wait_count=%d",
2736                                OrderAccess::load_acquire(&g_om_population),
2737                                OrderAccess::load_acquire(&g_om_in_use_count),
2738                                OrderAccess::load_acquire(&g_om_free_count),
2739                                OrderAccess::load_acquire(&g_om_wait_count));
2740   }
2741 
2742   ForceMonitorScavenge = 0;    // Reset




2743   GVars.stw_random = os::random();
2744   GVars.stw_cycle++;
2745   if (needs_special_deflation) {
2746     set_is_special_deflation_requested(false);  // special deflation is done
2747   }
2748 }
2749 
2750 void ObjectSynchronizer::deflate_thread_local_monitors(Thread* thread, DeflateMonitorCounters* counters) {
2751   assert(SafepointSynchronize::is_at_safepoint(), "must be at safepoint");
2752 
2753   if (AsyncDeflateIdleMonitors && !is_special_deflation_requested()) {
2754     // Nothing to do if a special deflation has NOT been requested.
2755     return;
2756   }
2757 
2758   ObjectMonitor* free_head_p = NULL;  // Local SLL of scavenged monitors
2759   ObjectMonitor* free_tail_p = NULL;
2760   elapsedTimer timer;
2761 
2762   if (log_is_enabled(Info, safepoint, cleanup) ||
2763       log_is_enabled(Info, monitorinflation)) {
2764     timer.start();
2765   }
2766 
2767   // Update n_in_circulation before om_in_use_count is updated by deflation.
2768   Atomic::add(OrderAccess::load_acquire(&thread->om_in_use_count), &counters->n_in_circulation);

2769 
2770   int deflated_count = deflate_monitor_list(&thread->om_in_use_list, &thread->om_in_use_count, &free_head_p, &free_tail_p);
2771   Atomic::add(OrderAccess::load_acquire(&thread->om_in_use_count), &counters->n_in_use);




2772 
2773   if (free_head_p != NULL) {
2774     // Move the deflated ObjectMonitors back to the global free list.
2775     // No races on the working list so no need for load_acquire().
2776     guarantee(free_tail_p != NULL && deflated_count > 0, "invariant");
2777     assert(free_tail_p->_next_om == NULL, "must be NULL: _next_om="
2778            INTPTR_FORMAT, p2i(free_tail_p->_next_om));
2779     prepend_list_to_g_free_list(free_head_p, free_tail_p, deflated_count);
2780     Atomic::add(deflated_count, &counters->n_scavenged);
2781     Atomic::add(deflated_count, &counters->per_thread_scavenged);
2782   }
2783 
2784   timer.stop();
2785   // Safepoint logging cares about cumulative per_thread_times and
2786   // we'll capture most of the cost, but not the muxRelease() which
2787   // should be cheap.
2788   counters->per_thread_times += timer.seconds();
2789 


2790   LogStreamHandle(Debug, monitorinflation) lsh_debug;
2791   LogStreamHandle(Info, monitorinflation) lsh_info;
2792   LogStream* ls = NULL;
2793   if (log_is_enabled(Debug, monitorinflation)) {
2794     ls = &lsh_debug;
2795   } else if (deflated_count != 0 && log_is_enabled(Info, monitorinflation)) {
2796     ls = &lsh_info;
2797   }
2798   if (ls != NULL) {
2799     ls->print_cr("jt=" INTPTR_FORMAT ": deflating per-thread idle monitors, %3.7f secs, %d monitors", p2i(thread), timer.seconds(), deflated_count);
2800   }
2801 }
2802 
2803 // Monitor cleanup on JavaThread::exit
2804 
2805 // Iterate through monitor cache and attempt to release thread's monitors
2806 // Gives up on a particular monitor if an exception occurs, but continues
2807 // the overall iteration, swallowing the exception.
2808 class ReleaseJavaMonitorsClosure: public MonitorClosure {
2809  private:


2820 
2821 // Release all inflated monitors owned by THREAD.  Lightweight monitors are
2822 // ignored.  This is meant to be called during JNI thread detach which assumes
2823 // all remaining monitors are heavyweight.  All exceptions are swallowed.
2824 // Scanning the extant monitor list can be time consuming.
2825 // A simple optimization is to add a per-thread flag that indicates a thread
2826 // called jni_monitorenter() during its lifetime.
2827 //
2828 // Instead of No_Savepoint_Verifier it might be cheaper to
2829 // use an idiom of the form:
2830 //   auto int tmp = SafepointSynchronize::_safepoint_counter ;
2831 //   <code that must not run at safepoint>
2832 //   guarantee (((tmp ^ _safepoint_counter) | (tmp & 1)) == 0) ;
2833 // Since the tests are extremely cheap we could leave them enabled
2834 // for normal product builds.
2835 
2836 void ObjectSynchronizer::release_monitors_owned_by_thread(TRAPS) {
2837   assert(THREAD == JavaThread::current(), "must be current Java thread");
2838   NoSafepointVerifier nsv;
2839   ReleaseJavaMonitorsClosure rjmc(THREAD);

2840   ObjectSynchronizer::monitors_iterate(&rjmc);

2841   THREAD->clear_pending_exception();
2842 }
2843 
2844 const char* ObjectSynchronizer::inflate_cause_name(const InflateCause cause) {
2845   switch (cause) {
2846     case inflate_cause_vm_internal:    return "VM Internal";
2847     case inflate_cause_monitor_enter:  return "Monitor Enter";
2848     case inflate_cause_wait:           return "Monitor Wait";
2849     case inflate_cause_notify:         return "Monitor Notify";
2850     case inflate_cause_hash_code:      return "Monitor Hash Code";
2851     case inflate_cause_jni_enter:      return "JNI Monitor Enter";
2852     case inflate_cause_jni_exit:       return "JNI Monitor Exit";
2853     default:
2854       ShouldNotReachHere();
2855   }
2856   return "Unknown";
2857 }
2858 
2859 //------------------------------------------------------------------------------
2860 // Debugging code


2874 u_char* ObjectSynchronizer::get_gvars_stw_random_addr() {
2875   return (u_char*)&GVars.stw_random;
2876 }
2877 
2878 void ObjectSynchronizer::audit_and_print_stats(bool on_exit) {
2879   assert(on_exit || SafepointSynchronize::is_at_safepoint(), "invariant");
2880 
2881   LogStreamHandle(Debug, monitorinflation) lsh_debug;
2882   LogStreamHandle(Info, monitorinflation) lsh_info;
2883   LogStreamHandle(Trace, monitorinflation) lsh_trace;
2884   LogStream* ls = NULL;
2885   if (log_is_enabled(Trace, monitorinflation)) {
2886     ls = &lsh_trace;
2887   } else if (log_is_enabled(Debug, monitorinflation)) {
2888     ls = &lsh_debug;
2889   } else if (log_is_enabled(Info, monitorinflation)) {
2890     ls = &lsh_info;
2891   }
2892   assert(ls != NULL, "sanity check");
2893 





2894   // Log counts for the global and per-thread monitor lists:
2895   int chk_om_population = log_monitor_list_counts(ls);
2896   int error_cnt = 0;
2897 
2898   ls->print_cr("Checking global lists:");
2899 
2900   // Check g_om_population:
2901   if (OrderAccess::load_acquire(&g_om_population) == chk_om_population) {
2902     ls->print_cr("g_om_population=%d equals chk_om_population=%d",
2903                  OrderAccess::load_acquire(&g_om_population),
2904                  chk_om_population);
2905   } else {
2906     // With lock free access to the monitor lists, it is possible for
2907     // log_monitor_list_counts() to return a value that doesn't match
2908     // g_om_population. So far a higher value has been seen in testing
2909     // so something is being double counted by log_monitor_list_counts().
2910     ls->print_cr("WARNING: g_om_population=%d is not equal to "
2911                  "chk_om_population=%d",
2912                  OrderAccess::load_acquire(&g_om_population),
2913                  chk_om_population);

2914   }
2915 
2916   // Check g_om_in_use_list and g_om_in_use_count:
2917   chk_global_in_use_list_and_count(ls, &error_cnt);
2918 
2919   // Check g_free_list and g_om_free_count:
2920   chk_global_free_list_and_count(ls, &error_cnt);
2921 
2922   if (HandshakeAfterDeflateIdleMonitors) {
2923     // Check g_wait_list and g_om_wait_count:
2924     chk_global_wait_list_and_count(ls, &error_cnt);
2925   }
2926 
2927   ls->print_cr("Checking per-thread lists:");
2928 
2929   for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = jtiwh.next(); ) {
2930     // Check om_in_use_list and om_in_use_count:
2931     chk_per_thread_in_use_list_and_count(jt, ls, &error_cnt);
2932 
2933     // Check om_free_list and om_free_count:
2934     chk_per_thread_free_list_and_count(jt, ls, &error_cnt);
2935   }
2936 
2937   if (error_cnt == 0) {
2938     ls->print_cr("No errors found in monitor list checks.");
2939   } else {
2940     log_error(monitorinflation)("found monitor list errors: error_cnt=%d", error_cnt);
2941   }
2942 
2943   if ((on_exit && log_is_enabled(Info, monitorinflation)) ||
2944       (!on_exit && log_is_enabled(Trace, monitorinflation))) {
2945     // When exiting this log output is at the Info level. When called
2946     // at a safepoint, this log output is at the Trace level since
2947     // there can be a lot of it.
2948     log_in_use_monitor_details(ls);
2949   }
2950 
2951   ls->flush();
2952 
2953   guarantee(error_cnt == 0, "ERROR: found monitor list errors: error_cnt=%d", error_cnt);
2954 }
2955 
2956 // Check a free monitor entry; log any errors.
2957 void ObjectSynchronizer::chk_free_entry(JavaThread* jt, ObjectMonitor* n,
2958                                         outputStream * out, int *error_cnt_p) {
2959   stringStream ss;
2960   if (n->is_busy()) {
2961     if (jt != NULL) {
2962       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
2963                     ": free per-thread monitor must not be busy: %s", p2i(jt),
2964                     p2i(n), n->is_busy_to_string(&ss));
2965     } else {
2966       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": free global monitor "
2967                     "must not be busy: %s", p2i(n), n->is_busy_to_string(&ss));
2968     }
2969     *error_cnt_p = *error_cnt_p + 1;
2970   }
2971   if (n->header().value() != 0) {
2972     if (jt != NULL) {
2973       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
2974                     ": free per-thread monitor must have NULL _header "
2975                     "field: _header=" INTPTR_FORMAT, p2i(jt), p2i(n),
2976                     n->header().value());
2977       *error_cnt_p = *error_cnt_p + 1;
2978     } else if (!AsyncDeflateIdleMonitors) {
2979       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": free global monitor "
2980                     "must have NULL _header field: _header=" INTPTR_FORMAT,
2981                     p2i(n), n->header().value());

2982       *error_cnt_p = *error_cnt_p + 1;
2983     }
2984   }
2985   if (n->object() != NULL) {
2986     if (jt != NULL) {
2987       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
2988                     ": free per-thread monitor must have NULL _object "
2989                     "field: _object=" INTPTR_FORMAT, p2i(jt), p2i(n),
2990                     p2i(n->object()));
2991     } else {
2992       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": free global monitor "
2993                     "must have NULL _object field: _object=" INTPTR_FORMAT,
2994                     p2i(n), p2i(n->object()));
2995     }
2996     *error_cnt_p = *error_cnt_p + 1;
2997   }
2998 }
2999 
3000 // Check the global free list and count; log the results of the checks.
3001 void ObjectSynchronizer::chk_global_free_list_and_count(outputStream * out,
3002                                                         int *error_cnt_p) {
3003   int chk_om_free_count = 0;
3004   for (ObjectMonitor* n = OrderAccess::load_acquire(&g_free_list); n != NULL; n = unmarked_next(n)) {
3005     chk_free_entry(NULL /* jt */, n, out, error_cnt_p);
3006     chk_om_free_count++;
3007   }
3008   if (OrderAccess::load_acquire(&g_om_free_count) == chk_om_free_count) {
3009     out->print_cr("g_om_free_count=%d equals chk_om_free_count=%d",
3010                   OrderAccess::load_acquire(&g_om_free_count),
3011                   chk_om_free_count);
3012   } else {
3013     // With lock free access to g_free_list, it is possible for an
3014     // ObjectMonitor to be prepended to g_free_list after we started
3015     // calculating chk_om_free_count so g_om_free_count may not
3016     // match anymore.
3017     out->print_cr("WARNING: g_om_free_count=%d is not equal to "
3018                   "chk_om_free_count=%d",
3019                   OrderAccess::load_acquire(&g_om_free_count),
3020                   chk_om_free_count);
3021   }
3022 }
3023 
3024 // Check the global wait list and count; log the results of the checks.
3025 void ObjectSynchronizer::chk_global_wait_list_and_count(outputStream * out,
3026                                                         int *error_cnt_p) {
3027   int chk_om_wait_count = 0;
3028   for (ObjectMonitor* n = OrderAccess::load_acquire(&g_wait_list); n != NULL; n = unmarked_next(n)) {
3029     // Rules for g_wait_list are the same as of g_free_list:
3030     chk_free_entry(NULL /* jt */, n, out, error_cnt_p);
3031     chk_om_wait_count++;
3032   }
3033   if (OrderAccess::load_acquire(&g_om_wait_count) == chk_om_wait_count) {
3034     out->print_cr("g_om_wait_count=%d equals chk_om_wait_count=%d",
3035                   OrderAccess::load_acquire(&g_om_wait_count),
3036                   chk_om_wait_count);
3037   } else {
3038     out->print_cr("ERROR: g_om_wait_count=%d is not equal to "
3039                   "chk_om_wait_count=%d",
3040                   OrderAccess::load_acquire(&g_om_wait_count),
3041                   chk_om_wait_count);
3042     *error_cnt_p = *error_cnt_p + 1;
3043   }
3044 }
3045 
3046 // Check the global in-use list and count; log the results of the checks.
3047 void ObjectSynchronizer::chk_global_in_use_list_and_count(outputStream * out,
3048                                                           int *error_cnt_p) {
3049   int chk_om_in_use_count = 0;
3050   for (ObjectMonitor* n = OrderAccess::load_acquire(&g_om_in_use_list); n != NULL; n = unmarked_next(n)) {
3051     chk_in_use_entry(NULL /* jt */, n, out, error_cnt_p);
3052     chk_om_in_use_count++;
3053   }
3054   if (OrderAccess::load_acquire(&g_om_in_use_count) == chk_om_in_use_count) {
3055     out->print_cr("g_om_in_use_count=%d equals chk_om_in_use_count=%d",
3056                   OrderAccess::load_acquire(&g_om_in_use_count),
3057                   chk_om_in_use_count);
3058   } else {
3059     // With lock free access to the monitor lists, it is possible for
3060     // an exiting JavaThread to put its in-use ObjectMonitors on the
3061     // global in-use list after chk_om_in_use_count is calculated above.
3062     out->print_cr("WARNING: g_om_in_use_count=%d is not equal to chk_om_in_use_count=%d",
3063                   OrderAccess::load_acquire(&g_om_in_use_count),
3064                   chk_om_in_use_count);
3065   }
3066 }
3067 
3068 // Check an in-use monitor entry; log any errors.
3069 void ObjectSynchronizer::chk_in_use_entry(JavaThread* jt, ObjectMonitor* n,
3070                                           outputStream * out, int *error_cnt_p) {
3071   if (n->header().value() == 0) {
3072     if (jt != NULL) {
3073       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
3074                     ": in-use per-thread monitor must have non-NULL _header "
3075                     "field.", p2i(jt), p2i(n));
3076     } else {
3077       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": in-use global monitor "
3078                     "must have non-NULL _header field.", p2i(n));
3079     }
3080     *error_cnt_p = *error_cnt_p + 1;
3081   }
3082   if (n->object() == NULL) {
3083     if (jt != NULL) {
3084       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT


3112       out->print_cr("ERROR: jt=" INTPTR_FORMAT ", monitor=" INTPTR_FORMAT
3113                     ": in-use per-thread monitor's object does not refer "
3114                     "to the same monitor: obj=" INTPTR_FORMAT ", mark="
3115                     INTPTR_FORMAT ", obj_mon=" INTPTR_FORMAT, p2i(jt),
3116                     p2i(n), p2i(obj), mark.value(), p2i(obj_mon));
3117     } else {
3118       out->print_cr("ERROR: monitor=" INTPTR_FORMAT ": in-use global "
3119                     "monitor's object does not refer to the same monitor: obj="
3120                     INTPTR_FORMAT ", mark=" INTPTR_FORMAT ", obj_mon="
3121                     INTPTR_FORMAT, p2i(n), p2i(obj), mark.value(), p2i(obj_mon));
3122     }
3123     *error_cnt_p = *error_cnt_p + 1;
3124   }
3125 }
3126 
3127 // Check the thread's free list and count; log the results of the checks.
3128 void ObjectSynchronizer::chk_per_thread_free_list_and_count(JavaThread *jt,
3129                                                             outputStream * out,
3130                                                             int *error_cnt_p) {
3131   int chk_om_free_count = 0;
3132   for (ObjectMonitor* n = OrderAccess::load_acquire(&jt->om_free_list); n != NULL; n = unmarked_next(n)) {
3133     chk_free_entry(jt, n, out, error_cnt_p);
3134     chk_om_free_count++;
3135   }
3136   if (OrderAccess::load_acquire(&jt->om_free_count) == chk_om_free_count) {
3137     out->print_cr("jt=" INTPTR_FORMAT ": om_free_count=%d equals "
3138                   "chk_om_free_count=%d", p2i(jt),
3139                   OrderAccess::load_acquire(&jt->om_free_count),
3140                   chk_om_free_count);
3141   } else {
3142     out->print_cr("ERROR: jt=" INTPTR_FORMAT ": om_free_count=%d is not "
3143                   "equal to chk_om_free_count=%d", p2i(jt),
3144                   OrderAccess::load_acquire(&jt->om_free_count),
3145                   chk_om_free_count);
3146     *error_cnt_p = *error_cnt_p + 1;
3147   }
3148 }
3149 
3150 // Check the thread's in-use list and count; log the results of the checks.
3151 void ObjectSynchronizer::chk_per_thread_in_use_list_and_count(JavaThread *jt,
3152                                                               outputStream * out,
3153                                                               int *error_cnt_p) {
3154   int chk_om_in_use_count = 0;
3155   for (ObjectMonitor* n = OrderAccess::load_acquire(&jt->om_in_use_list); n != NULL; n = unmarked_next(n)) {
3156     chk_in_use_entry(jt, n, out, error_cnt_p);
3157     chk_om_in_use_count++;
3158   }
3159   if (OrderAccess::load_acquire(&jt->om_in_use_count) == chk_om_in_use_count) {
3160     out->print_cr("jt=" INTPTR_FORMAT ": om_in_use_count=%d equals "
3161                   "chk_om_in_use_count=%d", p2i(jt),
3162                   OrderAccess::load_acquire(&jt->om_in_use_count),
3163                   chk_om_in_use_count);
3164   } else {
3165     out->print_cr("ERROR: jt=" INTPTR_FORMAT ": om_in_use_count=%d is not "
3166                   "equal to chk_om_in_use_count=%d", p2i(jt),
3167                   OrderAccess::load_acquire(&jt->om_in_use_count),
3168                   chk_om_in_use_count);
3169     *error_cnt_p = *error_cnt_p + 1;
3170   }
3171 }
3172 
3173 // Log details about ObjectMonitors on the in-use lists. The 'BHL'
3174 // flags indicate why the entry is in-use, 'object' and 'object type'
3175 // indicate the associated object and its type.
3176 void ObjectSynchronizer::log_in_use_monitor_details(outputStream * out) {






3177   stringStream ss;
3178   if (OrderAccess::load_acquire(&g_om_in_use_count) > 0) {
3179     out->print_cr("In-use global monitor info:");
3180     out->print_cr("(B -> is_busy, H -> has hash code, L -> lock status)");
3181     out->print_cr("%18s  %s  %7s  %18s  %18s",
3182                   "monitor", "BHL", "ref_cnt", "object", "object type");
3183     out->print_cr("==================  ===  =======  ==================  ==================");
3184     for (ObjectMonitor* n = OrderAccess::load_acquire(&g_om_in_use_list); n != NULL; n = unmarked_next(n)) {
3185       const oop obj = (oop) n->object();
3186       const markWord mark = n->header();
3187       ResourceMark rm;
3188       out->print(INTPTR_FORMAT "  %d%d%d  %7d  " INTPTR_FORMAT "  %s",
3189                  p2i(n), n->is_busy() != 0, mark.hash() != 0,
3190                  n->owner() != NULL, (int)n->ref_count(), p2i(obj),
3191                  obj->klass()->external_name());
3192       if (n->is_busy() != 0) {
3193         out->print(" (%s)", n->is_busy_to_string(&ss));
3194         ss.reset();
3195       }
3196       out->cr();
3197     }
3198   }
3199 




3200   out->print_cr("In-use per-thread monitor info:");
3201   out->print_cr("(B -> is_busy, H -> has hash code, L -> lock status)");
3202   out->print_cr("%18s  %18s  %s  %7s  %18s  %18s",
3203                 "jt", "monitor", "BHL", "ref_cnt", "object", "object type");
3204   out->print_cr("==================  ==================  ===  =======  ==================  ==================");
3205   for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = jtiwh.next(); ) {
3206     for (ObjectMonitor* n = OrderAccess::load_acquire(&jt->om_in_use_list); n != NULL; n = unmarked_next(n)) {
3207       const oop obj = (oop) n->object();
3208       const markWord mark = n->header();
3209       ResourceMark rm;
3210       out->print(INTPTR_FORMAT "  " INTPTR_FORMAT "  %d%d%d  %7d  "
3211                  INTPTR_FORMAT "  %s", p2i(jt), p2i(n), n->is_busy() != 0,
3212                  mark.hash() != 0, n->owner() != NULL, (int)n->ref_count(),
3213                  p2i(obj), obj->klass()->external_name());
3214       if (n->is_busy() != 0) {
3215         out->print(" (%s)", n->is_busy_to_string(&ss));
3216         ss.reset();
3217       }
3218       out->cr();
3219     }
3220   }
3221 
3222   out->flush();
3223 }
3224 
3225 // Log counts for the global and per-thread monitor lists and return
3226 // the population count.
3227 int ObjectSynchronizer::log_monitor_list_counts(outputStream * out) {
3228   int pop_count = 0;
3229   out->print_cr("%18s  %10s  %10s  %10s  %10s",
3230                 "Global Lists:", "InUse", "Free", "Wait", "Total");
3231   out->print_cr("==================  ==========  ==========  ==========  ==========");
3232   out->print_cr("%18s  %10d  %10d  %10d  %10d", "",
3233                 OrderAccess::load_acquire(&g_om_in_use_count),
3234                 OrderAccess::load_acquire(&g_om_free_count),
3235                 OrderAccess::load_acquire(&g_om_wait_count),
3236                 OrderAccess::load_acquire(&g_om_population));
3237   pop_count += OrderAccess::load_acquire(&g_om_in_use_count) +
3238                OrderAccess::load_acquire(&g_om_free_count);
3239   if (HandshakeAfterDeflateIdleMonitors) {
3240     pop_count += OrderAccess::load_acquire(&g_om_wait_count);
3241   }
3242 
3243   out->print_cr("%18s  %10s  %10s  %10s",
3244                 "Per-Thread Lists:", "InUse", "Free", "Provision");
3245   out->print_cr("==================  ==========  ==========  ==========");
3246 
3247   for (JavaThreadIteratorWithHandle jtiwh; JavaThread *jt = jtiwh.next(); ) {
3248     out->print_cr(INTPTR_FORMAT "  %10d  %10d  %10d", p2i(jt),
3249                   OrderAccess::load_acquire(&jt->om_in_use_count),
3250                   OrderAccess::load_acquire(&jt->om_free_count),
3251                   jt->om_free_provision);
3252     pop_count += OrderAccess::load_acquire(&jt->om_in_use_count) +
3253                  OrderAccess::load_acquire(&jt->om_free_count);
3254   }
3255   return pop_count;
3256 }
3257 
3258 #ifndef PRODUCT
3259 
3260 // Check if monitor belongs to the monitor cache
3261 // The list is grow-only so it's *relatively* safe to traverse
3262 // the list of extant blocks without taking a lock.
3263 
3264 int ObjectSynchronizer::verify_objmon_isinpool(ObjectMonitor *monitor) {
3265   PaddedObjectMonitor* block = OrderAccess::load_acquire(&g_block_list);
3266   while (block != NULL) {
3267     assert(block->object() == CHAINMARKER, "must be a block header");
3268     if (monitor > &block[0] && monitor < &block[_BLOCKSIZE]) {
3269       address mon = (address)monitor;
3270       address blk = (address)block;
3271       size_t diff = mon - blk;
3272       assert((diff % sizeof(PaddedObjectMonitor)) == 0, "must be aligned");
3273       return 1;
3274     }
3275     // unmarked_next() is not needed with g_block_list (no next field marking).
3276     block = (PaddedObjectMonitor*)OrderAccess::load_acquire(&block->_next_om);
3277   }
3278   return 0;
3279 }
3280 
3281 #endif
< prev index next >