A checklist that finds the cause of most "production is slow / broken" tickets in a clustered BPM / BAW environment, in the order that pays off:
- Scope it: one user, one app, one member, or everything? Compare members (hit each one directly through its port, bypassing the load balancer). Look at the time line: change windows, deployments (new snapshot?), batch jobs.
- Logs: SystemOut.log / SystemErr.log of every member around the time (grep for CWLLG, CWTBG, WSVR0605W hung threads, DSRA data source errors, CWSIx messaging, OutOfMemory), the IHS error_log and plug-in log for routing errors (503, no member available).
- Health of the engine: Process Admin > Event Manager > Monitor (queue depth, schedulers running?), Process Inspector for failed instances and stuck tokens (question 52), Instances failed list; Process Admin > Server Admin > Caches for cache statistics.
- Resources: JVM heap and GC (verbose GC or the PMI viewer), thread pools (WebContainer, EM / "BPM Process Server" thread pool), JDBC connection pool usage (Tivoli Performance Viewer > JDBC connection pools: waiting threads = pool too small or leaked connections), disk (logs, SIBus file store), CPU.
- Database: long-running statements (lsw_bpd_instance, lsw_task queries without indexes, PDW loads), lock waits, statistics out of date (runstats), table growth (instance cleanup overdue: run the instance deletion / BPMDeleteSnapshot housekeeping).
- Threads: three javacores 30 s apart on the slow member; look for threads stuck in the same place (Rhino loops, LDAP calls, remote web services, database waits).
- Network / security: LDAP response times (question 49), SSL handshakes to back ends, DNS.
- Capture for IBM: the BAW MustGather (logs, javacores, configuration, the Support Tools collectors) if you open a case.
# quick triage commands on a member
grep -E "CWLLG|CWTBG|WSVR0605W|OutOfMemory" SystemOut.log | tail -50
kill -3 <pid>; sleep 30; kill -3 <pid>; sleep 30; kill -3 <pid> # javacores in the profile's logs dir
curl -sk -u admin:pw "https://member1:9443/rest/bpm/wle/v1/systems" # engine answering on this member?
References