How do I read logs and take a thread dump of the Workflow Server on CP4BA (no SystemOut.log)?

Question

Support asks for SystemOut.log and javacores. Our BAW runs in pods; where are the logs, how do I enable trace for the process engine, and how do I produce a thread dump?

Answer 1

Everything is done with oc; the files you know still exist inside the container:

# logs (Liberty messages.log goes to stdout)
oc logs -f icp4adeploy-bawins1-baw-server-0 -n cp4ba
oc logs --previous icp4adeploy-bawins1-baw-server-0 -n cp4ba          # crashed container
oc exec icp4adeploy-bawins1-baw-server-0 -- ls /logs/application/liberty/   # messages.log, trace.log, ffdc/
oc cp cp4ba/icp4adeploy-bawins1-baw-server-0:/logs/application/liberty/messages.log ./messages.log

# trace: set the Liberty trace specification through the CR (workflow server section) - takes effect without a manual restart
spec:
  baw_configuration:
  - name: bawins1
    logs:
      trace_specification: "*=info:com.ibm.bpm.*=fine:WLE.*=fine"      # or the engine strings support asks for
    # older releases: 'trace_specification' under 'log' / 'logs' - check the CR reference of your version

# thread dumps / heap dumps: Liberty's server dump inside the pod
oc exec icp4adeploy-bawins1-baw-server-0 -- /opt/ibm/wlp/bin/server javadump defaultServer --include=thread
oc exec icp4adeploy-bawins1-baw-server-0 -- /opt/ibm/wlp/bin/server dump defaultServer --include=heap   # heap dump (large)
oc exec icp4adeploy-bawins1-baw-server-0 -- ls /opt/ibm/wlp/output/defaultServer/   # javacore.*.txt, heapdump.*.phd
# quick alternative: kill -3 on the JVM pid
oc exec icp4adeploy-bawins1-baw-server-0 -- bash -c 'kill -3 $(pgrep -f defaultServer)'

Central logging: the OpenShift logging stack (Loki / Elasticsearch) collects stdout of every pod; filter on the pod label app=icp4adeploy-bawins1-baw-server. Metrics: Liberty exposes /metrics (MicroProfile) scraped by the cluster Prometheus; the operator also reports the pods' readiness in the CR status. For a support case collect: the CR (oc get icp4acluster -o yaml), operator logs (oc logs deploy/ibm-cp4a-operator), the pod logs and dumps above, and the output of oc describe pod.

References

• CP4BA documentation - troubleshooting Business Automation Workflow (logs, traces, dumps)