How do I debug a service flow running on a Process Server in production (trace, instance data, replay)?

Question

A service flow fails in production for some instances only. We cannot attach the designer's debugger there. What can we do to see the inputs, the failing step and the exception without redeploying?

Answer 1

Production troubleshooting tools, in the order that gives the fastest answer:

1. The instance's error: Process Admin > Process Inspector > the failed instance > the failed step shows the exception message and the stack of the service; the REST equivalent: PUT /rest/bpm/wle/v1/process/errors?instanceIds=… and the instance's executionTree. Nine times out of ten the message names the step and the data problem.
2. The instance data at the time: GET /rest/bpm/wle/v1/process/{piid}?parts=data shows the variables the step received; compare a failing and a working instance.
3. Engine trace for the service: enable a targeted trace on the member without restart - admin console > Troubleshooting > Logs and trace > the member > Change log detail levels > runtime: the WLE.* loggers (WLE.wle_engine, WLE.wle_javascript, WLE.wle_rest) at fine / all for a few minutes, reproduce with Retry on the failed instance, then reset to info; the trace log shows each script's execution and the exception with line numbers.
4. Retry with added logging: if the app can be redeployed quickly, add log.info(JSON.stringify(tw.local.x)) lines around the failing step in a new snapshot, migrate the failed instances to it (migration policy), retry - the logs tell the story. Keep the log statements behind an EPV switch so that they can stay.
5. Replay: Process Admin's Retry on a failed instance re-runs the failed step with the same data; for UCA / timer work the Event Manager's replay; for a service called from a coach, run the service directly with the same inputs through the REST API on a test server that has a copy of the data.
6. Environment differences: environment variables / EPV values on the production snapshot, Server definitions (host, credentials), truststore, the toolkit snapshot versions - compare with test using the Operations REST (env_vars, epvs, containers).

# targeted trace on a cluster member without restart (wsadmin Jython)
AdminControl.setAttribute(AdminControl.queryNames('type=TraceService,process=Member1,*'), 'traceSpecification', '*=info:WLE.wle_engine=fine:WLE.wle_javascript=all')
# ... reproduce ... then
AdminControl.setAttribute(AdminControl.queryNames('type=TraceService,process=Member1,*'), 'traceSpecification', '*=info')
# CP4BA: the trace specification field of the workflow CR (logs section) - operator applies it without redeploying the app

Prevention: services that throw typed errors with the input summary in the message (throw new Error("Validate order failed for " + tw.local.order.number + ": " + e.message)) make step 1 sufficient in most cases.

References

• BAW documentation - troubleshooting and tracing (WLE trace loggers)
• Process REST API - errors, executionTree, data