Outgoing mail is a WebSphere / BAW server setting, per environment; the Process Center is just another cell. Two things to configure:
1. The SMTP server the process engine uses for the OOB Send E-Mail service (System Data toolkit) - set in 100Custom.xml of the cell (<profile>/config/cells/<cell>/nodes/<node>/servers/<server>/process-center/config/100Custom.xml, or process-server) and restart:
<properties>
<server merge="mergeChildren">
<email merge="mergeChildren">
<mail-server-address merge="replace">smtp.example.com</mail-server-address>
<mail-server-port merge="replace">587</mail-server-port>
<valid-from-required merge="replace">true</valid-from-required>
<default-from-address merge="replace">bpm@example.com</default-from-address>
<!-- authentication (BAW 20+): -->
<mail-server-user merge="replace">bpm@example.com</mail-server-user>
<mail-server-password merge="replace">{xor}encoded</mail-server-password>
<mail-server-transport merge="replace">smtps or smtp</mail-server-transport>
</email>
</server>
</properties>On BAW 20+ the same values are available in the Process Admin console (Server Admin > Admin Tools > Manage E-Mail settings on some builds) and through BPMConfig -update; the password can be encoded with PropFilePasswordEncoder.
2. Task notification / portal e-mails ("you have a new task") use the same server; enable them per user in Process Portal preferences and globally with <task-notification> settings in the same file. For TLS-only relays (Office 365, Gmail) import the relay's certificate chain into the cell truststore (retrieveSigners or the admin console) and use port 587 with STARTTLS.
Test it without a process: WebSphere Mail sessions are not used by BPM; run the OOB Send E-Mail service (System Data toolkit) from the designer with your own address as the recipient and read SystemOut.log for the SMTP error if nothing arrives.
References