Universal Forwarder Troubleshooting Commands
If your Deployment Server does not display a record for a particular Universal Forwarder, check the following:
1. Is the universal forwarder contacting the deployment server component on the intermediate forwarder?
$SPLUNK_HOME/bin/splunk show deploy-poll
If you need to reset the deployment server URI, run:
$SPLUNK_HOME/bin/splunk set deploy-poll <ds-name-or-ip>:8089
$SPLUNK_HOME/bin/splunk restart
2. Does the universal forwarder know where to send its data?
$SPLUNK_HOME/bin/splunk btool outputs list | grep "server"
If you see any mistakes, go back to the deployment server (your intermediate forwarder) and edit the $SPLUNK_HOME/etc/deployment-apps/uf_base/local/outputs.conf file.
To re-deploy to edited app, run:
$SPLUNK_HOME/bin/splunk reload deploy-server
3. Is forwarding active on the universal forwarder?
$SPLUNK_HOME/bin/splunk list forward-server
If not, $SPLUNK_HOME/bin/splunk restart and check the forwarder internal log:
cat $SPLUNK_HOME/var/log/splunk/splunkd.log | grep 'ERROR\|WARN'
4. Is the receiving port on the intermediate forwarder correct?
$SPLUNK_HOME/splunk display listen
5. Check the intermediate forwarder's splunkd.log:
egrep 'ERROR|WARN' $SPLUNK_HOME/var/log/splunk/splunkd.log
6. Deployment server reload:
$SPLUNK_HOME/bin/splunk reload deploy-server
Note: Based on content from the Splunk Cloud Admin course with my edits. Feel free to suggest tweaks or additions.