Sign in Subscribe
Siem Engineer

Siem Engineer

Understanding Splunk

1. GUI

a. Macros

  • Splunk quick-call commands

b. Indexes

  • Where the data is kept

c. Source

  • Where is the source of the data

d. Source Types

  • What type of data is stored

e. Data Models

  • Efficiently pull pieces of data instead of all fields

f. Saved Searches

  • Automated queries that will populate on the interval you proclaim

g. Lookups

  • Table files and translation documents to store lists, tables, and graphs to be used in searching

h. Apps

  • Graphical depictions to display regularly run searches in a user-friendly way
  • May require additional add-ons.

i. Add-ons

  • Technical/literal translations to pull additional information from data to make searching easier or more intuitive

j. Forwarding and Receiving

  • Splunk data coming in; all data going out

k. Data Inputs

  • Non-Splunk data coming in (from open ports or monitored files)

l. Forwarder Management

  • Manage connected universal forwarders/endpoints

m. Indexer Clustering

  1. Enable Indexer Clustering
  2. Manage connected indexes and search heads
  3. Push indexer configuration bundle

n. Licensing

  • All Splunk instances require at least one license to ingest
  • Can configure license manager/monitor license usage

o. Ingest Actions

  • Filter out unwanted data

2. Files

a. inputs.conf

  • Specifies where data will be collected from.
    • [splunktcp://<port>] (Incoming Splunk data, from a Splunk-based forwarder)
    • [tcp(udp)://<port>] (Incoming TCP/UDP data)
    • [tcp(udp)-ssl://<port>] (Incoming encrypted traffic, requires an SSL.pem and password)

b. outputs.conf

  • Specifies where to send data.
  • Must be configured on forwarders.

c. props.conf

  • Generates additional fields from existing field data (calculated), alias data, and call transforms.
  • Forwarders and ingestion points must have identical copies of this file.
  • Calls:
    • EVAL
    • ALIAS
    • TRANSFORMS

d. transforms.conf

  • Uses regex and metadata to scan data for new (extracted) fields.
  • Forwarders and ingestion points must have identical copies of this file.

e. limits.conf

  • Controls Splunk resource usage.
    • Example: [thruput] maxKBps = 0 (removes limit on ingestion/forwarding)

f. server.conf

  • Configures Splunk instance settings.
  • Can modify:
    • Hostname
    • Clustering
    • Management

g. deploymentclient.conf

  • Configures the Splunk universal forwarder.

3. Directories

Directory Description
/opt/splunk $SPLUNK_HOME
/opt/splunk/bin Splunk binaries and commands (start, stop, restart)
/opt/splunk/var/log/splunk Logs for errors and troubleshooting
/opt/splunk/etc/apps Add-ons/apps installed locally
/opt/splunk/etc/deployment-apps Add-ons sent to endpoints
/opt/splunk/etc/system/local Configuration changes affecting the entire Splunk instance

4. Types of Add-ons

a. Manager Add-ons (MA)

  • indexes.conf

b. Technical Add-ons (TA)

  • inputs.conf
  • outputs.conf
  • transforms.conf
  • props.conf
  • limits.conf

5. Add-ons

  • Splunk_TA_windows – Windows endpoints, contains necessary props and transforms.
  • WEC – For a specific Windows box, includes inputs and outputs configuration.
  • TA-onionForwarder-main – Security Onion endpoints, inputs for 60+ indexes.
  • TA-onionImport-main – Companion for forwarders and indexers, holds necessary props and transforms.

6. Machines

Role Description
Manager (Cluster) The boss of the indexers, ensuring they work together and share data properly.
Indexer(s) The library that stores and organizes all logs for retrieval.
Search Head(s) The detective that searches through stored logs for answers.
Captain The leader of the search heads, ensuring teamwork and coordination.
Deployer Provides instructions to search heads in a cluster.
Deployment Server Distributes updates and tools to smaller Splunk helpers like forwarders.
Heavy Forwarder A smart messenger that can read and filter logs before sending them to indexers.

7. Ports

Port Protocol Used By Purpose
8000 HTTP(S) Splunk Web UI Web interface for users.
8089 HTTPS Splunk Management REST API and internal communication.
9997 TCP Indexers Default port for receiving forwarded data.
514 TCP/UDP Syslog Server Receives Syslog data.
8088 HTTP HTTP Event Collector (HEC) Receives events via HTTP/HTTPS.
8191 TCP KV Store Splunk's Key-Value Store (MongoDB-based).
9200 HTTP Splunk ES Used for Splunk Enterprise Security (ES).
9998 TCP Indexer Clustering Communication between indexers.
17000 TCP Search Head Cluster Used for SHC captain election and synchronization.

8. How App Folders Translate

Manager Indexer Search Head Deployer Deployment Manager Heavy Forwarder
/manager-apps → /peer-apps /apps ← /apps /deployment-apps /apps

9. Troubleshooting

1. Check Real-Time Logs for Issues

tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log | grep -E "ERROR|WARN|FATAL"

2. Check Indexer Connection Issues

splunk list forward-server

3. Verify Data Inputs Are Active

splunk list monitor

4. Check Search Head Cluster Status

splunk show shcluster-status

5. Check Disk Space Issues (Splunk May Stop Ingesting)

df -h | grep splunk

6. Check Running Processes (Splunk Daemon)

ps aux | grep splunk

7. Restart Splunk If Needed (Safely)

splunk stop
splunk start

8. Test If Splunk Web and REST API Are Responding

curl -k https://localhost:8089/services/server/info -u admin:<password>

9. Check Splunk’s Port Usage

netstat -tulnp | grep splunk

10. Find Licensing Issues

splunk list licenser-pools

11. Investigate Why an App Isn’t Loading

tail -f $SPLUNK_HOME/var/log/splunk/python.log

12. Check Indexing Rate (Ingest Performance)

splunk query 'index=_internal source=*metrics.log group=per_index_thruput | stats sum(kb) by series'