I'm looking into Fluent-Bit as a method of shipping logs to a Splunk Indexer. And the goal is to send logs securely from fluent-bit to a Splunk Indexer.

I currently have a free-tier Splunk sandbox setup for testing purposes. And I'm currently testing with the default certificate that comes prepackaged with Splunk. I believe I have to enable HTTPS for the web server, as HEC uses this as well as the Web Server. So that's done. Though the cert domain don't match currently (aws web server).

Within Fluent-Bit I'm currently testing this configuration, but it is failing. I'm not sure why yet:

[OUTPUT]
    Name                        splunk
    Match                       RuntimeLogs
    Host                        192.168.110.45
    Port                        8088
    Splunk_Token                asdf-asdf-asdf-asdf-cbd182697ef2
    Event_sourcetype            runtime:log
    TLS                         On
    # Not sure if TLS.VERIFY should be on or off
    TLS.VERIFY                  On
    tls.crt_file                /apps01/wdtvs/splunk/etc/auth/splunkweb/cert.pem
    tls.key_file                /apps01/wdtvs/splunk/etc/auth/splunkweb/privkey.pem
    #Unsure if I need to configure the http user and password values
    http_user                   U$3rn@ME1!
    http_passwd                 P@ssW0rd!
    Splunk_Send_Raw             On

I believe, under splunkweb/ this is the key and certificate I should be using. Even reviewing the fluent-bit logs, this cert/key pair seem to work without issue. Fluent-Bit starts up without issue... and there aren't any new logs being sent.

Reviewing fluent-bit's logs reveals these error messages consistently:

[ warn] [engine] failed to flush chunk '18123-1684289660.696297957.flb', retry in 11 seconds: task_id=6, input=tail.1 > output=splunk.1 (out_id=1) [error] [tls] error: unexpected EOF [error] [engine] chunk '18123-1684289660.858658465.flb' cannot be retried: task_id=4, input=tail.0 > output=splunk.0

I'm not sure what to do at this point in time to resolve the error with Fluent-Bit. I do see these lines in the Splunkd.log file, but I'm unsure if these are red herrings or actual errors related to my problem. Any advice is appreciated:

INFO  TailReader [27529 tailreader0] - Batch input finished reading file='/apps01/wdtvs/splunk/var/spool/splunk/tracker.log
WARN  SSLCommon [27843 webui] - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
WARN  HttpListener [27843 webui] - Socket error from 192.168.110.45:10550 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown

At this point I'm at a bit of a loss. Any advice is appreciated.