Is it possible to reconfigure journald.conf without rebuilding bottlerocket?

[yaroslav-nakonechnikov]

Hello,

we are using bottlerocket as nodes for EKS.
And as our central solution for logging is splunk - i found how to start host-container with splunk, which has access to /.bottlerocket/

and there is no any issue with getting logs from pods/containers, but system logs are not working as expected.

Splunk docs saying that it is possible to read journald files, but there is no way to point where to read it.
And i'm thinking to reconfigure a bit journald.conf on bottlerocket, so it will create old-fashioned txt logs.

or, in other way, am i understand correctly, that there is no way to affect on how things are being logged?

[stmcginnis]

Hi @iaroslav-nakonechnikov. journald.conf is not a file you can modify with any settings at runtime. Its contents can be seen here.

According to the Splunk documentation, it should be able to read directly from the journal. There are some troubleshooting tips in their docs that might be able to help diagnose what is happening. Running ps aux | grep journalctl to see if it is actually able to stream the logs would be an interesting check.

It really would be more efficient, storage space and processing-wise, for it to read directly from the journal rather than having things written out somewhere. Perhaps Splunk support can provide some insight?

I mention this only as a possible workaround or maybe for some insights by comparing it, but there was a write up on using fluent-bit that gets systemd logs. That maybe has some more configuration options for systemd, and can forward logs to Splunk.

[yaroslav-nakonechnikov]

yes, i'm asking splunk also.

but as journald is in strong dependency with systemd - so any container should have systemd installed. (i may be wrong here). For example, alpine doesn't have any native way to work with it.

atm i'm using our internal image and with symliking it looks working. But sad story, that splunk doesn't support native support of journald and they offer wrapper script. And i'm looking more native solution, and i'm a bit upset that journald.conf can't be updated.

But you answered my question, so will mark as solution for now.

[yaroslav-nakonechnikov]

so.
splunk gets config from systems systemd.
and there are 2 things: it can be configured in containers with splunk universal forwarder
or, it is possible to add symlinks from /.bottlerocket/rootfs. Don't forget to add group with correct ID so user in container can read file.

for me it works.

[stmcginnis]

Fantastic, thanks for adding that detail @iaroslav-nakonechnikov!