We have configured the Audit Log Streaming service to push audit logs to AWS S3 and we're attempting to ingest those logs into our Elastic service. The issue is that the Harness batch sends to the logs to S3 every 30 minutes as a single file, which includes a single json object for that time period's logs.

Log collectors can't really work with this type of file. We expect each event to be its own line for JSON so that log collectors can read events one at a time. While we can read bulk events from a JSON list, the file still needs to end with a new line to indicate the process finished writing. Since many collectors checksum to first line of a file to ensure they catch changes when restarting, the json files are unhashable. The log collector will wait for a newline. Adding a newline at the end of the file is enough to unblock us, but there is risk of losing multiple events if any part of the file is corrupted. We prefer one event per line as other vendors do.