There are a couple other features along the same lines:
But I like to suggest a different method that may be easier to implement.
We currently have steps that generate logs > 5000 lines (because reasons) thus exceeding the logs available to view in the streaming console, but also when the logs are downloaded they are truncated to the last 5000ish lines.
We need the full logs for audit purposes and also for debugging and troubleshooting.
Ideally we'd like to stream the execution logs to Splunk in real time, with the ability to decorate the lines with (just off the top of my head):
- pipeline name
- execution id
- stage name
- step name
- delegate hostname
But I assume supporting many 3rd party logging endpoints would be a fait bit of work and why the current feature requests have been marked long term. I'd like to suggest the ability to log the execution logs locally to the delegate (along with keeping the current streaming back to harness), that way we can use a log collector like otel or fluentbit to send the logs to our 3rd party of choice.