Summary

Add a configurable retention (TTL / auto-expiry) policy to the Harness IDP custom integration, so entries ingested through it are removed automatically after a set period.

Customer context

A customer recently went live with a Jenkins custom integration. Their only feedback after go-live was that the custom integration has no built-in way to expire old data.

Problem

Entries pushed through the custom integration accumulate indefinitely. To keep the data clean, the customer built and now maintains a separate scheduled pipeline whose only job is to delete old entries.

Current workaround

A standalone pipeline scheduler that periodically finds and deletes stale entries. This works, but it adds:

an extra pipeline to build, run, and monitor

custom deletion logic that has to be maintained

a risk of stale data lingering if the cleanup job fails or is misconfigured

Requested behavior

A retention setting on the custom integration (e.g. “delete entries older than N days”), configurable per integration.

Age is measured from the entry’s last update or ingestion timestamp (the option to choose which would be ideal).

Expired entries are removed automatically, with no external pipeline.

Default is “no retention” so existing integrations are unaffected.

Nice to have

Different retention periods per entry type or per integration.

An audit log or event when entries are purged.

A dry-run or preview of what would be deleted.

Business impact

Removes the need for a separate cleanup pipeline.

Keeps IDP data accurate, since stale CI/build data won’t show up in the catalog or scorecards.

Lowers operational overhead for any customer using custom integrations.