Does qdrant support ttl indexes

[xtyDoge]

I need store my vector data in recent 2 weeks, so I must use a schedule job to delete expired documents. Can qdrant support ttl index like mongodb to expired these documents?

[timvisee]

Can qdrant support ttl index like mongodb to expired these documents?

No, this is not supported at this time.

We'd recommend to tag each point with datetime payload. You can create a datetime index on them to efficiently search over them. While searching, simply provide a datetime range filter. For deleting old data you can do the same with a delete filter in a cron job.

We have been exploring the ability for setting TTLs on points, but didn't have a strong use case for it while alternatives are available.

[shaneutt]

We're also hoping for expiry and a TTL, more specifically: set a TTL which gets automatically bumped on access, and automatically reaped when time is up. This is especially helpful in the use case where you're using qdrant for semantic caching.

[generall]

For TTL at large scale we recommend to use https://qdrant.tech/documentation/guides/distributed_deployment/#user-defined-sharding

[shaneutt]

Thank you, but we'd be looking for TTL at small scales [as well].

[timvisee]

While we don't have this yet, you might be able to achieve something similar with a datetime index.

It would roughly work like this:

• create a datetime index index, probably on a ttl field that would store the point in time at which it would expire
• when inserting points, attach this ttl
• when updating existing points, simply attach the new ttl (to overwrite the existing one)
• when searching/scrolling, use a date time filter on the ttl <= now()
• have a periodic task that does delete points by filter, wiping all expired entries

The date time index has microsecond resolution. You can also use a simple integer with an integer index for unix timestamps, though then you'd likely prefer a parameterized integer index with just the range capability. It should not make a big difference though as the underlying index is the same.

One problem here is that delete points by filter is not super efficient. Whether this is measurable depends on what scale you plan to use this at.

Edit: I now see I've effectively answered the same above. My bad! Since this is a bit more elaborate, I'll leave this here. 😄