Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add message key and headers to the Cypher ingestion strategy #513

Open
conker84 opened this issue Nov 24, 2021 · 1 comment
Open

Add message key and headers to the Cypher ingestion strategy #513

conker84 opened this issue Nov 24, 2021 · 1 comment

Comments

@conker84
Copy link
Contributor

Several customers asked about having access to the Kafka message key and headers in the Cypher query Strategy.
In order to preserve backwards compatibility, we leave $event as-is and we'll inject two more parameters $eventKey and $eventHeaders

@moxious
Copy link
Contributor

moxious commented Dec 10, 2021

@conker84 the general principle of "exposing to the cypher layer anything that the kafka client can deliver" is probably a good one. I can imagine valid use cases why a user would want this information, and Kafka is delivering it so why shouldn't they have it.

However there are probably going to be multiple different types of data about this. What I might suggest instead of $eventKey and $eventHeader is to inject a

:param meta => {
    key: 'abc',
    headers: { 'foo': 'bar' }
}

Reason here is that $event is a very clean simple interface, but we don't want to spread the interface and add lots more things. It might be better to group them in one place for extensibility. In future or older versions of kafka what fields can be delivered and how they're named could change, so you don't want to couple our interface ($eventHeader cypher param) to a potentially changing kafka interface; rather give them $meta and then deliver with that just whatever extra bits of information about the event that Kafka will permit your client layer to inspect. In doing it this way, you can in our docs refer back to the kafka docs about "meta" information and just say "Hey, we're going to copy whatever is there with the same naming" and have a generally flexible interface that we're not responsible for maintaining, rather we just copy whatever the kafka client convention is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants