Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: improve quickstart documentation and related setup #192

Open
Meghajit opened this issue Sep 15, 2022 · 0 comments
Open

doc: improve quickstart documentation and related setup #192

Meghajit opened this issue Sep 15, 2022 · 0 comments

Comments

@Meghajit
Copy link
Member

Meghajit commented Sep 15, 2022

Summary
As of now, there is some issues in the local setup of Dagger:

  1. The local.properties assumes there is a TestMessage proto present. However, it is not the case. This causes the dagger not to start properly.
  2. The other test protos which come inbuilt with Dagger are in test module which are not usable in local.properties.
  3. The watermark delay is set to 0 in the local.properties which might cause issues in some cases, such as when events are coming in late due to issues with kafka. Also, this watermark delay needs to be much higher when Parquet daggers are being setup.
  4. With multiple ways people can create daggers like with parquet source, kafka source, with preprocessors, etc, ideally there should be specific examples on how to do each

Proposed solution

These are some suggestions based on the feedback received from internal discussions as well as from users:

  1. Creating a Dockerfile/Docker compose with all dependencies bundled( local kafka cluster, sample protos, etc). User just need to run it and check the logs for output. No manual setup required except Docker.

  2. Organizing the documentation explaining how to do different things with Dagger in a single place. For example,

  3. Create a Dagger with Kafka as a source

  4. Create a Dagger with Parquet Files as a source

  5. Create a Dagger using preprocessors

Currently, the reference is not streamlined and is organised in multiple sections.

What it will achieve

Following benefits are expected:

  1. Faster onboarding of users: People can utilise the examples to try out different features of daggers without help by the committers/maintainers
  2. Wider adoption: Faster ways to try out different features by minimal setup means people can quickly check whether the product satisfies their needs or not. Will help in a faster feedback loop.

Additional context
None

@Meghajit Meghajit linked a pull request Sep 20, 2022 that will close this issue
@Meghajit Meghajit removed a link to a pull request Sep 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant