Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TASK][MEDIUM] Make KSHC binary compatible with multiple Spark versions #6247

Closed
3 tasks done
pan3793 opened this issue Apr 3, 2024 · 4 comments
Closed
3 tasks done

[TASK][MEDIUM] Make KSHC binary compatible with multiple Spark versions #6247

pan3793 opened this issue Apr 3, 2024 · 4 comments
Assignees
@zhouyifan279

Comments

@pan3793
Copy link
Member

pan3793 commented Apr 3, 2024 •
edited

What's the level of this task?

MEDIUM

Code of Conduct

Search before creating

  • I have searched in the task list and found no similar tasks.

Mentor

  • I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide contributors through the task.

Skill requirements

  • Knowledge of GitHub Actions, Java Reflections, Hive, Spark

Background and Goals

Currently, the KSHC(Kyuubi Spark Hive Connector) supports Spark 3.3, 3.4, and 3.5, but unlike the Spark engine, the KSHC jar built with Spark 3.5 can not work on Spark 3.4 runtime.

Implementation steps

  • Tune the KSHC code to fix binary-compatible issues
  • Tune the test code and set up the GitHub Actions workflow to do cross-version verification

Additional context

Introduction of 2024H1 Kyuubi Code Contribution Program
@beryllw
Copy link
Contributor

beryllw commented Apr 3, 2024

Looks like what I worked for before #5384 , please assign to me if possible.

@pan3793
Copy link
Member Author

pan3793 commented Apr 3, 2024

@beryllw thank you

@SteNicholas
Copy link
Member

@pan3793, any update? I would like to take this ticket.

@beryllw beryllw removed their assignment Apr 30, 2024
zhouyifan279 added a commit to zhouyifan279/kyuubi that referenced this issue May 31, 2024
@zhouyifan279
[KYUUBI apache#6247] KSHC cross-version test
13f528d
@zhouyifan279 zhouyifan279 mentioned this issue May 31, 2024
Closed
4 tasks
zhouyifan279 added a commit to zhouyifan279/kyuubi that referenced this issue May 31, 2024
@zhouyifan279
[KYUUBI apache#6247] KSHC cross-version test
17cb671
@zhouyifan279
Copy link
Contributor

@pan3793 @SteNicholas I created a PR #6436 to do KSHC cross-version test. I'd like to take this ticket if it's OK.

@pan3793 pan3793 assigned zhouyifan279 and unassigned SteNicholas May 31, 2024
@pan3793 pan3793 closed this as completed in 3ed912f Jun 1, 2024
pan3793 pushed a commit that referenced this issue Jun 1, 2024
@zhouyifan279 @pan3793
[KYUUBI #6247] Make KSHC binary compatible with multiple Spark versions
126ff4e 
# 🔍 Description
## Issue References 🔗

This pull request closes #6247

This also closes #6431

## Describe Your Solution 🔧
Add a job `spark-connector-cross-version-test` in GitHub Actions to:
1. Build KSHC package with maven opt `-Pspark-3.5`
2. Run KSHC tests with maven opt `-Pspark-3.3` and `-Pspark-3.4` and KSHC package built in step 1
3. Fix the binary-compatible issue via reflection.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6436 from zhouyifan279/kshc-cross-version-test.

Closes #6247

d3ac2ef [zhouyifan279] Tune the KSHC code to fix binary-compatible issues
4e14edc [zhouyifan279] Fix invalid unit-tests-log name
56ca45d [zhouyifan279] Fix invalid unit-tests-log name
4c5ab7b [zhouyifan279] Update test log name
8a84e88 [zhouyifan279] Add matrix scala
17cb671 [zhouyifan279] [KYUUBI #6247] KSHC cross-version test

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 3ed912f)
Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhouyifan279 zhouyifan279

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[KYUUBI #6247] Make KSHC binary compatible with multiple Spark versions zhouyifan279/kyuubi
4 participants
@SteNicholas @beryllw @pan3793 @zhouyifan279