Reapply "[#30789] Add support for Flink 1.18 (#31062)"

[je-ik]

Reintroduce Flink 1.18 support.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[shunping]

R: @Abacn

[je-ik]

I don't think we should merge this as is. It would likely cause the same issues, because the underlying issue is not resolved.

[je-ik]

I was able to generate the pom.xml that is used in the test. The following command generates it into word-count-beam:

mvn archetype:generate \
  --update-snapshots \
  -DarchetypeGroupId=org.apache.beam -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
  -DarchetypeVersion=2.57.0-SNAPSHOT \
  -DgroupId=org.example \
  -DartifactId=word-count-beam \
  -Dversion="0.1" \
  -Dpackage=org.apache.beam.examples \
  -DinteractiveMode=false

It needs to have apache snapshots repository configured in ~/.m2/settings.xml. Now the issue is that in the current version, the generated pom references beam-runners-flink-1.17, while the gradle task runs on 1.18. Because the archetype is updated asynchronously from the test, we should probably wait before the updated archetype is deployed?

There also seems to be some discrepancy in the release process of the archetype. Running the above command with -DarchetypeVersion set to 2.56.0 generates pom.xml with <beam.version>2.55.0</beam.version> which is likely a bug.

[je-ik]

The problem seems to be related to classloading. When running the test using mvn exec:java I can see the code (sometimes) tries to load flink-related classes using AppClassLoader (which contains only apache-maven-3.8.4/boot/plexus-classworlds-2.6.0.jar). Maven exec plugin creates URLClassLoader, which contains all the required dependencies, but that is sometimes not used. I can see it is used for loading some classes. This is very likely Flink bug (1.18).

I'll see if I can change the test to run using mvn exec:exec, with correct classpath.

[je-ik]

@Abacn this fixes the issue with the test.