feat(pyspark): provide a mode option to manage both batch and streaming connections

[chloeh13q]

Description of changes

Provide a mode option to manage both batch and streaming connections

[chloeh13q]

Along the lines of #8584, it's worth pointing out the fact that pyspark in streaming mode requires all read and write paths to be directories rather than files. I think Impala has something similar

[chloeh13q]

xref: ibis-project/testing-data#9

[jcrist]

I think a mode="streaming"/mode="batch" kwarg to connect makes sense. However, is it possible to implement this while keeping everything within a single Backend class?

• Our current magical ibis.<backend>.connect method isn't really setup to handle things the way you're doing here (as you've found by having to reimplement it).
• It looks like a lot of the methods in the streaming backend are pretty similar to the batch implementations - uniting them and branching when necessary feels doable. If necessary, we can always create a small abstraction layer internal to the Backend class if needed.
• For the new read_kafka/to_kafka methods, raising NotImplementedError (or something) when in batch mode seems fine.

We can always split the implementations later on if it grows unmaintainable, but for now it looks to me to be simpler to keep everything within one class.

[chloeh13q]

We don't technically have read_parquet_directory implemented yet but I figured that it may be okay because it's going in in a separate PR? But I can delete this if needed.

[jcrist]

One small fixup, otherwise this looks good to me!

[jcrist]