Configuration-based use of HF hub-hosted datasets for training

[chimezie]

Per the title, allow a structured hf_dataset YAML configuration parameter for specifying an HF hub-hosted dataset (via name) to use with training and the ability to use datasets' local file system caching, named splits, named configurations (via configuration), split slicing syntax for specifying train, validation, and test datasets, etc.

The dataset feature names that correspond to attributes or keys for prompt/completion value pairs in the datasets or those with single pure text values (via text_feature in that case) can be specified.

Added YAML parameters, for example (train on the first 1K in the train split and validate with the last 100 (no test data set):

hf_dataset:
  name: "billsum"
  train_split: "train[:1000]"
  valid_split: "train[-100:]"
  prompt_feature: "text"
  completion_feature: "summary"

See: Splits and Configurations, billsum, & HF Dataset API

[chimezie]

Motivated by need to reproduce #620 with an open dataset

[awni]

Suggested change

	"Test set not found or empty. Must provide test set for ev aluation."
	"Test set not found or empty. Must provide test set for evaluation."

[awni]

You can make text_feature configurable in the regular dataset, default it to text and then just reuse that class.

Same comment for the completions dataset.

[awni]

This is very nice, opens up a lot of datasets without too much complexity. I'd like to simplify the dataset classes (see inline comment). After that, I think we should merge it.