feature: Allow merging/extending select arrays in environments

[melgazar9]

Feature scope

API

Description

I would like meltano.yml to be able to select all with exception to one or two . For example, I have a tap-mongodb tap that I am running, which selects 80+ documents in the parent configuration and is set up to run for dev, testing, staging, and production with the same meltano.yml file only changing the credentials. The problem I'm facing is that one collection in testing environment has inconsistent datatypes and breaks the meltano pipeline. Since this is not expected and doesn't occur in staging & production I would like to remove this collection from select in meltano.yml rather than list out 79 collections I want to select.

version: 1
send_anonymous_usage_stats: true
project_id: tap-mongodb

default_environment: dev

state_backend:
  type: remote
  uri: ${AZURE_TAP_MONGODB_STATE_URI}
  azure:
    connection_string: ${AZURE_TAP_MONGODB_STATE_CONNECTION_STRING}

plugins:
  extractors:
  - name: tap-mongodb
    namespace: tap_mongodb
    pip_url: git+https://github.com/melgazar9/tap-mongodb.git@20738c1272ff12eb403abb9f8019200e5acd573f
    capabilities:
      - state
      - catalog
      - discover
      - about
      - stream-maps

    config:
      add_record_metadata: true
      allow_modify_change_streams: true

    select:
      - 'collection1.*'
      - 'collection2.*'
      - 'collection3.*'

environments:
  - name: dev
    config:
      plugins:
        extractors:
          - name: tap-mongodb
            config:
              mongodb_connection_string: ${DEV_MONGODB_CONNECTION_STRING}
              database: Dev
        loaders:
          - name: target-snowflake
            env:
              TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA: MONGODB_DEV

  - name: testing
    config:
      plugins:
        extractors:
          - name: tap-mongodb
            config:
              mongodb_connection_string: ${TESTING_MONGODB_CONNECTION_STRING}
              database: Testing
            select:
              - '!collection3.*'  # bug in mongodb data - I want to disregard collection3 when environment = 'testing'
        loaders:
          - name: target-snowflake
            env:
              TARGET_SNOWFLAKE_DEFAULT_TARGET_SCHEMA: MONGODB_TESTING         

[tayloramurphy]

@melgazar9 agreed this would be useful. I don't think it'd be too hard to add, but we're not going to prioritize it at this time. cc @edgarrmondragon