Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accept pipeline input for stor insert values #11433

Open
NotTheDr01ds opened this issue Dec 28, 2023 · 7 comments · May be fixed by #12859 or #12882
Open

Accept pipeline input for stor insert values #11433

NotTheDr01ds opened this issue Dec 28, 2023 · 7 comments · May be fixed by #12859 or #12882
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@NotTheDr01ds
Copy link
Contributor

NotTheDr01ds commented Dec 28, 2023

Related problem

While it's been established that into sqlite's ability to append to a database is more a "side-effect" than part of its design, it seems so much simpler than stor insert. For example, with a REST call that returns a JSON items with a list of records (i.e. a Nu table):

With into sqlite:

let res = (http get $endpoint)
$res.items
    | select colname1? colname2? colname5?
    | into sqlite foo.sqlite

This currently works when both creating the database as well as updating, as long as the values match their original positions.

With stor insert:

let res = (http get $endpoint)
$res.items
    | each {
        |item|
        stor insert -t main -d {
            colname1: $item.colname1,
            colname2: $item.colname2,
            colname5: $item.colname5
        }
    }

(^ from memory - Hope I got those right)

Describe the solution you'd like

Allow stor insert to accept pipeline input in the form of either a record:

{bool1: true, int1: 5, float1: 1.1, str1: fdncred, datetime1: 2023-04-17} | stor insert --table-name nudb
$table | select bool1? float1? str1? datetime1? | stor insert --table-name nudb

In the second case, I'm guessing the column names would need to be inferred from the first record, which into sqlite seems to do when creating (but not updating) the database (to support streaming). Any change in the signature of future records would result in an error.

Describe alternatives you've considered

No response

Additional context and details

Would this also work for stor update?

@NotTheDr01ds NotTheDr01ds added enhancement New feature or request needs-triage An issue that hasn't had any proper look labels Dec 28, 2023
@fdncred
Copy link
Collaborator

fdncred commented Dec 28, 2023

I'd support that. It should be pretty easy to do. You just have to check for input and use it and make sure you don't have input and --data-record parameters at the same time.

I'm all for making these stor commands as ergonomic as possible.

@fdncred fdncred removed the needs-triage An issue that hasn't had any proper look label Dec 28, 2023
@JoaoFidalgo1403
Copy link
Contributor

Hi, me and my colleague would like to know if it is still possible to work on this enhancement!

@fdncred
Copy link
Collaborator

fdncred commented Apr 25, 2024

@JoaoFidalgo1403 Sure. I was just thinking that it could work both ways. If the -d parameter is they, work the way I wrote it. If the -d parameter is missing and there is input being passed in as a record, then use that, but like I said, if you have both input and -d, that's an error case.

@fdncred fdncred added the good first issue Good for newcomers label Apr 25, 2024
@friaes
Copy link
Contributor

friaes commented Apr 25, 2024

@fdncred Also, would it be ok if we did the same work on stor update ?

@fdncred
Copy link
Collaborator

fdncred commented Apr 25, 2024

@fdncred Also, would it be ok if we did the same work on stor update ?

i think so

@bobby-palmer
Copy link

        let columns: Option<Record> = call.get_flag(engine_state, stack, "data-record")?;
        match _input {
            PipelineData::Empty => {
                 // do nothing
            },
            PipelineData::Value(v, _) => {
                if columns.is_some() {
                    //error
                }
                else {
                    columns = Some(v.into_record()?);
                }
            },
            _ => {
                   // error
            }
        }

Would something like this be on the right track? I'm new to the code base and a little unfamiliar with the code style.
@fdncred

@fdncred
Copy link
Collaborator

fdncred commented May 13, 2024

@bobby-palmer I think that's probably part of it, although input shouldn't have an underscore if it's being used. If there's no input, you should look for the data as a parameter. You might dig through the command to see which ones can work with input and parameters. I don't have any in mind right now.

@bobby-palmer bobby-palmer linked a pull request May 13, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
5 participants