Allow Renovate's azure-pipeline's manager to read azure repositories defined in resource blocks

[JackRhodes]

What would you like Renovate to be able to do?

TLDR: allow renovate to update azure repository references within azure-pipelines yaml files. It currently isn't very well documented that it doesn't support this

Renovate has support for azure-pipelines, it does this by reading the Resources block defined in an azure pipelines yaml file

From this, it is able to update:

• Containers
• Git repositories

A git repository in azure pipeline has a type depending on where the repo is hosted, docs here:

• git, (azure repository reference)
• github,
• githubenterprise,
• bitbucket`.

Within Renovate's current manager implementation , it only supports github and will return null for all other types. This is not currently documented within Renovate and it is not clear that this is the case unless you view the implementation.

I would like to propose that support for git type is added, which will enable it to work within Azure Devops. Alongside this work, I would like to propose that the documentation is updated to state which repository types renovate supports in the manager to better define what it supports.

GIVEN an azure pipeline, WHEN it has an azure repository reference in its resources THEN it should check if the repository has a newer version

Please let me know if any more detail is needed, or if I am misunderstanding anything.

If you have any ideas on how this should be implemented, please tell us here.

(I am not a typescript dev and this is my first attempt at a suggestion)

If the repository reference is of type git, and the renovate platform is set to azure and has an endpoint defined, then we can construct the repository url reference.

To construct this, the following inputs need to be handled

• myRepository which would produce: <endpoint>/currentProject/_git/myRepository
• otherProject/MyRepositoryInOtherProject which would produce <endpoint>/otherProject/_git/MyRepositoryInOtherProject

This can be added to lib/modules/manager/azure-pipelines/extract.ts

I am unsure the cleanest way to achieve this as your would be mixing the manager domain with the platform domain, but I believe the actual change should be relatively simple code change. Azure repositories can only be used within ADO so coupling the platform and manager is not as bad as it sounds.

I am happy to implement this, but would require guidance about the best way to handle this.

Is this a feature you are interested in implementing yourself?

Maybe

[rarkins]

Is it possible to create a public example on Azure DevOps?

[JackRhodes]

Yes I believe so. I'll try to setup a test ADO project with an example

[github-actions[bot]]

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[JackRhodes]

Apologies for the delay! I had to setup an ADO project which took a while

Renovate:

• Example Renovate run
• Renovate server setup

The issue:

• Affected yaml file : this has an example of a github reference (that works), and an ADO git reference (that renovate does not generate for)
• Pull Requests : it has only generated a PR for the github dependency
• Affected line in logs : as you can see, it only sees the github repo. This is because of what I've described in my original issue, where only repositories with type: github are looked at

Please let me know if anything is unclear or if you can't access any of the links!

[rarkins]

What is the expected result for the git dependency?

[JackRhodes]

In the example , renovate should be upping the version of Renovate-Playground/example-repository from 0.1.0 to 0.2.0

Currently, Renovate ignores the git dependency and only creates a PR for github https://dev.azure.com/combarcapes/Renovate-Playground/_git/issue-15028/pullrequests?_a=active

[rarkins]

So it's defined as a "git" dependency yet it's implicitly pointing to Azure DevOps Services?

[JackRhodes]

Correct

Docs here

[rarkins]

If the git reference is e.g. MyBitbucket/vscode then doesn't that mean we know the full name of the repo without needing to know anything about the current platform, endpoint or repository? i.e. it's always going to point to azure devops services?

[hypervtechnics]

as you can see in the docs it only will fetch from bitbucket if type: bitbucket. The type indicates rhe source of the repo. If type: git it will point to Azure DevOps (git sounds generic although Azure DevOps is meant)

[rarkins]

That doesn't answer my question

[hypervtechnics]

When you use type: git and name: my-repository it will point to https://dev.azure.com/my-organization/my-project/_git/my-repository. We can not infer my-project and my-organization from the azure pipelines file. That's only doable when we have the properties webUrl or remoteUrl available from the repository details.

[JackRhodes]

If the git reference is e.g. MyBitbucket/vscode then doesn't that mean we know the full name of the repo without needing to know anything about the current platform, endpoint or repository? i.e. it's always going to point to azure devops services?

So yes, git will always point to the current azure devops organization. We know the full name of the repository, and potentially the project, but we do not have any way to see the organization from just the yaml file. Azure pipelines with a git reference will implicitly use the organization url for you since they sit in the same organization so it never gets explicitly defined in a pipeline

[rarkins]

In that case you can follow the example from GitLab: https://github.com/renovatebot/renovate/blob/main/lib/modules/manager/gitlabci-include/extract.ts#L68

[hypervtechnics]

Thanks for the hint 😄 Unfortunetaly this works only in a subset of cases. We also have to know about the project name which is often omitted in the pipeline file if a repository in the same project is referenced. We would need to have the repository details available.

[JackRhodes]

@hypervtechnics For a first iteration, it could be ok to make it so we explicitly add the project name in the resource reference and not support implicit project lookups.

But yes, you are correct that for full behavior, then we would also need a way to lookup the current project that a repository is located

[hypervtechnics]

But I think we can already start discussion on handling it properly in the second iteration. Adding the project name everywhere is kind of a very huge task when you have a lot of projects and pipelines.

The only way I can think of is having the repository details (including the git clone uri) to fetch the project somewhat reliably. The endpoint url in the configuration is not suitable because it only points to the organization and not the project in the organization.

[HonkingGoose]

When you've created this feature, please edit the https://github.com/renovatebot/renovate/blob/main/lib/modules/manager/azure-pipelines/readme.md file, to remove the part of the admonition that mentions feature 15028. 😉

[sheldonhull]

UPDATE: Had to do quite a few iterations to get this right but finally got it working! I believe the issue was the filematch expression failing and a mismatch between local and remote (due to the way the docker container test runs and clones inside itself). Appreciate the hard work on this, as it's great to see version pinning working and updates coming in. Cheers

I'm a little confused on the state here. I reviewed the #20380 and indicates that as long as the Project/Repo format is followed (instead of also handling implicit project determination) then it should be merged in and working right?

I am running a test case with "renovateVersion": "35.146.1" docker version locally, and while one pipeline seems to be matched, it never takes any action on the out of date tag.
The docs also still reference this admonition. If i need to open another discussion instead I am willing, just thought this would be the appropriate follow-up place.

[ceng-p]

@sheldonhull you got it working as in the feature works for you? Could you share a working example, as I can't seem to get it working. It correctly gets the dependencies and current tag, but fails to see updates:

DEBUG: packageFiles with updates (repository=myorg/myrepo, baseBranch=main)
       "config": {
         "azure-pipelines": [
           {
             "deps": [
               {
                 "autoReplaceStringTemplate": "refs/tags/{{newValue}}",
                 "currentValue": "1.7.0",
                 "datasource": "git-tags",
                 "depName": "myproject/mytemplates",
                 "depType": "gitTags",
                 "packageName": "https://dev.azure.com/myorg/myproject/_git/myrepo",
                 "replaceString": "refs/tags/1.7.0",
                 "updates": [],
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "sourceUrl": "https://dev.azure.com/myorg/myproject/_git/myrepo",
                 "currentVersion": "1.7.0",
                 "fixedVersion": "1.7.0"
               },
               ```

running a git tag -l

...

1.7.0
1.7.1
1.7.2
1.7.3
1.7.4
1.7.5

[sheldonhull]

Not sure now, as I noticed something might have changed, as I'm not getting them anymore. If I figure it out I'll try and post up.

[sheldonhull]

Tried a few variations and even reverting to sever versioning type, no luck.
Something changed that broke the azure pipeline templates repos I had setup for version updates coming through, so I'm not getting any detected upstream tags now. @rarkins is this possible to flip back to an active issue instead of a discussion or do we need to do something else? Seems to have continued to come back up.

[rarkins]

This discussion is too messy to directly convert to an issue. Please create a minimal reproduction repo first demonstrating the problem. Also please check in case renovatebot/azure-devops-marketplace#28 is somehow related

[sheldonhull]

Regarding the PAT, I'm not certain that is related.
I'm using the agent system access token that is ephemeral.
Used to work too, but lately no azure-pipeline upgrades for the pinned template repos are being detected (as mentioned above it's all there, but the available tags aren't being found by renovate).

I am not concerned with the marketplace tasks, as much as the azure pipeline template repo being able to be bumped. I'll have to look through minimal reproduction directions and see if it's possible to get some example up and running due to the additional complexity of reproducing with azure devops. Thanks for the follow-up!

[sheldonhull]

@ceng-p so I just looked and it did create a pull request for me.
here's what I think I did that impacted and might be relevant for you.

I think I mixed up the settings so that my attempt to only bump the major version of the azure pipeline tasks in error was blocking version updates on the minor tag version updates for the templates. Even after all these years using renovate, it's easy to get snagged on so many options!

So make sure something like this what your package rules and settings look like and maybe it will work then?

I mapped in my custom/non-standard matching paths and then the settings below.

"azure-pipelines": {
    "enabled": true,
    "fileMatch": [
      "azure.*pipelines?.*\\.ya?ml$",
      "build/.*\\.ya?ml$",
      "pipelines/.*\\.ya?ml$",
      "ci/.*\\.ya?ml$",
      ".ci/.*\\.ya?ml$",
      ".azure-devops/.*\\.ya?ml$"
    ]
},
"packageRules": [
 {
      "description": "Azure pipeline templates",
      "matchManagers": ["azure-pipelines"],
      "groupName": "🚀 azure-pipelines-templates",
      "commitMessageTopic": "🚀 azure-pipeline-templates",
      "automerge": true,
      "prPriority": 99
    },
    {
      "description": "Only evaluate the major version updates for azure pipeline tasks. This reduces risk of issues with mismatch between upstream registry and azure devops setup in availability, and reduces noise.",
      "matchDatasources": ["azure-pipelines-tasks"],
      "extractVersion": "^(?<version>\\d+)",
      "groupName": "🚀 azure-pipelines-tasks",
      "commitMessageTopic": "🚀 azure-pipeline-tasks",
      "automerge": true,
      "prPriority": -1
    }
]