Skip to content

Wrapper for Jenkins execution inside of GitLab

Problem to solve

Converting Jenkins installations into GitLab can be complex and error prone. What if instead, we just made it easy to put Jenkins in a little box and run it inside of GitLab while providing it the right inputs and gathering its outputs? It might be the best of both worlds to support an ongoing migration.

Then, we can follow up to take the longer effort of writing various translators which will have more complexity.

Intended users

Further details

Previously discussed in the meta epic &2735 (comment 295172334)

Proposal

Our initial MVC will be oriented around the simultaneous work to document @ctimberlake1's blog post (https://lackastack.net/posts/gitlabci-jenkinsfile/) that is happening in #37032 (closed). This currently is an entirely manual process, but offers some opportunities for automation to make it easier to do. As such, our plan is to start with a CLI in #208848 (closed) to achieve the first part of the configuration needed in the blog post.

This tool would:

  • Run on the customer's current Jenkins server
  • Be pointed at the Jenkins home directory and the needed jenkinsfile-runner version
  • Once triggered, would create and push a container to our registry that can be used as the runtime environment used in the remainder of the blog post.

Official documentation being delivered in #37032 (closed) would then need to be updated to ensure it has the latest details on:

  • How to create a runner capable of running the container (there are sizing concerns, and a shared runner would likely not be sufficient)
  • How to use set up your pipeline correctly to run your Jenkinsfile in the container
  • How to make use of some GitLab features, e.g. uploading artifacts created by the Jenkinsfile jobs, capturing logs, or running only in the presence of a Jenkinsfile for example.

There is already a proof of concept set up following the blog post documented at #37032 (comment 298623934) which can be referred to here.

Research notes

Additional notes from @ctimberlake1:

One of the issues with remaking the pipelines is that some of our customers have massive pipelines. One of our major and strategic partners - Who has requested this - I just learned has over 200,000 Jenkins Pipelines that are older. There's no possible way to move all of those over by hand, and even if we narrowed it down to active pipelines and critical pipelines, I suspect that number will still be out of range.

What if we look at the spirit of this request as opposed to the letter of the request? We merely want to run Jenkins pipelines in GitLab Seemlessly.

I've done a lot of thinking on this; It's something important to me, and I'd like to help however I can. Hear me out - We're thus far approaching this from the perspective of converting the pipelines. We're also lumping the pipelines and process together, or atleast I have been up until now.

Speaking on the basis of Jenkins Groovy/Imperative Pipelines only:

Consider the downsides of an interpreted conversion. We'll never be able to 100% interpret and convert an Jenkins Imperative pipelines, as it's basically an application unto itself. That means there will be hand holding no matter what. This shifts the burden of maintaining, and conversion onto the individual pipeline. So if something fails or is incompatible, It's with that individual pipeline. We'll be tracing these issues forever. Customers will be upset and we'll encounter "Well, all X of my pipelines work, but this one doesn't.", and Teams with "Well, we tried it but 2% of our pipelines didn't work, so we're staying with Jenkins."

New/Old Proposal? Linked above is an article I wrote about integrating Jenkins and running it inside of GitLab as an image. Jenkins+Jenkins Worker as one component that can be invoked by GitLab-Runner and ran. I mention numerous pitfalls and issues with it. But what if we could use that process as a jumping off point to build a compatibility layer inside of GitLab and the GitLab Runner, where we could run any/all Jenkins Pipelines inside it.

This would of course require GitLab-Runner to have enough ram and resources to spin it up. It would also require some configuration of plugins from the user. But this is something we could potentially automate. The upside to this is we would have 100% Compatibility now and into the future. We would also have a single point of failure to maintain and manage.

We ship a GitLab Docker Image w/ Jenkins installed in it. Then we pull any plugins/config necessary for Jenkins from a repo's individual .jenkins directory. The GitLab-Runner spins up the image; The image then does a git pull, moves the .jenkins directory into jenkins_home, spins up Jenkins, runs the CI, and dies.

For companies with more established versions of Jenkins or support. We include instructions on how to build their own Jenkins Docker image based off ours that doesn't require a .jenkins directory, they can just plugin all their plugins and configuration in that image and call it good.

Then, when a Repo has a .Jenkinsfile in it, we can offer the option to "Import Jenkins", which will drop a preconfigured .gitlab-ci.yml file in the repo that is setup to used Jenkins. We can even leverage the Jenkins Integrations and improve it to pass data to/from GitLab.

Permissions and Security

Documentation

Availability & Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Links / references

Sub-tasks

  • Validate that passing Directives to the jenkinsfile-runner in the docker container works. (@furkanayhan)
  • How does Bamboo importer handle declarative syntax? (@fabiopitino)
  • What are the common plugins and how do we get them running in docker? (@matteeyah)
  • Research common customer use cases. Which customer types will the proposed solution work for? (@jyavorska @thaoyeager @ctimberlake1)
Edited by 🤖 GitLab Bot 🤖