GDK on Remote Development/Workspaces

The purpose of this is to document my journey getting GDK running in a remote-development workspace. From here I'll create other issues (and MRs) to make it work better. This can also serve as a starting point for a blog post.

The Short Story

I have a fork of gdk with a .devfile.yaml in the root, using a modified version of the gdk bootstrapped container I built for remote development (pre workspaces) I’ve had to fiddle with some things along the way, and there’s a bit of handcrafting once the workspace is up, but it’s reproducible.

The Long Story

1. Followed the prerequisite steps here
   1. Used a GKE environment, created by terraform. This allows me to 'quickly' destroy and/or recreate the cluster.
   2. Terraform was lifted out of this tutorial with some small modifications:
      1. Created the cluster in a zone rather than a whole region. This keeps it to a single initial node.
      2. Enabled autoscaling from 1 to 3 nodes.
   3. Installed ingress-nginx
   4. Created records in a personal dns zone.
   5. Created a set of certs (using certbot and allowing for dns challenges)
   6. Created the oauth application in gitlab
   7. Installed the gitlab-workspaces-proxy
2. I decided to do much of this in a personal group (gitlab.com/srza) just to be able to get the 'full' experience.
3. I created a private project in my group for the gitlab agent, and created a blank config.yaml
4. Installed the gitlab agent for Kubernetes into my cluster
5. Reconfigured the agent for remote-development
6. For a 'public project' I initially simply forked this demo into the same group.
7. I initially encountered an issue which needed me to recreate my agent - I possibly could have simply bounced the pod. Remote Development Setup: Workspaces not creati... (gitlab-org/gitlab#413971 - closed)
8. At this point I was able to spawn a workspace for the example project forked above.
9. I then created a fork in the same group of gitlab-org/gitlab.
10. Created a devfile (with appropriate limits). I used a container from the work I did several weeks ago, with a small modification here. I also annotated the ports I thought I might need access to. (This allows the proxy to speak to them).
11. After creating a workspace I noticed the workspace creation was failing. Allow configuration of project cloning approach (gitlab-org/gitlab#414011)
12. I switched up to a fresh fork of the GDK project: https://gitlab.com/srza/gdk-rdev
13. Adding devfile to the GDK fork enabled me to spawn a workspace. However, I was unable to do much inside the workspace, as asdf and the bootstrapping was all done as the user remote and our running workspace has a different uid.
14. The next step was to update my container following the guidelines from the workspaces docs: https://gitlab.com/gitlab-org/community-relations/contributor-success/gitlab-rd-web-ide-docker/-/commit/3d2ca498dff39281e41be80ae48f29a09daf80db
15. This still didn't work as asdf by default expects to find itself and other files in a .asdf directory inside your $HOME. After a bit of searching, I found you could tell asdf to install in a different place. https://gitlab.com/gitlab-org/community-relations/contributor-success/gitlab-rd-web-ide-docker/-/commit/10476edd9fabf1385b821dc15d492035af8e36a7
16. At this point, I could create a workspace with GDK bootstrapped, and the asdf and gdk commands were being picked up from the path (after running exec bash in the terminal)
17. gdk, however, was still not happy with an 'unknown' user. But here (for now) we hardcode the uid inside the container to 5001. - So I updated my container to use a uid of 5001: https://gitlab.com/gitlab-org/community-relations/contributor-success/gitlab-rd-web-ide-docker/-/commit/3c5b72facbac9c6209997290b22c5df3d7c204ad (Also updated my devfile as this created a new tag)
18. Success! I was able to run gdk in a new workspace. I proceeded with running gdk install
19. GDK installed and could run in the background, but I still had no access to it through the proxy, as it only listens on a loopback interface by default.
20. Several iterations of a gdk.yml later and some reconfigures/restarts - I was able to access the 'gdk is booting' screen through the proxy.
21. At this point I noticed that rails seemed to be crashing regularly after about 45 seconds. Looking at the logs, it seemed that fs.inotify.max_user_watches was set to a low number (~12500) gitlab-org/gitlab-development-kit#307 (closed)
22. As you're running inside a cluster, you're not able to elevate your privileges to set kernel parameters. I built a DaemonSet to install on my cluster that runs with elevated privileges and sets the kernel parameter and effectively resets it regularly. (A Kubernetes DaemonSet effectively runs a copy of the pod on every node in the cluster. If we later add more nodes, it'll 'magically' run there too.)
23. After confirming that the kernel parameter was now set appropriately, I (gracefully) restarted gdk and violá!

Actions

This is an unsorted braindump and starting point to create issues and/or MRs in the right places

• Workspace needs to inject URL into ENV: Workspaces: Inject domain template as an enviro... (gitlab-org/gitlab#415328 - closed)
• Make something that can generate the required gdk.yml from information available
• Run certain commands on workspace spinup (gdk install?)
• Fix/clean up the container image and build it regularly
• Make the Daemonset more robust and documented
• Add documentation into the GDK to help others along this path
• Anything else?