Stack 2023 - Going cheaper with self-hosted services fail

NOTE:
I went thru quite a wild journey here and am publishing this post months after the struggle I have had with the whole situation.
Maybe some of you are smart enough to learn just from reading about it. I definitely had to experience it on my own just to end up somewhere else.  

#the idea

Having a balanced shared environment with the idea of containers in k8s that won't make your system "dirty" with the time of playing around, it seemed like a good idea to containerize everything from source code management to deployer, CI builds, automatic hostings and push to deploy automation based on the branches, etc.

#definition of a plan

• an instance of Gitlab for source code for all projects
• reusable CI builds for all the projects
• automatic builds and deploys on git push
• all other actions should be project / k8s namespaced
• per project (k8s namespace) Gitlab workers
• per project (namespace) docker registry
• per project "production" and "development" hosting namespaces
• production deploy for pre-defined domain on manual request or based on some (release/master) branch
• automatic hostings (dev subdomains) and deploys for every feature branch or set of branches based on some rule

#expectations

with time it should be:

• easy to start new projects
• easy to spawn project hostings
• easy to (automatically available) spawn development/staging hostings
• easy to monitor and restrict resources per project
• easy to scale/move a project to a cluster with better hardware when needed

#why didn't I use existing solutions

Any popular service gets hacked sooner or later, and with the latest service I was looking into just announcing a breach, I didn't want to expose another public thing that is not under my control.

I am no expert on security, don't understand me wrong. I am "just" a developer and learned things with time and fails that production experience brings.

And I am aware I can not do things better, but in reality, the "obscurity over security" still wins in the end, at least until your service becomes popular.

#reality

Oh boy, was I wrong on everything ...

#installing Gitlab (security nightmare)

Not knowing enough about Kubernetes and self-hosted GitLab at that point, I followed one of the really old tutorials for days to come. The biggest pain was that Gitlab persisted data only till the first restart (volume issues). Finally dropping out of the idea of doing it manually, I used Helm and realized I wasnt even close to the end solution. It worked, but the risk of losing all the project source code just because I would make another mistake in the future was simply too big.

The next plan was to do it on a dedicated server and go thru all the configs and possible risks. Again, it worked as intended but did not solve any of the performance issues I planned to solve. Bottleneck it with workers was not an acceptable solution.

Starting with workers sounds all legit in idea, but in reality, having a dynamically scaled set of workers becomes its own risk. Having them in the cluster with a bit too much access level opened countless new blog posts I had to read on security in k8s.

...

#conclusion

I failed successfully big time in so many ways on so many levels!

Why successfully?

• I believe the idea for that kind of tool is the right one
• I learned a lot about Gitlab on-premise and got even more respect for the sys guys
• learned something new about workers, security issues, and limitations
• deeper dive into k8s security, bottlenecks of GitOps, single source of truth, and the risk of putting all into a single point of failure system design
• understanding why there are not a lot of solutions like that and why only big companies who are host providers and enterprise ones offer solutions on a level like that

What to do next?

The whole idea sounded like a great SaaS to make, just to realize weeks later how big of a project and market you should have for it to support the paychecks of involved people. The risk of responsibility simply wasnt worth it to continue.

I am so glad I went thru that whole story from a "simple idea" to reality, back to the idea of creating a SaaS and dropping out of it before even more time and money would be involved. A lifecycle of a project idea was done in weeks instead of months like it usually happened. I call that a total victory!

Till next time, stay sexy and hydrated.