```
NOTE:
I went thru quite a wild journey here and am publishing this post months after the struggle I have had with the whole situation.
Maybe some of you are smart enough to learn just from reading about it. I definitely had to experience it on my own just to end up somewhere else.  
```

# the idea

Having a balanced shared environment with the idea of containers in k8s that won't make your system "dirty" with the time of playing around, it seemed like a good idea to containerize everything from source code management to deployer, CI builds, automatic hostings and push to deploy automation based on the branches, etc.

# definition of a plan
- an instance of Gitlab for source code for all projects
- reusable CI builds for all the projects
- automatic builds and deploys on git push
- all other actions should be project / k8s namespaced
- per project (k8s namespace) Gitlab workers
- per project (namespace) docker registry
- per project "production" and "development" hosting namespaces
- production deploy for pre-defined domain on manual request or based on some (release/master) branch
- automatic hostings (dev subdomains) and deploys for every feature branch or set of branches based on some rule

# expectations
with time it should be:
- easy to start new projects
- easy to spawn project hostings
- easy to (automatically available) spawn development/staging hostings
- easy to monitor and restrict resources per project
- easy to scale/move a project to a cluster with better hardware when needed

# why didn't I use existing solutions
Any popular service gets hacked sooner or later, and with the latest service I was looking into just announcing a breach, I didn't want to expose another public thing that is not under my control.

I am no expert on security, don't understand me wrong. I am "just" a developer and learned things with time and fails that production experience brings.

And I am aware I can not do things better, but in reality, the "obscurity over security" still wins in the end, at least until your service becomes popular.

# reality
Oh boy, was I wrong on everything ...

## installing Gitlab (security nightmare)

Not knowing enough about Kubernetes and self-hosted GitLab at that point, I followed one of the really old tutorials for days to come. The biggest pain was that Gitlab persisted data only till the first restart (volume issues). Finally dropping out of the idea of doing it manually, I used Helm and realized I wasnt even close to the end solution. It worked, but the risk of losing all the project source code just because I would make another mistake in the future was simply too big.

The next plan was to do it on a dedicated server and go thru all the configs and possible risks. Again, it worked as intended but did not solve any of the performance issues I planned to solve. Bottleneck it with workers was not an acceptable solution.

Starting with workers sounds all legit in idea, but in reality, having a dynamically scaled set of workers becomes its own risk. Having them in the cluster with a bit too much access level opened countless new blog posts I had to read on security in k8s.

...

# conclusion

I failed successfully big time in so many ways on so many levels!

Why successfully?
- I believe the idea for that kind of tool is the right one
- I learned a lot about Gitlab on-premise and got even more respect for the sys guys
- learned something new about workers, security issues, and limitations
- deeper dive into k8s security, bottlenecks of GitOps, single source of truth, and the risk of putting all into a single point of failure system design
- understanding why there are not a lot of solutions like that and why only big companies who are host providers and enterprise ones offer solutions on a level like that

What to do next?

The whole idea sounded like a great SaaS to make, just to realize weeks later how big of a project and market you should have for it to support the paychecks of involved people. The risk of responsibility simply wasnt worth it to continue.

I am so glad I went thru that whole story from a "simple idea" to reality, back to the idea of creating a SaaS and dropping out of it before even more time and money would be involved. A lifecycle of a project idea was done in weeks instead of months like it usually happened. I call that a total victory!

Till next time, stay sexy and hydrated.


Stack 2023 - going cheaper fail

Stack 2023 - Going cheaper with self-hosted services fail


# the idea
As described in the previous post of this series, I want to replace the old days' deployment flows and make hosting more scalable and usable for multiple functionalities without the risk of making the system dirty with unwanted services when playing around with new technologies.

A friend of mine ( [Omer's blog](https://omeratagun.net/) ) suggested using [Rancher](https://www.rancher.com/) as the next thing. He also prepared [a post](https://omeratagun.net/blog/how-to-install-rancher) for the topic of our conversation, which triggered me enough to replace everything I was using.

# trying out a new server provider
My last reliable providers were [Linode](https://www.linode.com/) and [DigitalOcean](https://www.digitalocean.com/). I wanted to go cheaper for the new playing around and keep them for existing production stuff. I had some production experience with [Hetzner](https://www.hetzner.com/) in the past. To round the experience, it all works as expected till it doesn't. And at those times, cheaper solutions always left me in deep shit with clients.

This time I am going to try [Contabo](https://contabo.com).

# installation of k8s (Kubernetes) and Rancher

```
Note: You may wanto to jump to #issues first, as there are many tips and tricks related to the installation.

!!! WARNING !!!

The content of this post was tested and used in Aug 2022, and if it's months/years from now, follow these instructions with caution so you don't repeat my mistake!

!!! WARNING !!!
```

I went with the idea of HA (high availability) cluster with an embedded database. Not recommended, but it's a risk I am willing to take or solve in the future.

[This link](https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli#install-rancher-with-helm) provides most of the info, but I will pin all the other links as we go.

As versions of documentation change with new releases, I am also pasting a Rancher v2.6 link [here](https://ranchermanager.docs.rancher.com/v2.6/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli).

## operating system
I just used the latest Ubuntu LTS.

## installing Kubernetes cluster using k3s (Lightweight Kubernetes)
Following instructions [here](https://docs.k3s.io/installation/ha-embedded). It should work out of the box on the first run.

These are my notes:
```
NOTE: replace the version in commands
```

Install and init ha cluster primary server with an embedded database.
```
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.21.7+k3s1" sh -s - server --cluster-init
```
Alternative: Install and init ha cluster primary server with the external database.
```
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.21.7+k3s1" sh -s - server --cluster-init \
  --datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name"
```

Add more servers to the primary server:
- choose a good name for "K3S_NODE_NAME" as you can not change it later (without a hustle)
- for renaming the primary server, I had to remove and re-add it to the cluster with a new name, which worked on the first try
```
curl -sfL https://get.k3s.io | K3S_NODE_NAME="{server-1}" INSTALL_K3S_VERSION="v1.21.7+k3s1" K3S_URL=https://{REPLACE_WITH_PRIMARY_SERVER_IP}:6443 K3S_TOKEN={REPLACE_WITH_K3S_TOKEN} sh -s - server
```
For the HA cluster, it is suggested to have (at least) 3 servers.

Add workers with:
- choose a good name for "K3S_NODE_NAME" as you can not change it later (without a hustle)
```
curl -sfL https://get.k3s.io | K3S_NODE_NAME="worker-1" INSTALL_K3S_VERSION="v1.21.7+k3s1" K3S_URL=https://{REPLACE_WITH_PRIMARY_SERVER_IP}:6443 K3S_TOKEN={REPLACE_WITH_K3S_TOKEN} sh -
```

You can add cluster access to your local machine by running this from the local machine terminal.
```
scp root@{REPLACE_ME_WITH_SERVER_IP}:/etc/rancher/k3s/k3s.yaml ~/.kube/config
```

To get the server token, run this on the server.
```
cat /var/lib/rancher/k3s/server/node-token
```

## install Rancher using helm charts
The latest guide is (should be) available [here](https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli) (v2.6 [here](https://ranchermanager.docs.rancher.com/v2.6/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli)).

```
!!! WARNING !!!

The guide uses the LATEST version, not the STABLE one. Change links accordingly.

!!! WARNING !!!
```

My notes, using cert-manager 1.9.1 and the STABLE version of Rancher:
```
helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

kubectl create namespace cattle-system

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.crds.yaml

helm repo add jetstack https://charts.jetstack.io

helm repo update

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.9.1 \
  --set installCRDs=true

helm upgrade --install rancher-stable/Rancher
--namespace cattle-system
--create-namespace
--set hostname={REPLACE_ME_WITH_RANCHER_DOMAIN}
--set bootstrapPassword={REPLACE_ME_WITH_SOME_BOOT_PASSWORD}
--set ingress.tls.source=letsEncrypt
--set letsEncrypt.email={REPLACE_ME_WITH_EMAIL}
```

If you have correctly directed domain DNS to the rancher server (also check CloudFlare issue fix), you should be able to visit the login page of Rancher and use the boot password to set up login credentials.

## setting up certificates manager
The next thing was to make domains use genuine certificates. I had a lot of issues setting this up, as every version had a few different instructions, and tutorials always missed a thing or two. I have been stuck here for days!

```
!!! WARNING !!!

You need to set up `cluster-issuer`, not `issuer', as you are using HA cluster, not a single server.
```

```
!!! WARNING !!!

Tutorials all showed examples for `nginx`, but I had to use `traefik` for the `solver` property. This took me days to find out.
```

I ended up changing/setting up 2 credential issuers.
```
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates and issues related to your account.
    email: {info@example.com}
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-staging-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: traefik
```

```
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
  namespace: cert-manager
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates and issues related to your account.
    email: {info@example.com}
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-production-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: traefik
```

```
!!! WARNING !!!

If you are behind CloudFlare, you need to allow HTTP (not secure) requests (disable FORCE HTTPS), etc.; otherwise, the challenge won't complete!
```

You should probably go with a different challenge than http01, like DNS or something similar.

Now you can request a new certificate by creating a new "certificate" under the certificate manager, "Certificates." The certificate will see if a secret key exists for it; otherwise, it triggers certificate requests, challenges, etc ... I had many issues here, so welcome to ping me for help.

Certificate for rancher domain:
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dashboard-example-com-tls
  namespace: Rancher
spec:
  # choose your own
  secretName: dashboard-example-com-tls
  dnsNames:
  - dashboard.example.com
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
    group: cert-manager.io
```

Certificate for an application that lives in namespace 'test-application.'
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  # I use the same names for a secret key, but that is not needed
  name: test-application-com-tls
  namespace: test-application
spec:
  secretName: test-application-com-tls
  dnsNames:
  - test.application.com
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
    group: cert-manager.io
```

## setting up disk management with longhorn

I followed the install thru the "Apps" navigation menu. Nothing special is needed there, but things won't work out of the box. Some requirements could be more noticeable when doing the installation. I had to read longhorn documentation line by line to find those fu\*\*\*rs.

This needs to be run on every cluster node/server.

Check requirements with script their script (change the version) and install dependencies till it shows green for all checks:
```
curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.3.1/scripts/environment_check.sh | bash
```

Dependencies (I had to install):
```
apt-get install jq
apt-get install open-iscsi
apt-get install nfs-common
```

## testing
The idea is simple, but I won't lie, have good luck trying it out, as anything can go wrong, and nothing will work.

Idea:
- create application namespace
  - !!! everything needs to be part of that namespace from now on
- create a new deployment configuration
  - docker image source
  - disk requests
  - load balancer
  - ...
- add a certificate for the domain
- add ingress rule
  - add domain
  - select certificate
  - connect to the load balancer port

What I have done in the beginning:
- created my own docker image FROM nginx, replacing just index.html hello content
- uploaded it to the docker hub
- added docker hub credentials to Rancher (application-specific namespace)
- used some subdomain where the dashboard is (testing.example.com)
- used letsencrypt staging first
- visited ports directly
- ...


# issues
Following different tutorials, the whole k8s and Rancher progress was a mess, and it required lots of nerves and enthusiasm from me to react to the goal I wanted. Being just a month late on the tutorial, nothing worked. The whole thing reminded me of the JavaScript npm world. There was even a point where I followed the Gitlab installation tutorial dated just a couple of months old, to realize at the end it was just a copy of a years-old tutorial published elsewhere. Just FML at so many points, I was sure I was having a nightmare.

That's why I will pinpoint some issues and solutions, paste some links I followed, etc.


## issue: tutorials and docs use "latest," not "stable"
Be careful, as most documentation refers to the latest releases rather than stable ones. Expect lots of issues there and always check the version and read the commands you are going to run carefully.

## issue: servers from Contabo can't see each other
Before installing a k8s cluster, some things must be solved first.

I suggest you do a sanity check and try pinging servers from one another to avoid problems when installing a k8s cluster.

The Contabo issue may be present only for some servers (I did not investigate the problem), but for the ping to work between them, I had to follow the instructions [here](https://contabo.com/blog/creating-static-routes/).

I would sum it up with the following:
- you have server A with IP (1.2.3.101)
- you have server B with IP (1.2.3.102)
- you have server C with IP (1.2.3.103)
- public route within is 1.2.3.4

Go to each server and check if they can see each other with ping. If you get a response like the one below, you need mentioned changes.

```
root@servername: ping 1.2.3.102
PING 1.2.3.102 (1.2.3.102) 56(84) bytes of data.
From {1.2.3.4} icmp_seq=1 Destination Host Unreachable
From {1.2.3.4} icmp_seq=2 Destination Host Unreachable
From {1.2.3.4} icmp_seq=3 Destination Host Unreachable
```

Edit network file with:
```
vi /etc/netplan/01-netcfg.yaml
```

Where you see:
```
- to: 0.0.0.0/0
  via: 1.2.3.4
  on-link: true
```

Add this record before it:
```
- to: 1.2.3.102/32
  via: 1.2.3.4
```

And do that for every server. I am not sure if restart was needed or not. Try pinging and see for yourself.

## issue: CloudFlare domain had too strict rules
I am using the development domain I will call example.com to manage development stuff.
I have set it as a wildcard domain in CloudFlare and registered Rancher as dashboard.example.com, but I had to
```
Mark CloudFlare domain TLS to FULL (and not STRICT)
```
So the requests still get thru to the Rancher dashboard, even when the encryption is only self-signed at the beginning.

I have changed my CloudFlare rules for the domain example.com to:
- removing force https in security
- adding the first rule:
```
*example.com/.well-known/ => SSL: off
```
- optional rule in between, for www redirect with preserving URL path:
```
www.example.com/* => Forwarding URL (Status Code: 301 - Permanent Redirect, Url: https://example.com/$1)
```
- adding the last rule for forcing https:
```
*example.com/ => Always Use HTTPS
```

## issue: error localhost:8080 not reachable
```
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
```

## issue: error cert-manager timeout error
do loops until it succeeds
```
helm fetch jetstack/cert-manager --untar && cd cert-manager &&
while ! helm install --namespace cert-manager --create-namespace --set installCRDs=true -- cert-manager . ;
do helm uninstall cert-manager -n cert-manager ; done
```

## issue: error incompatible kube version
```
Error: INSTALLATION FAILED: chart requires kubeVersion: < 1.24.0-0 which is incompatible with Kubernetes v1.24.3+k3s1
```

install older version of k3s (see releases [here](https://github.com/k3s-io/k3s/releases/))
```
curl https://get.k3s.io | INSTALL_K3S_VERSION=v1.23.9+k3s1 ...
```

## issue: forgotten finish installation URL
```
echo https://dashboard.example.com/dashboard/?setup=$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}')
```

## suggestion
On your local machine, install the following:
- helm
- kubectl

It is much easier to work from a local terminal than thru ssh terminal.

# where I am today and what is the plan

My development flow could be better, but we are getting there.

I am stuck with having docker registries available in every application namespace. Doing a local build of docker images with automatic upload to the registry and deployment thru the local terminal connected to the cluster.

I will upgrade and automate things as I go. But, for now, it is enough.

# conclusion

The experience was one of the hardest I have ever had in my 20+ years in this business. And I have done dockers from scratch and used vim as the primary editor.

Even though things are not perfect, the upgraded deployment flow is still blazingly fast and reliable, and all I was dreaming about before starting the whole upgrading trip.

Would I do it again? Yes, but with more beer and more breaks in between.

Till next time, stay sexy and hydrated.




Stack 2023 - kubernetes with Rancher

# time to learn something new
I got sick of my overhead with the current workflow, local environment, hosting, servers, deployment services, server maintenance, multiple instances, and separating more robust servers from temporary ones. Etc.

## disclaimer
Most of my projects are just that and nothing else. So it's the playground to get the production experiences I can use at my primary job, upgrade my CV, and prepare for the next gig.
Even though we talk about personal projects, I am highly bothered that I can not simply jump on the idea I have at the moment but am pulled back by the simplest things like local environments, domains, hosting, build flows, testing, or otherwise temporary domains, certificates.
Other inspirations come from my primary job circumstances, where things are always exciting.

## unsubscribe from current SaaS
I do not have many subscriptions to services, but at the end of the month, I could pay for a couple of streaming services with that money. So for them to just be there, that is too much.
I want a full self-hosted or managed environment where I can spawn a new socket server or five, a custom docker registry, a couple of CI/CD workers, etc. And if / when things become resource-consuming, I can scale just hardware without paying the upgraded subscription.

## quick option to switch servers
Whenever I set a new project and assign the server I want the project to run on, it soon becomes an issue when the resources are needed somewhere else; I am removing the server or simply upgrading to a better one or with another provider.

## no manual scaling overhead
Related to switching servers, making things scale big or small in a short time is a huge burden and an overhead I want to make as simple as possible.

## single hosting dashboard
Related to SaaS + multiple servers, multiple clients, ..., things soon become a bureaucracy what is where. I want to have it all in one place.

## simple management of application
Encapsulate the application for real, try things out, and don't worry about other things you also host. For example, adding one should be a matter of a couple of lines, not 15 screens and eight wizards.

## zero downtime for the applications
In the old world, we uploaded the new files of the application and hoped for the best. Usually, turning the maintenance page on or trying some symlink magic to achieve zero downtimes. Unfortunately, production rollbacks are still a mess and a full-blown risk.
There must be a better world somewhere today.

## multiple environments for features testing and previews
Having a test environment is already a privilege not all developers have. Pre-production and staging are almost like spam mail from a "prince that wants to give you money." What about a situation where the product is enormous, and each feature testing needs its environment? Or when you start developing on some huge refactor or new core functionality like replacing the whole internal system for events, sockets, etc.? What if a client also wants to have its sandbox/demo to play? Etc.

# the plan
Let's jump into the topic and see how far I can get before admitting all the mistakes I have made in the name of "Why should I check the documentation that takes 15 minutes if I can do that on my own in 5 hours."

Till next time, stay sexy and hydrated.

