```
NOTE:
I went thru quite a wild journey here and am publishing this post months after the struggle I have had with the whole situation.
Maybe some of you are smart enough to learn just from reading about it. I definitely had to experience it on my own just to end up somewhere else.  
```

# the idea

Having a balanced shared environment with the idea of containers in k8s that won't make your system "dirty" with the time of playing around, it seemed like a good idea to containerize everything from source code management to deployer, CI builds, automatic hostings and push to deploy automation based on the branches, etc.

# definition of a plan
- an instance of Gitlab for source code for all projects
- reusable CI builds for all the projects
- automatic builds and deploys on git push
- all other actions should be project / k8s namespaced
- per project (k8s namespace) Gitlab workers
- per project (namespace) docker registry
- per project "production" and "development" hosting namespaces
- production deploy for pre-defined domain on manual request or based on some (release/master) branch
- automatic hostings (dev subdomains) and deploys for every feature branch or set of branches based on some rule

# expectations
with time it should be:
- easy to start new projects
- easy to spawn project hostings
- easy to (automatically available) spawn development/staging hostings
- easy to monitor and restrict resources per project
- easy to scale/move a project to a cluster with better hardware when needed

# why didn't I use existing solutions
Any popular service gets hacked sooner or later, and with the latest service I was looking into just announcing a breach, I didn't want to expose another public thing that is not under my control.

I am no expert on security, don't understand me wrong. I am "just" a developer and learned things with time and fails that production experience brings.

And I am aware I can not do things better, but in reality, the "obscurity over security" still wins in the end, at least until your service becomes popular.

# reality
Oh boy, was I wrong on everything ...

## installing Gitlab (security nightmare)

Not knowing enough about Kubernetes and self-hosted GitLab at that point, I followed one of the really old tutorials for days to come. The biggest pain was that Gitlab persisted data only till the first restart (volume issues). Finally dropping out of the idea of doing it manually, I used Helm and realized I wasnt even close to the end solution. It worked, but the risk of losing all the project source code just because I would make another mistake in the future was simply too big.

The next plan was to do it on a dedicated server and go thru all the configs and possible risks. Again, it worked as intended but did not solve any of the performance issues I planned to solve. Bottleneck it with workers was not an acceptable solution.

Starting with workers sounds all legit in idea, but in reality, having a dynamically scaled set of workers becomes its own risk. Having them in the cluster with a bit too much access level opened countless new blog posts I had to read on security in k8s.

...

# conclusion

I failed successfully big time in so many ways on so many levels!

Why successfully?
- I believe the idea for that kind of tool is the right one
- I learned a lot about Gitlab on-premise and got even more respect for the sys guys
- learned something new about workers, security issues, and limitations
- deeper dive into k8s security, bottlenecks of GitOps, single source of truth, and the risk of putting all into a single point of failure system design
- understanding why there are not a lot of solutions like that and why only big companies who are host providers and enterprise ones offer solutions on a level like that

What to do next?

The whole idea sounded like a great SaaS to make, just to realize weeks later how big of a project and market you should have for it to support the paychecks of involved people. The risk of responsibility simply wasnt worth it to continue.

I am so glad I went thru that whole story from a "simple idea" to reality, back to the idea of creating a SaaS and dropping out of it before even more time and money would be involved. A lifecycle of a project idea was done in weeks instead of months like it usually happened. I call that a total victory!

Till next time, stay sexy and hydrated.


Stack 2023 - going cheaper fail

Stack 2023 - Going cheaper with self-hosted services fail

# love at first sight
Scrolling thru lego sets on their website weekly produces a strong, opinionated wish list and level of excitement one or the other set will (probably) bring. Once this set was announced, all other wishes were forgotten in a blink!

I checked the website every couple of days to see if I remembered the date correctly, even if the date was far away at that time. In the end, the check was done daily! When the button changed from "coming soon" to "add to bag," I was there, and the package was on its way.

# what I have expected
I knew the set would be a must-have collection item, deserving a spot on one shelf with lights. The idea of wildlife and a reminder of what I want to practice at some point in my life growing bit older. I took it as a sign that things needed to start moving one way or the other.

Why not start with a meditative time of bringing this set to life and defining the life I will proceed with.

With the set being on a lower scale of how many pieces it has, my expectations were primarily on the outer details of groups of topics one sees at first glance. Let's see what happens as I build.

# what I really got
One of the cutest builds I have had with a set. The instructions are split into smaller sections, which look like a set of long paths in front of you but come out as calming achievement after every topic. It looked even too simple at some point, but the satisfaction of continuing with the next one was amazing.

The details on every part of the build are fascinating, and I think I haven't seen a set that would bring me such fascination. See the details at the end of the article and see it for yourself.

# conclusion
I can say "one of the cutest sets" to build, and collection-wise, one of my favorite items.

![Part of the collection](/media/lego/a-frame-cabin-featured.jpeg)

# details that fascinated me during build

![a-frame-cabin-unboxed.jpeg](/media/lego/a-frame-cabin-unboxed.jpeg)

![a-frame-cabin-back-details.jpeg](/media/lego/a-frame-cabin-back-details.jpeg)

![a-frame-cabin-back-details-2.jpeg](/media/lego/a-frame-cabin-back-details-2.jpeg)

![a-frame-cabin-back-details-3.jpeg](/media/lego/a-frame-cabin-back-details-3.jpeg)

![a-frame-cabin-bed-details.jpeg](/media/lego/a-frame-cabin-bed-details.jpeg)

![a-frame-cabin-bird.jpeg](/media/lego/a-frame-cabin-bird.jpeg)

![a-frame-cabin-camera.jpeg](/media/lego/a-frame-cabin-camera.jpeg)

![a-frame-cabin-chair-1.jpeg](/media/lego/a-frame-cabin-chair-1.jpeg)

![a-frame-cabin-chair-2.jpeg](/media/lego/a-frame-cabin-chair-2.jpeg)

![a-frame-cabin-cute-figures.jpeg](/media/lego/a-frame-cabin-cute-figures.jpeg)

![a-frame-cabin-designs.jpeg](/media/lego/a-frame-cabin-designs.jpeg)

![a-frame-cabin-detailed-trees.jpeg](/media/lego/a-frame-cabin-detailed-trees.jpeg)

![a-frame-cabin-detailed-trees-2.jpeg](/media/lego/a-frame-cabin-detailed-trees-2.jpeg)

![a-frame-cabin-figures-2.jpeg](/media/lego/a-frame-cabin-figures-2.jpeg)

![a-frame-cabin-front-details.jpeg](/media/lego/a-frame-cabin-front-details.jpeg)

![a-frame-cabin-front-stairs.jpeg](/media/lego/a-frame-cabin-front-stairs.jpeg)

![a-frame-cabin-front-table.jpeg](/media/lego/a-frame-cabin-front-table.jpeg)

![a-frame-cabin-guitar.jpeg](/media/lego/a-frame-cabin-guitar.jpeg)

![a-frame-cabin-hammers.jpeg](/media/lego/a-frame-cabin-hammers.jpeg)

![a-frame-cabin-hidden-animals.jpeg](/media/lego/a-frame-cabin-hidden-animals.jpeg)

![a-frame-cabin-kitchen.jpeg](/media/lego/a-frame-cabin-kitchen.jpeg)

![a-frame-cabin-office.jpeg](/media/lego/a-frame-cabin-office.jpeg)

![a-frame-cabin-organic-back.jpeg](/media/lego/a-frame-cabin-organic-back.jpeg)

![a-frame-cabin-organic-front.jpeg](/media/lego/a-frame-cabin-organic-front.jpeg)

![a-frame-cabin-organic-roof.jpeg](/media/lego/a-frame-cabin-organic-roof.jpeg)

![a-frame-cabin-organic-sides.jpeg](/media/lego/a-frame-cabin-organic-sides.jpeg)

![a-frame-cabin-otter.jpeg](/media/lego/a-frame-cabin-otter.jpeg)

![a-frame-cabin-records.jpeg](/media/lego/a-frame-cabin-records.jpeg)

![a-frame-cabin-roof-windows-alignment-idea.jpeg](/media/lego/a-frame-cabin-roof-windows-alignment-idea.jpeg)

![a-frame-cabin-small-details.jpeg](/media/lego/a-frame-cabin-small-details.jpeg)

![a-frame-cabin-spider.jpeg](/media/lego/a-frame-cabin-spider.jpeg)

![a-frame-cabin-stone-front-made-of-hammers.jpeg](/media/lego/a-frame-cabin-stone-front-made-of-hammers.jpeg)

![a-frame-cabin-upper-floor-details.jpeg](/media/lego/a-frame-cabin-upper-floor-details.jpeg)

![a-frame-cabin-upper-floor-details-2.jpeg](/media/lego/a-frame-cabin-upper-floor-details-2.jpeg)

![a-frame-cabin-upper-floor-details-3.jpeg](/media/lego/a-frame-cabin-upper-floor-details-3.jpeg)

![a-frame-cabin-wood-storage.jpeg](/media/lego/a-frame-cabin-wood-storage.jpeg)

![a-frame-cabin-wood-storage-2.jpeg](/media/lego/a-frame-cabin-wood-storage-2.jpeg)


Till next time, stay sexy and hydrated.


A-Frame Cabin - review


# the idea
As described in the previous post of this series, I want to replace the old days' deployment flows and make hosting more scalable and usable for multiple functionalities without the risk of making the system dirty with unwanted services when playing around with new technologies.

A friend of mine ( [Omer's blog](https://omeratagun.net/) ) suggested using [Rancher](https://www.rancher.com/) as the next thing. He also prepared [a post](https://omeratagun.net/blog/how-to-install-rancher) for the topic of our conversation, which triggered me enough to replace everything I was using.

# trying out a new server provider
My last reliable providers were [Linode](https://www.linode.com/) and [DigitalOcean](https://www.digitalocean.com/). I wanted to go cheaper for the new playing around and keep them for existing production stuff. I had some production experience with [Hetzner](https://www.hetzner.com/) in the past. To round the experience, it all works as expected till it doesn't. And at those times, cheaper solutions always left me in deep shit with clients.

This time I am going to try [Contabo](https://contabo.com).

# installation of k8s (Kubernetes) and Rancher

```
Note: You may wanto to jump to #issues first, as there are many tips and tricks related to the installation.

!!! WARNING !!!

The content of this post was tested and used in Aug 2022, and if it's months/years from now, follow these instructions with caution so you don't repeat my mistake!

!!! WARNING !!!
```

I went with the idea of HA (high availability) cluster with an embedded database. Not recommended, but it's a risk I am willing to take or solve in the future.

[This link](https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli#install-rancher-with-helm) provides most of the info, but I will pin all the other links as we go.

As versions of documentation change with new releases, I am also pasting a Rancher v2.6 link [here](https://ranchermanager.docs.rancher.com/v2.6/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli).

## operating system
I just used the latest Ubuntu LTS.

## installing Kubernetes cluster using k3s (Lightweight Kubernetes)
Following instructions [here](https://docs.k3s.io/installation/ha-embedded). It should work out of the box on the first run.

These are my notes:
```
NOTE: replace the version in commands
```

Install and init ha cluster primary server with an embedded database.
```
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.21.7+k3s1" sh -s - server --cluster-init
```
Alternative: Install and init ha cluster primary server with the external database.
```
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.21.7+k3s1" sh -s - server --cluster-init \
  --datastore-endpoint="mysql://username:password@tcp(hostname:3306)/database-name"
```

Add more servers to the primary server:
- choose a good name for "K3S_NODE_NAME" as you can not change it later (without a hustle)
- for renaming the primary server, I had to remove and re-add it to the cluster with a new name, which worked on the first try
```
curl -sfL https://get.k3s.io | K3S_NODE_NAME="{server-1}" INSTALL_K3S_VERSION="v1.21.7+k3s1" K3S_URL=https://{REPLACE_WITH_PRIMARY_SERVER_IP}:6443 K3S_TOKEN={REPLACE_WITH_K3S_TOKEN} sh -s - server
```
For the HA cluster, it is suggested to have (at least) 3 servers.

Add workers with:
- choose a good name for "K3S_NODE_NAME" as you can not change it later (without a hustle)
```
curl -sfL https://get.k3s.io | K3S_NODE_NAME="worker-1" INSTALL_K3S_VERSION="v1.21.7+k3s1" K3S_URL=https://{REPLACE_WITH_PRIMARY_SERVER_IP}:6443 K3S_TOKEN={REPLACE_WITH_K3S_TOKEN} sh -
```

You can add cluster access to your local machine by running this from the local machine terminal.
```
scp root@{REPLACE_ME_WITH_SERVER_IP}:/etc/rancher/k3s/k3s.yaml ~/.kube/config
```

To get the server token, run this on the server.
```
cat /var/lib/rancher/k3s/server/node-token
```

## install Rancher using helm charts
The latest guide is (should be) available [here](https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli) (v2.6 [here](https://ranchermanager.docs.rancher.com/v2.6/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli)).

```
!!! WARNING !!!

The guide uses the LATEST version, not the STABLE one. Change links accordingly.

!!! WARNING !!!
```

My notes, using cert-manager 1.9.1 and the STABLE version of Rancher:
```
helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

kubectl create namespace cattle-system

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.crds.yaml

helm repo add jetstack https://charts.jetstack.io

helm repo update

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.9.1 \
  --set installCRDs=true

helm upgrade --install rancher-stable/Rancher
--namespace cattle-system
--create-namespace
--set hostname={REPLACE_ME_WITH_RANCHER_DOMAIN}
--set bootstrapPassword={REPLACE_ME_WITH_SOME_BOOT_PASSWORD}
--set ingress.tls.source=letsEncrypt
--set letsEncrypt.email={REPLACE_ME_WITH_EMAIL}
```

If you have correctly directed domain DNS to the rancher server (also check CloudFlare issue fix), you should be able to visit the login page of Rancher and use the boot password to set up login credentials.

## setting up certificates manager
The next thing was to make domains use genuine certificates. I had a lot of issues setting this up, as every version had a few different instructions, and tutorials always missed a thing or two. I have been stuck here for days!

```
!!! WARNING !!!

You need to set up `cluster-issuer`, not `issuer', as you are using HA cluster, not a single server.
```

```
!!! WARNING !!!

Tutorials all showed examples for `nginx`, but I had to use `traefik` for the `solver` property. This took me days to find out.
```

I ended up changing/setting up 2 credential issuers.
```
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates and issues related to your account.
    email: {info@example.com}
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-staging-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: traefik
```

```
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
  namespace: cert-manager
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates and issues related to your account.
    email: {info@example.com}
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-production-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: traefik
```

```
!!! WARNING !!!

If you are behind CloudFlare, you need to allow HTTP (not secure) requests (disable FORCE HTTPS), etc.; otherwise, the challenge won't complete!
```

You should probably go with a different challenge than http01, like DNS or something similar.

Now you can request a new certificate by creating a new "certificate" under the certificate manager, "Certificates." The certificate will see if a secret key exists for it; otherwise, it triggers certificate requests, challenges, etc ... I had many issues here, so welcome to ping me for help.

Certificate for rancher domain:
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dashboard-example-com-tls
  namespace: Rancher
spec:
  # choose your own
  secretName: dashboard-example-com-tls
  dnsNames:
  - dashboard.example.com
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
    group: cert-manager.io
```

Certificate for an application that lives in namespace 'test-application.'
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  # I use the same names for a secret key, but that is not needed
  name: test-application-com-tls
  namespace: test-application
spec:
  secretName: test-application-com-tls
  dnsNames:
  - test.application.com
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
    group: cert-manager.io
```

## setting up disk management with longhorn

I followed the install thru the "Apps" navigation menu. Nothing special is needed there, but things won't work out of the box. Some requirements could be more noticeable when doing the installation. I had to read longhorn documentation line by line to find those fu\*\*\*rs.

This needs to be run on every cluster node/server.

Check requirements with script their script (change the version) and install dependencies till it shows green for all checks:
```
curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.3.1/scripts/environment_check.sh | bash
```

Dependencies (I had to install):
```
apt-get install jq
apt-get install open-iscsi
apt-get install nfs-common
```

## testing
The idea is simple, but I won't lie, have good luck trying it out, as anything can go wrong, and nothing will work.

Idea:
- create application namespace
  - !!! everything needs to be part of that namespace from now on
- create a new deployment configuration
  - docker image source
  - disk requests
  - load balancer
  - ...
- add a certificate for the domain
- add ingress rule
  - add domain
  - select certificate
  - connect to the load balancer port

What I have done in the beginning:
- created my own docker image FROM nginx, replacing just index.html hello content
- uploaded it to the docker hub
- added docker hub credentials to Rancher (application-specific namespace)
- used some subdomain where the dashboard is (testing.example.com)
- used letsencrypt staging first
- visited ports directly
- ...


# issues
Following different tutorials, the whole k8s and Rancher progress was a mess, and it required lots of nerves and enthusiasm from me to react to the goal I wanted. Being just a month late on the tutorial, nothing worked. The whole thing reminded me of the JavaScript npm world. There was even a point where I followed the Gitlab installation tutorial dated just a couple of months old, to realize at the end it was just a copy of a years-old tutorial published elsewhere. Just FML at so many points, I was sure I was having a nightmare.

That's why I will pinpoint some issues and solutions, paste some links I followed, etc.


## issue: tutorials and docs use "latest," not "stable"
Be careful, as most documentation refers to the latest releases rather than stable ones. Expect lots of issues there and always check the version and read the commands you are going to run carefully.

## issue: servers from Contabo can't see each other
Before installing a k8s cluster, some things must be solved first.

I suggest you do a sanity check and try pinging servers from one another to avoid problems when installing a k8s cluster.

The Contabo issue may be present only for some servers (I did not investigate the problem), but for the ping to work between them, I had to follow the instructions [here](https://contabo.com/blog/creating-static-routes/).

I would sum it up with the following:
- you have server A with IP (1.2.3.101)
- you have server B with IP (1.2.3.102)
- you have server C with IP (1.2.3.103)
- public route within is 1.2.3.4

Go to each server and check if they can see each other with ping. If you get a response like the one below, you need mentioned changes.

```
root@servername: ping 1.2.3.102
PING 1.2.3.102 (1.2.3.102) 56(84) bytes of data.
From {1.2.3.4} icmp_seq=1 Destination Host Unreachable
From {1.2.3.4} icmp_seq=2 Destination Host Unreachable
From {1.2.3.4} icmp_seq=3 Destination Host Unreachable
```

Edit network file with:
```
vi /etc/netplan/01-netcfg.yaml
```

Where you see:
```
- to: 0.0.0.0/0
  via: 1.2.3.4
  on-link: true
```

Add this record before it:
```
- to: 1.2.3.102/32
  via: 1.2.3.4
```

And do that for every server. I am not sure if restart was needed or not. Try pinging and see for yourself.

## issue: CloudFlare domain had too strict rules
I am using the development domain I will call example.com to manage development stuff.
I have set it as a wildcard domain in CloudFlare and registered Rancher as dashboard.example.com, but I had to
```
Mark CloudFlare domain TLS to FULL (and not STRICT)
```
So the requests still get thru to the Rancher dashboard, even when the encryption is only self-signed at the beginning.

I have changed my CloudFlare rules for the domain example.com to:
- removing force https in security
- adding the first rule:
```
*example.com/.well-known/ => SSL: off
```
- optional rule in between, for www redirect with preserving URL path:
```
www.example.com/* => Forwarding URL (Status Code: 301 - Permanent Redirect, Url: https://example.com/$1)
```
- adding the last rule for forcing https:
```
*example.com/ => Always Use HTTPS
```

## issue: error localhost:8080 not reachable
```
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
```

## issue: error cert-manager timeout error
do loops until it succeeds
```
helm fetch jetstack/cert-manager --untar && cd cert-manager &&
while ! helm install --namespace cert-manager --create-namespace --set installCRDs=true -- cert-manager . ;
do helm uninstall cert-manager -n cert-manager ; done
```

## issue: error incompatible kube version
```
Error: INSTALLATION FAILED: chart requires kubeVersion: < 1.24.0-0 which is incompatible with Kubernetes v1.24.3+k3s1
```

install older version of k3s (see releases [here](https://github.com/k3s-io/k3s/releases/))
```
curl https://get.k3s.io | INSTALL_K3S_VERSION=v1.23.9+k3s1 ...
```

## issue: forgotten finish installation URL
```
echo https://dashboard.example.com/dashboard/?setup=$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}')
```

## suggestion
On your local machine, install the following:
- helm
- kubectl

It is much easier to work from a local terminal than thru ssh terminal.

# where I am today and what is the plan

My development flow could be better, but we are getting there.

I am stuck with having docker registries available in every application namespace. Doing a local build of docker images with automatic upload to the registry and deployment thru the local terminal connected to the cluster.

I will upgrade and automate things as I go. But, for now, it is enough.

# conclusion

The experience was one of the hardest I have ever had in my 20+ years in this business. And I have done dockers from scratch and used vim as the primary editor.

Even though things are not perfect, the upgraded deployment flow is still blazingly fast and reliable, and all I was dreaming about before starting the whole upgrading trip.

Would I do it again? Yes, but with more beer and more breaks in between.

Till next time, stay sexy and hydrated.




Stack 2023 - kubernetes with Rancher

# the "pending blocker" is finally gone
I've had this set for six months, and it had as many intentions as only a developer can have with shelf projects. The last one is "no more new sets till this one is done." So there was just one other option to do. Buy new sets anyway!

The anticipation of getting new ones left me with no choice but to bring this one to life while waiting.

# what I have expected
Back in the day, there was little focus on smooth and clean lego designs. All the sets required quite a lot of imagination to be seen as life-like, which made childhood so much better and with more options. Seeing how lego changes things by adding many small details to the finishing touches, where you don't even notice that an item is a lego. They adequately addressed what grownup fans (with money) want.

It's a thing that needs to live on a shelf. Not a build-and-destroy kind of thing, but a piece of a happy memory and all the possibilities that we bring from our childhood.

# what I really got
A much more extensive set than I anticipated. I knew the sets were getting twice or three times as big counting the pieces needed for the size of the final product. This one had twice as many legos as it looked at first sight.

All the parts are firmly placed together in a way that at no point do you have any fear of breaking a single thing. Well, almost ... only the front mask of the car had some issues, but ignore those 3 parts.

Building it showed the author's masterpiece using lego in ways never used in the older sets. At least not at that scale and at all stages of the build. Perpendicular builds look so simple when done, but only a few of us could integrate such a strong structure.

The model's ratio still needs to be clarified. I am not used to building things just for the model itself, but always for lego or technic figures. On the other hand, this opened so much room for the details that wouldn't be here otherwise.

# details that fascinated me

![Curtains on all windows](/media/lego/volkswagen-curtains.jpeg)

![Fridge content](/media/lego/volkswagen-fridge.jpeg)

![Kitchen counter details](/media/lego/volkswagen-kitchen.jpeg)

![Perpendicular masterpiece and model firmness](/media/lego/volkswagen-perpendicular-masterpiece.jpeg)

![Perpendicular masterpiece and model firmness](/media/lego/volkswagen-perpendicular-masterpiece-1.jpeg)

![Propane behind the kitchen facade](/media/lego/volkswagen-propane.jpeg)

![Seats](/media/lego/volkswagen-seats-1.jpeg)

![Seats](/media/lego/volkswagen-seats-2.jpeg)

![Smooth finish touches for all surfaces](/media/lego/volkswagen-smooth-finish-1.jpeg)

![Working steering](/media/lego/volkswagen-steering.jpeg)

# conclusion
In the end, it took too much energy to assemble. But looking back, I can't find a thing I would be willing to sacrifice from the set for it to be done quicker.

I recommend this set to anyone who firmly believes in lego, as the symbolic message outweighs everything else.

![Part of the collection](/media/lego/volkswagen-collection.jpeg)


Till next time, stay sexy and hydrated.


Volkswagen T2 Camper Van - review

## who we are
The team of web backend developers, doing custom cms solutions in Laravel for a couple of years now. Solutions are pretty advanced, and following one of the biggest names in the field definitely puts us on the high end of our skill sets and the understanding level of Laravel we provide.

## Laracon eu Lisbon 2023
The company promoted visiting gatherings like that to gain new ideas and inspiration, so we found Laracon EU in Lisbon, which was happening in a month or two. So the unplanned team-building, brainstorming, and lots of fun were waiting for us in one way or the other.

## the day before
Nothing special was happening in the team the day before. However, some small plans were made about how we should travel to the airport and how independent we wanted to be in the preparations.

## trip happening
Trying not to hurry things, we plan to arrive a day before Laracon starts, taking the flight as soon as 6 in the morning to have a complete rest day and the whole evening. None of us was ever in Lisbon before.
Some started the journey at midnight, whereas most of us gathered around 3 in the morning to catch the flight. We even had sandwiches and coffee as we had back in school, so a total nerd trip.

This was our first flying experience for one of us, so a trip's fun was guaranteed.

# day 1
The atmosphere was just as I remembered it, and it triggered the developer part of me with a feeling of enthusiasm and a "things are gonna happen!" voice in my head. It was nice to see Jeffrey Way, with whom many developers grow. The hard truth by Stefan Bauer describes a story of struggle. A must-new tool presented by Nuno Maduro that will change the whole merge review flow requirements.

And thanks god for the food they prepared for us there as things in Lisbon were not playing out well for us. We may be the unluckiest visitors, but here we were.

# day 2
Backend guys already love tailwind, as it allowed us to update our CV to a full-stack developer now. The topic of Termwind by Francisco Madeira made us want to do our own websites in the terminal from now on. As it is hard to imagine anything better than a website in the terminal, we were awakened by the shock of chicken on pizza presented at Batchable Jobs by Bobby Bouwmann, but we still need to say huge thanks to that guy as he gave us the solution we were craving for at the time. I am talking about "you can add jobs to batch as you go."

All the evenings ended with late coffee, smoking, and lively conversations, and that one was no different, no matter that it was the last day.

# going home
It was a circus of its own. Flights being late, connected flights waiting for us, lost luggage, and no smoking time. All were happening to the guy flying for the first time. Nothing stopped us even for a second, as it was still a full-blown, relaxed gathering, no matter what happened, with lots of new inside jokes

The image tells a lot of stories, but only we know what really happened :)

![He was not so generous that day](/media/blog/laracon-eu-lisbon-2023-story.JPG)

Till next time, stay sexy and hydrated.


Laracon EU Lisbon 2023


[//]: # (TODO add content index component here)

# Application requirements
Let's start by defining what an application usually needs so the idea can start and step into the real world. Not even going into the long-term project that grows.
- API data endpoints
- user sessions, authentication, and authorization
- queues/background workers
- application UI, usually SPA
- homepage/application presentation page (for not logged users)
- about page
- blog posts
- news
- subscription managing
- SEO, robots.txt, and sitemap.xml
- other website-related pages and promotions like landing pages, mailings, etc
- deployment (and rollback)
- backups
- database
- persistent storage
- in-memory caching (Redis etc.)
- (sending) mails
- servers/hosting
- (local) development environment
- staging environment
- production environment
- code versioning

# How have things been done till now, and what needs an upgrade in 2023
## servers, deployment, hosting, and hosted environments
I got away from self-hosted servers a while ago and let it all be managed by [Laravel Forge](https://forge.laravel.com/), as it helped me remove most of the self-managed LAMP stack from the previous stack and also added some new ideas I wanted to implement. In the end, it is a fantastic tool that helped me grow to the next level as a freelancer, but with time, it has become (just) an automation of managing LAMP stack servers. Therefore, I am still promoting it as an excellent tool for beginners, freelancers, small project teams, and smaller projects. But I need more now.

Review of current/old flow:

Pros:

- simple push to deploy
- servers are managed by Forge
- security auto-update services
- adding (buying) servers thru the dashboard
- letsencrypt automation
- repository (branch) based deploy integration

Cons:

- site setup overhead and missing auto-setup for some services (e.g., wrong Postgres port)
- hard to edit/manage project-specific Nginx config and some other services
- setting up multiple environments for the same project (stg, dev, ...)
- no scaling / single server application limitation
- risks of no proper backup or failover
- no zero-downtime deploys
- limitation to the shared database and redis services (be careful with redis db ids)
- no resource management / buying a dedicated server upfront based on predictions
- production includes services that are there just for the build time
- semi-solution for tests and build process
- no automatic solution for moving the application to another server
- deploy the whole application no matter how small of a change needed

```
Conclusion: the whole stack needs to change
```

## users, sessions, authentication, authorization, API endpoints
I have been sticking with the Laravel framework ecosystem for managing backend stuff for the past couple of years. It guided me well into the topics and patterns I now understand are a must for every larger project. I liked the idea of Lumen, but I still used it as a monolith.

Review of current/old flow:

Pros:
- great suggested patterns for growing as a developer on larger projects
- solves most application topics (workers, router, validation, ...)
- Test Driven Development of APIs
- I know it inside out / source code is my primary documentation
- fast-developing

Cons:

- cookie-based sessions (used it because of less security overhead as the browser took care of storing it properly)
- hard to target and handle needed performance improvements and monitoring in larger teams with lots of codebase and new features
- monitoring of applications, database, queues, ...
- low performance (not using proper approaches like octane etc.)

```
Conclusion: partial upgrades are needed, but sticking with the stack
```

## frontend framework, spa, blades, compiling assets, CSS framework, auth pages
I was switching my primary stack back when Laravel was version 5.6. When vue version 2 was the default language in the starter project, it seemed a natural next step from the jQuery world. I've tried angular, ember, and some other frameworks at the time, but vue felt the most natural way of doing things. For the CSS, there was bulma, bootstrap, and materializecss. SPA application was served thru url root html output from Laravel blades, and compiling was done thru Laravel mix. Everything is already bundled from the Laravel starting project.

Review of current/old flow:

Pros:
- it all came out of the box, easy to start
- known approaches
- all in one project/stack / single repository
- fast mockup using CSS frameworks

Cons:
- mixing things from blades to vue is a mess
- its hard to maintain/define the proper long-term structure of the app
- vue SPA inbound to Laravel loading things
- mixing Laravel rendering for some things and others in SPA is a mess
- hard to customize or add additional functionalities with CSS frameworks
- CSS frameworks usually require js logic out of the box

```
Conclusion: the whole stack needs to change
```

## webpage specific pages, blog, news, SEO
Using the same stack for web pages as I do for the applications is a nightmare. You can choose between having another stack, like WordPress, or developing new cms as part of your application. As I want to avoid ending up with multiple stacks just to make it look nice, I will focus on finding a better solution that will bring the website and application parts together.

Review of current/old flow:

Pros:
- (none found)

Cons:
- can solve (some) things by having a separate website-specific solution / new stack
- can solve (some) things by developing or including package dependency features needed for websites
- SEO / no proper html output for robots without javascript
- dual development to make the experience between website and application as smooth as possible
- maintenance of multiple systems
- hard to onboard developers onto the project
- ... (this can go on forever)

```
Conclusion: the whole stack needs to change
```

## local environment
Using docker-compose as part of the project, being project specific, was a go-to tool for some time.
Note: I have tried using valet with local npm (nvm), database, etc., but did not like that my system is full of services with multiple versions, etc. Startup scripts started to fail with upgrades or needed changes. Even a bigger mess that took much more time with new machines etc.

Review of current/old flow:

Pros:
- trust in the quick setup of the project environment
- encapsulated services required for a project
- good simulation and testing environment for production requirements

Cons:
- copy-paste from one project to another / no proper boilerplate from which to initialize
- manually editing hosts file for project-specific domains
- wildcard subdomains manual setup
- using a secure connection locally
- performance issues with the local filesystem
- running only a single instance / project at the time

```
Conclusion: some improvements wanted
```

# expectations for the new stack
- independent parts / each part of the application can be quickly replaced with a better solution
- best SEO practices
- scalability
- portability
- result-oriented / quick start with the new app
- long-term maintainability

```
Conclusion: let's do this shit
```

Till next time, stay sexy and hydrated.


Stack 2023 - review the old way

I am going into the new year by refreshing the working stack and a new website. I will address and solve some topics that have bothered me since web development invented the LAMP stack.

I will keep all the functionalities of the modern blog website as a tool for sharing information and expressing myself without following most of the known patterns and components a blog usually has.

## different approach to website
I am bored with the standard website structure, menus, contact pages, etc. There needs to be an easier way to present the info the visitor comes for.

## website components
I will remove standard components like the contact form, comments, search, etc. Instead, give the content and the organic feeling of content flow without presenting the page's structure.

## website views/page templates
Approaching the whole website as a simple gallery with all the available articles being there as you land on the website. To the experience of zooming the image / viewing the article, you want to read.
One and a half of a template is required for everything the website needs. Categories, series index, etc., can all be presented in this gallery and zoomed image view.

## going static
I've been playing with the idea of having a fully static application or website with no moving parts for some time now. The idea feels light and secure, and it gives me peace.
Putting the moving parts on the side of a browser and the users' services (like google drive for storage) is also a proper approach for the maintainability of the solution.

## maintenance
With lots of "hello world" projects hanging around, from a todo app to the chat socket server, there is always this nagging feeling when things will break one way or the other. The same goes for the website, even if we only talk about the website.

## security
Upgrading packages in all the projects take a lot of time.

## services
Dependency on the services like databases brings its own risk of maintaining them and keeping them up all the time.

## storage
There is always a chance some unwanted files end up there, or things need to sync correctly.

## frameworks
From WordPress to Laravel, lots of overhead is there to run them properly through time.

## scaling
Scaling the application servers horizontally is a pain. Shared database and storage the old way is a challenge I want to avoid.

## backups
I need some automation to go away from the Schrödinger backups.

## servers
The risk of upgrading them with all the projects is a pain I don't want anymore.

## conclusion
Achieving all the goals will be a considerable challenge, but it is time to finally make it a reality and put the idea I've had for years out of my memory and make room for some new ones.

Till next time, stay sexy and hydrated.


Going stateless

## from resolve() dark magic to expected dark magic

I am talking about the magic resolver of Laravel resolve() that you see in many articles and tutorials. If you haven't already, I suggest you learn how does container and magic binding resolver work in Laravel, as it changes how you approach things in the long run.

## The old way
```php
/** @var MyCustomClass $instance */
$instance = resolve(MyCustomClass::class);
```
  
Pros:
- can resolve any kind of type
- shorter signature
 
Cons:
- can resolve any kind of type
- requires docblock for autocompletion
- refactoring code may not be fully automatic
- prone to typos in docblock

## The new way
```php
$instance = (static fn():MyCustomClass => resolve(MyCustomClass::class)());
```
Pros:
- ensures the resolution is of correct type
- automatic and ensured autocompletion in IDE
- easy refactoring of the code and finding usage points

Cons:
- longer signature

Till next time, stay sexy and hydrated.


Self calling resolve function with return type hint

One more Christmas gift is waiting to become part of the collection.

Hogwarts™ Express train from King’s Cross Station is an iconic scene with Platform 9¾™ and a Dementor™ as a bonus. It's like getting a glow-in-the-dark ghost for the first time back in the era of no internet.

I hope my "lego writers' block" ends soon, as I need a couple more sets in my collection. And some lego fun too.


harry-potter-express - Hogwarts™ Express

harry-potter-express - Hogwarts™ Express - pending

# Volkswagen T2 Camper Van pending

Note: still waiting in the box

Since the first look at it, I was amazed at how far lego came with the details. I wanted to have it right away and am still surprised I waited for a couple of months before buying it myself.

Now it has some other sentimental values, and I can't start to bring it to life.

Fingers crossed, it happens soon, as I need new ideas and wishes to fulfill.


Volkswagen T2 Camper Van - pending

Planned days over the week did not go as planned, so the Saturday morning before Christmas, when we should all get some sleep and be rested for the next day, my body wanted its party at 5 in the morning.

Taking the opportunity for myself, I also made it an easy morning as I wanted to go on the road with a takeaway coffee available only from 6 onwards.

Roadtrip with freshly baked croissants and warm coffee, with a Tolkien third LOTR book in the background, was done in like 10 min.

...

Ski slopes again, remembering all the fun and relaxed feeling I had during prev season, the smile on my face that did not go away even if I wanted. I almost ran towards the area, where I finished gearing up and stepped into the board. "Take it easy," I said, and skipped the first slope and took the easy path to see how the board behaved.

No turning, gripping the snow at the edges, slow motion, and the first fall after just a couple of meters. "This is gonna be a fun season" rang in my mind, which continued with walking towards the chairlift, as the "easy path" was not connected to the desired destination.

Man, it took me two or three rides to properly handle the carvings where my body and gravitation started to do their work instead of my clumsy forcing the rotation.

After that, the day became as beautiful as I remember it from the last season. Even the scare of speed became manageable fun, and the freedom feeling I was craving was back again.

With each lift, I looked at small bumps where I could make my first jump of the season and begin my trip towards proper jumps and 180 that I set for myself. Each ride brought more fun of riding, and another missed opportunity to do the first jump. A couple of hours later, it finally happened.

I stopped before a bump and released the break in a strange cramp of doing "the thing" I was shocked that the snow was not soft as expected but icy hard, which my body confirmed a second later.

And that is how my first jump became a fall.

I called it a successful day anyway, so I continued my day with LOTR in the background and more coffee for the next road trip I had planned for that day.

Till next time, stay sexy and hydrated.


Snowboarding again - on the slopes

For some time now, I have said yes to as many things my mind comes across.

So, in that spirit, I started snowboarding prev year again. I took a mentor and did some days on my own. The experience turned into one the most relaxing and easy things in my life, realizing that the struggles I had in the first years of boarding all came out of having the wrong boots, causing strange leg pains where I could never enjoy the whole day at a resort, blaming it on all on the stamina and simply not being good at it.

The first day on my own was full of strange and already forgotten feelings of excitement, almost triggering hardcore anxiety attacks. Starting with the first steep track, leading to the main chairlift, to using it with my board in my hands to avoid falling when going off of it or the scary feeling of height when being a couple of meters off the ground :shrug:

This time, all I did differently compared to the first 8-ish years was to stop and try to feel the body, the board, and the snow. After a couple of hours, my fun started to explode. Just one more ride, one more, only this one, till the chairlift ended and the resort started closing.

In the spirit of a prev winter success, I set myself a goal to learn how to jump and do the 180 by the end of this year's season.

As I knew that finding days to be out and on the snow was a spontaneous decision happening as we go, I wanted to be unrestricted by a single resort and renting availability. So I made a good excuse and a commitment to get out as much as possible for a hobby where computers are not involved. It was time to buy my gear.

Knowing way too little about the gear properties and the requirements, I turned to Extreme Vital store, where the enthusiast of snowboarding took time, from the current level to my future goals, spending way too much time on finding the boots and shaping them to my final needs up to suggestions where and how to get most out of the learning experience coming towards me.

Of course, I had to do the whole gear-up at home as well; who wouldn't :giggle:

Yes, that is me smiling, have you never seen a developer in your life?

I can't wait for the snow journey to start soon.

Till next time, stay sexy and hydrated.


Snowboarding again - a gift to me

## enabling it
In PhpStorm settings, go to Settings > Appearance & Behaviour > New UI and enable it. You may need to update to the latest version on PhpStorm first.

![Enable new ui](/media/bits/phpstorm-ui-2023-enable.png)

## new era
And now it looks like we are in a year after 2020 ...

Cute slim bars, with nice icons. Looks like clean design, compared to the default layout PhpStorm comes with. Aliasing and smooth borders gives this easy feeling of focus and no distraction. See it for yourself as no comments can paint the experience there.

![First impression](/media/bits/phpstorm-ui-2023-impression.png)

## bugs / issues
The only issue I see is with scrolling lagging from time to time. Enabling or disabling smooth scroll did not change a thing and I simply accepted it till the stable release comes out. 

## customize for no-distraction mode
I prefer to work in a no-distraction kind of work, where:
- making things clean
  - there are no file tabs, just fuzzy search
  - other bars are hidden by default, showing them with a shortcut "double click cmd"
  - when project window is hidden, all I see is a file editor
  - removing top-bar requires you to clean all the items from the bar
- browsing files with fuzzy search shortcuts only
  - open file
  - open class
  - recent files (game changer)
- browsing files manually
  - hiding the project window (quick open with a shortcut)
  - enabling "always select open file"
  - enabling "open files with single click"
- running commands shortcuts
  - starting a bash script, that runs dockers
  - tests
  - Laravel artisan commands
- other tools hidden in sidebars, not used as often
  - database connection / access
  - todo list
  - terminal (run mostly thru shortcut or commands)

## conclusion

![Editing a file](/media/bits/phpstorm-ui-2023-file-editing.png)
![With project window files list](/media/bits/phpstorm-ui-2023-with-project.png)

Looked nice at start, but I ended in the same clean way as with prev ui, so changing UI makes almost no difference to me.


PhpStorm UI 2023

# time to learn something new
I got sick of my overhead with the current workflow, local environment, hosting, servers, deployment services, server maintenance, multiple instances, and separating more robust servers from temporary ones. Etc.

## disclaimer
Most of my projects are just that and nothing else. So it's the playground to get the production experiences I can use at my primary job, upgrade my CV, and prepare for the next gig.
Even though we talk about personal projects, I am highly bothered that I can not simply jump on the idea I have at the moment but am pulled back by the simplest things like local environments, domains, hosting, build flows, testing, or otherwise temporary domains, certificates.
Other inspirations come from my primary job circumstances, where things are always exciting.

## unsubscribe from current SaaS
I do not have many subscriptions to services, but at the end of the month, I could pay for a couple of streaming services with that money. So for them to just be there, that is too much.
I want a full self-hosted or managed environment where I can spawn a new socket server or five, a custom docker registry, a couple of CI/CD workers, etc. And if / when things become resource-consuming, I can scale just hardware without paying the upgraded subscription.

## quick option to switch servers
Whenever I set a new project and assign the server I want the project to run on, it soon becomes an issue when the resources are needed somewhere else; I am removing the server or simply upgrading to a better one or with another provider.

## no manual scaling overhead
Related to switching servers, making things scale big or small in a short time is a huge burden and an overhead I want to make as simple as possible.

## single hosting dashboard
Related to SaaS + multiple servers, multiple clients, ..., things soon become a bureaucracy what is where. I want to have it all in one place.

## simple management of application
Encapsulate the application for real, try things out, and don't worry about other things you also host. For example, adding one should be a matter of a couple of lines, not 15 screens and eight wizards.

## zero downtime for the applications
In the old world, we uploaded the new files of the application and hoped for the best. Usually, turning the maintenance page on or trying some symlink magic to achieve zero downtimes. Unfortunately, production rollbacks are still a mess and a full-blown risk.
There must be a better world somewhere today.

## multiple environments for features testing and previews
Having a test environment is already a privilege not all developers have. Pre-production and staging are almost like spam mail from a "prince that wants to give you money." What about a situation where the product is enormous, and each feature testing needs its environment? Or when you start developing on some huge refactor or new core functionality like replacing the whole internal system for events, sockets, etc.? What if a client also wants to have its sandbox/demo to play? Etc.

# the plan
Let's jump into the topic and see how far I can get before admitting all the mistakes I have made in the name of "Why should I check the documentation that takes 15 minutes if I can do that on my own in 5 hours."

Till next time, stay sexy and hydrated.



Matevž Novak