title	description
Monorepo for the Joel Mühlena internet presence	Readme for joel.muehlena.de Monorepo

This repository should unite all existing projects as well as new projects. On the one hand so that I have a better overview and on the other hand so that others have a better overview. Also I wanted to deal with Bazel a little bit.

[[TOC]]

Where can I see the whole thing live?

My Homepage

Info page with git and build related stuff

TODO

Who am I

My name is Joel Mühlena. I'm 21 years old and currently studying computer science in my fourth semester at the university of darmstadt (h_da). I like to program and am interested in (for me) new technologies and concepts. Since high school I'm running my own website on a Bare Metal Kubernetes cluster at Hetzner Online. Because of the microservice architecture I have accumulated some repositories and it is getting more and more annoying with the maintenance and development my next step is the migration to a monorepo. However, since I am usually very busy at the university and in my job, this project will probably take some time.

Reminder for myself

The postgres connection as well as the config server fetch is somehow affected by the default wsl resolv.conf nameserver. It has to be disabled and set to any e.g. 8.8.8.8 working DNS server instead

pgx version needs further investigation. Currently v5.2.0 is used. However the latest (at the moment 18/02/23) is 5.3.0 but this version does not work with Bazel because the internal BUILD.bazel file is not generated correctly. The issue is located in the internal/nbconn nbconn_real_non_block.go module of pgx.

All handlers temporarily are restricted to only accept application/json for the methods POST and PATCH because gin currently does not support types like []byte or any special custom type for binding especially with form binding. Due to this restriction it is not possible to bind UUIDs. However json works because if gets delegated to the json marshal / unmarshal interfaces. https://github.com/gin-gonic/gin/pull/3045 could fix this becuase it would allow gin to use encoding.TextUnmarshaler.

TODO

• Add header middleware to gomstoolkit handler chain (svc can control auth by themselves then) -> use authorization chain
• comment on handlers (go svces) which permission should be required for quick overview
• improve logging (error o.ä) -> clean and not duplicated
• add audit logging
• clean up architecture
  • move handler content into use case package
  • use grpc with the new use cases for inter service communication
  • switch to repository pattern
  • abstract gomstoolkit cache with interface
  • use tx everywhere to perform rollbacks on error (edit: only where required. Query & Exec should be preferred because it will not commit on DB errors anyways and is easier to handle then tx)
  • use exec not query for pgx where possible
• add UserService to ds-auth bzw. Iam with default one for own db, but allow external api
• implement config for gomstoolkit
• add grpc support for gomstoolkit
• add ws support for gomstoolkit
  • server
  • client
• Add metrics lib for Prometheus metrics
• Add opentelentry lib for telemetry
  • Add traces in sub fns like uc, and repo
  • Add traces to mongodb
  • Optional add traces to pgx -> lib?
• Add google uuid support to pgx
• Add google uuid support to mongodb
• rename shared/ to lib/
• Add jaeger for telemetry
• add fallback exporter for error cases (console, or file, or just empty)
• Add exec to pgx pool if
• Move common errors to lib
• Move to go/sqlc for sql code gen from RAW sql and save a lot of time and headache for pg repos (use pgx anyways)
• Use the following part to return json field names instead of go field names (edit: not working 1:1 paste currently) - use custom error for validation
• Replace casbin with own implementation -> more dynamic and not that bloated
• Redesign auth service to follow SSO pattern -> like OAuth2 PKCE flow (check playground for more info)
• add serf & raft to api-gateway to run distributed
• pnpm workspaces vscode integrations somehow not working -> use yarn (with node_modules) for now pnp => (yarn dlx @yarnpkg/sdks vscode)
• use k8s gateway-api instead plain networking ingress
• Replace UUID with lexically sortable UUIDs
• Use https://skeletonreact.com/ for skeleton loading

validate.RegisterTagNameFunc(func(fld reflect.StructField) string {
    name := strings.SplitN(fld.Tag.Get("json"), ",", 2)[0]
    if name == "-" {
        return ""
    }
    return name
})

Roadmap (currently in progress)

• [~] PostgreSQL data replication (logical replication) for user data not possible because I have no superuser role on the db. Use gRPC to fetch Cross service info because this is more microservice like

• Author Service 3

• User service 2

• Post service 4

• Config server

• role service 1

• Auth service V2 7

• API-Gateway V2 5

• Blog frontend refactoring

• Public sections

• Private sections

• Home header animation

• NPM config server addon

• Nodejs ms tools (express server tool)

• source common stuff to libs

  • logging
  • http server (pass http handler and return server) for http, tls, mtls and use in gomstoolkit and api gw

• config Server addon periodically pull config changes

• about svc

• CI pipeline adjustments for node.js

• education service

• skill service

• experiences service

• projects service

• cv service

• message service

• upload service

• web frontend refactoring

----------------------------------> Do an architecture review and improvement

• refactor config

  • api-gw
  • config-server
  • go-ms-toolkit
  • auth server
  • telementry

• Add tests for recently added services without tests (coverage 80 - 90 % is the goal)

• CI Pipeline -> build, test, docker push (docker auth problem in ci pipeline works local)

• Helm charts

• Setup mTLS with smallstep/autocert

• migrate to https / http2

• services

• traeffik

• Add Test certs for local running

• Create correct Postgres users for blog shared db

--> break and redeploy blog stuff with new and refactored services

• admin client

• refactor go svcs to use other package scheme (handlers/cmd/operations)

• update deps & bazel

• terraform scripts

• Hetzner K8s

• move static to astro.js -> started

  • combine astro with other ressources e.g. Swagger, Coverage, go package doc, ng docs, react docs etc.

• redeployment and reconfigure prometheus and grafana

--> Very optional if time and motivation is there

• add module tests (mock all) -> robot & go with eg httptest or Websocket server
• add integration tests -> robot

Get chainguard digests

IMAGE_NAME=chainguard/static
IMAGE_TAG=latest

tok=$(curl "https://cgr.dev/token?scope=repository:${IMAGE_NAME}:pull" \
  | jq -r .token)

curl -H "Authorization: Bearer $tok" \
  https://cgr.dev/v2/$IMAGE_NAME/_chainguard/history/$IMAGE_TAG | jq

Open questions

Here are currently open questions noted.

How to work with cross service requests

This question is about how to handle the single responsibility design. For example there are the post, tag and category service. But if I create a post I maybe need to create the tags and categories as well. The question is how to handle an error in the request chain during post creation. My current approach would be to pass an error to the calling client but do not touch the category or tag (if created) in any way because they could be referenced by another post. Because POST operations are not idempotent I think this behavior would be appropriate.

Workspace Settings

To resolve the go modules and imports correctly I'm using gopackagesdriver of rulesgo Github of Setup. The script is located under _tools/create_bazelisk_ci_image.sh ud called by the settings.json in .vscode. Because ${workspaceFolderBasename} is somehow not working at the moment the bazel folder name is hardcoded. Will be fixed if a fix is available or I found a suitable workaround.

For a simpler go dependency management create a go workspace at root level with go work init. Then you can add all go projects automatically by running go work use -r ./. For upgrading the go deps with gazelle then run bazel run //:update_go_deps

Status

Currently the project is in the migration phase. That means I'm trying to move my complete codebase here.

Migration

Here is documented what I still plan to do and what I have already migrated. In addition there are some additional infos to the projects.

Api Migration

• api-gateway (go)
• config server (go)
• request logger (requires kafka setup first, so pending) (node.js, ts)

Base API

• about-svc (node.js, ts)
• education-svc (node.js, ts)
• experiences-svc (node.js, ts)
• skill-svc (node.js, ts)
• message-svc (node.js, needs adjustments -> rewrite in go)
• uploads-svc (under development)
• user-svc (under development)
• auth-svc (node.js, ts)
• role-svc (not yet created)
• projects-svc (node.js, ts)
• cv-svc (node.js, js)

Blog API

• [~] category-svc (under development, go)
• post-svc
• comment-svc
• [~] tag-svc
• author-svc

Frontend Migration

• JM Web-Frontend (React, yarn)
• Blog Frontend (React, yarn)
• Admin Client (Angular, yarn)
• Request-Logger Client (React, yarn)

Tools migration

• npm config server addon - requires improvements and refactoring
• go config server addon (go)
• go microservice tools (under development, go)
• Create ansible scripts for setting up and upgrading k8s cluster nodes
• Create terraform scripts to setup new Hetzner servers for cloud

Goals

Deploy to K8s with combined Helm charts and create an API documentation with Swagger/OpenAPI in separate folders. Maybe need to create custom Bazel rules for that.

Outlook

At the end of the migration all frontend clients, api parts and other stuff should be here.

Bazel

For the management of the monrepo I use https://bazel.build/. On the one hand because I want to deal with this tool a little, on the other hand because here several programming languages and concepts come together.

Usually my approach is like this: I create the binary of the application if necessary. Then I run the tests. If the tests are all successful I create a docker image from the files.

General tool

• [https://github.com/bazelbuild/bazel-skylib](Bazel Skylib)
• [https://github.com/bazelbuild/rules_docker](Docker rules)

To copy certain files I use the bazel filegroup rule together with the skylib rules (copy_file)

Go tools

• [https://github.com/bazelbuild/rules_go](Go rules)
• https://github.com/bazelbuild/bazel-gazelle

Web/JS Tools

• [https://github.com/bazelbuild/rules_nodejs](JavaScript Rules)
• [https://github.com/bazelbuild/rules_sass](Sass Rules)

Programming languages

• Go
• JavaScript / TypeScript

Frameworks

• ReactJS
• Angular
• Express.js
• GinGonic

Infrastructure

TODO

What is where?

.
├── shared/ Content which is shared over multiple applications
│   ├── keys/ - Shared keys e.g. public keys used by multiple applications
│   ├── sql/ - Shared sql scripts and functions which are used in multiple databases or for multiple applications
│   └── sass/ - Shared styles
├── static/ - Content for gitlab pages which will be merged with other bazel outputs to show some info
├── tools/ - Tool for this repo e.g. creating the bazelisk image for CI
├── utility/ - Utility applications and libs
└── bzl/ - Custom bazel rules

bazel run @rules_go//go -- clean -modcache to clean the go module cache if something is not working as expected.

License

TODO