How do I protect my services

My portal lists the technologies I use in my domain. Some like phpMyAdmin are designed with their own security features while others like Longhorn are unable to do even basic authentication despite they manage critical components. So how did I proceed to secure my environment ?

Statement of needs

In a few posts like this one and this one, I explained how critical it is to complete a statement of needs and have the right answers when doing one. So here are the needs I identified for my case :

• A standard solution for everything running over HTTP
• The application itself should be protected even against 0-day and known vulnerabilities attempted by outsiders
• A solution that will ensure authentication, integrity and confidentiality over the network
• A security control that is strong enough to endure the typical attacks including brute force, stolen passwords and network interception (read-only and read-write)
• Easy to deploy and operate

In a previous version of my setup, I used mTLS for that. mTLS, or Mutual TLS, is when the client requires the server a valid TLS certificate just like for any HTTPS site but the server also requires the client a similar TLS certificate before going any further. Thanks to that, both client and server are authenticated. mTLS also makes TLS impossible to decrypt by web filtering solutions deployed in infrastructure, either by enterprises, public network providers or attack devices like stingrays. Another benefit is that not a single packet will make it to the application before the mTLS handshake succeeds.

That solution was working great but was not perfect. The main problem was to manage the client certificates. To deploy them, to exchange them securely with people outside, some technologies were not able to use them like non-Safari browsers on iPad and more.

As such, I refined my statement of needs looking for a new solution :

• Easy to deploy and operate
  • Avoid client certificates
    • Without the client certificate, TLS offers a real security but is not unbreakable as it was with mTLS. As such, the new solution must protect Authentication itself, without relying exclusively on TLS for that.
  • Avoid wildcard certificates
    • Without a wildcard certificate, the solution needs to manage many standalone certificates automatically.

From these needs, I designed my new solution :

• Cert-Manager will manage every service's certificate automatically
• Keycloak will be used for client authentication, enforcing either TOTP or WebAuthN (Passkeys ; password-less authentication) to protect authentication against typical attacks and even compromised components.
• Keycloak will also implement SSO so users do not have to re-authenticate time after time.
• OAuth2-Proxy to protect each TCP socket and requiring the strong authentication before forwarding anything to a protected application

How it's done here

Here is the configuration for one of these OAuth2-Proxy protected deployment :

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: echoserver
  name: echoserver
  namespace: prod-services
spec:
  replicas: 2
  selector:
    matchLabels:
      app: "echoserver"
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - image: gcr.io/google_containers/echoserver:1.9
        imagePullPolicy: IfNotPresent
        name: echoserver
      - name: oap-echoserver
        image: "quay.io/oauth2-proxy/oauth2-proxy:v7.6.0"
        env:
  [...]
        - name: OAUTH2_PROXY_CODE_CHALLENGE_METHOD
          value: S256
        - name: OAUTH2_PROXY_PROVIDER
          value: oidc
        - name: OAUTH2_PROXY_UPSTREAMS
          value: "http://127.0.0.1:8080/"
        - name: OAUTH2_PROXY_PROVIDER_DISPLAY_NAME
          value: "Keycloak JBLan"
        - name: OAUTH2_PROXY_EMAIL_DOMAINS
          value: "*"
        - name: OAUTH2_PROXY_ALLOWED_GROUPS
          value: "Clients"
        - name: OAUTH2_PROXY_COOKIE_DOMAINS
          value: "echo.jblan.ca"
        - name: OAUTH2_PROXY_WHITELIST_DOMAINS
          value: "echo.jblan.ca"
        - name: OAUTH2_PROXY_SCOPE
          value: "email openid profile"
        - name: OAUTH2_PROXY_HTTP_ADDRESS
          value: "0.0.0.0:4180"
        - name: OAUTH2_PROXY_CLIENT_ID # keep this name - it's required to be defined like this by OAuth2 Proxy
          valueFrom:
            secretKeyRef:
              name: oap-echo-secret
              key: client-id
        - name: OAUTH2_PROXY_CLIENT_SECRET # keep this name - it's required to be defined like this by OAuth2 Proxy
          valueFrom:
            secretKeyRef:
              name: oap-echo-secret
              key: client-secret
        - name: OAUTH2_PROXY_COOKIE_SECRET # keep this name - it's required to be defined like this by OAuth2 Proxy
          valueFrom:
            secretKeyRef:
              name: oap-echo-secret
              key: cookie-secret
        ports:
        - containerPort: 4180

The environment variable about the upstream server shows that once authenticated, the client is served from a server listening at 127.0.0.1. Obviously, there is no way for a client to reach that service other than going through the proxy. Access control is also enforced by another variable. Not only authentication must be completed, the user must be member of the group "Clients". For critical services like Longhorn, required group is Administrators.

This approach reduces the attack surface drastically. No matter the service to protect is Portainer, Longhorn, Grafana or anything else, none of them can be attacked before a successful authentication. The only piece of software that is reachable is OAuth2-Proxy itself, which is very minimalistic. It is easy to do threat intelligence to detect if a vulnerability against it is identified and it will be easy to apply the update when needed.

Authentication is strong and will survive a compromised channel thanks to either TOTP or Passkeys.

Encryption and integrity in transit are also very strong thanks to TLS 1.3 and a strict configuration of allowed ciphers. Because JBLan.ca is limited to client facing services, I also preloaded HSTS for it. Should I wish to double check TLS and be sure that it was not intercepted, I just need to validate that the server's certificate is the one from Lets Encrypt.

Certificates are created and managed automatically. Users can create their own accounts themselves. By default, these accounts are created as Clients, so can use a service like the Echo-Server illustrated above. Clients can also re-use one of their existing accounts if they wish instead of creating yet another account.

Thanks to well defined needs, I was able to produce a solution that answers all of them. My setup is properly secured, easy to use and completely functional. When security is part of the needs and design, it is not an obstacle to functionality. It becomes such an obstacle only when the entire solution is designed first and security is added as an after-thought.