Week 7 - Security, secrets, and deployment automation¶

During this week you will focus on improving the security and deployment practices of your application. This includes preventing common web vulnerabilities, managing secrets securely, and setting up automated deployment to a virtual machine environment.

Development requirements¶

Security becomes increasingly important as your application grows. The goal of this week is to ensure that your app adheres to secure coding standards and is safe to expose publicly.

Operational requirements¶

This week focuses on operational security, automated delivery and exporting metrics.

Application metrics (part 2)¶

In week 6 your backend started exposing metrics on /metrics: a counter, a gauge and a histogram. This week you run Grafana Alloy on the same VM. Alloy reads that endpoint once a minute and pushes the metrics to the course monitoring system, where your team can see them in Grafana.

Mandatory metrics¶

This phase the only mandatory metrics exposed by your application are:

• app_build_info with a label to show application version.
• Any metrics needed to solve the analysis tasks.

Before continuing, make sure your application publishes these metrics.

Install Alloy¶

Install Alloy from Grafana's package repository. Choose the tab for the OS your VM runs.

Configure Alloy to push metrics¶

Alloy's configuration is a small pipeline of components. Each component has a type (what it does) and a name you choose, and they are wired together with forward_to. Ours has two parts: scrape the backend and push everything out.

Replace the contents of /etc/alloy/config.alloy with the following, changing the team name and the push URL:

// 1. Your backend's metrics: the /metrics endpoint from last week
prometheus.scrape "backend" {
  targets = [{
    "__address__" = "localhost:9464", # Use the metrics port you defined for your application.
    "job"         = "backend",
    "instance"    = constants.hostname,
  }]
  metrics_path    = "/metrics" # Check that you are also using the correct path.
  scrape_interval = "60s"
  forward_to      = [prometheus.remote_write.course.receiver]
}

// 2. Push the metrics to the course monitoring system
prometheus.remote_write "course" {
  external_labels = {
    team = "<team>",   // CHANGE THIS to your team name.
  }
  endpoint {
    # Fill in your <team> name according to the reference above
    url = "https://metrics.kubernetes.devops.cs.ut.ee/<team>/prometheus/api/v1/write"
  }
}

If your backend exposes metrics on more than one process or port, add one entry per target to the targets list.

Check the syntax, then start Alloy and enable it on boot:

alloy fmt /etc/alloy/config.alloy > /dev/null && echo "config OK"
sudo systemctl enable --now alloy

After later changes to the config, apply them without a restart:

sudo systemctl reload alloy

Browse your application metrics on Grafana¶

Grafana is an open-source tool for visualising and exploring metrics, logs and traces. You connect it to data sources such as Prometheus or Victoria Metrics, then build dashboards and alerts on top of them. In this course, your team uses it to see the metrics Alloy pushes from your backend.

After gaining access, you can take a look, if your team's metrics are reaching Grafana.

For a quick overview navigate from the left hand menu to Drilldown -> Metrics. If you see the metrics you have defined there, then your metrics exporter is working correctly!

Feel free to explore Grafana further, as we will be using it in phases 3 and 4.

• Implement secrets detection in your CI/CD pipeline.

  • Secrets (API keys, passwords, tokens, etc.) must not appear in pipeline logs or environment variables printed to stdout/stderr.
  • Use your CI/CD platform’s secret management tools to store sensitive information.
  • Add automated checks to prevent accidental secret exposure in commits or build stages.

• If you have not done it already, set up automatic deployment to a virtual machine.

  • When a new version is pushed to the main branch (or another designated branch), it should trigger a deployment job.
  • The deployment must:
    • Transfer the new application build or container to the VM.
    • Restart or reload the service automatically.
  • Document how deployment is triggered and verified.

• Implement Grafana Alloy for exporting metrics.

Documentation requirements¶

Analysis tasks¶

Like in phase 1, you write two short analysis documents. Put them in docs/analysis/ and merge them to be a part of your main protected branch. There is no single correct answer to either of them: different teams can make different choices and still get full points. We grade the reasoning.

Analysis A: delivery metrics in phase 2¶

File: docs/analysis/phase2-delivery-metrics.md

In phase 1 you measured the four DORA metrics through proxy events in GitLab, because nothing was deployed. Now every change on main reaches a running system, so you can measure much closer to the real thing.

1. Extend your table. Copy your phase 1 table into the new file and fill in the Phase 2 definition column. Add these columns:

   Column	Content
   Signal and source	Which event or metric you use, and where it comes from: git, CI/CD, your application, or something else.
   Query or calculation	The PromQL query, API call, or calculation that turns the signal into the number.
   How it still differs from the real metric	One or two sentences: what your phase 2 definition still cannot see, and in which direction that biases the number. This is the graded column.

2. Implement what you can. Implement at least deployment frequency and lead time for changes as panels on your team dashboard, based on real deployments. If a metric needs data your system does not produce yet, add it: a metric in your application, a step in your pipeline, or anything else you can justify.

3. (Recommended) Run one failure drill. Deploy a change that passes your pipeline but breaks something users would notice. Then detect it and restore service. Write down the timeline (commit, deployment, detection, restore) and calculate change failure rate and time to restore using your own definitions. Then answer:

   • Did your dashboard show the failure? If not, what signal would have?
   • With one sample every 15 seconds, what is the shortest failure your metrics could have missed?

Analysis B: stack review after the VM deployment¶

File: docs/analysis/phase2-stack-review.md

In phase 1 you predicted what would be difficult when containerising your application. You have now deployed it to a real machine, which already exposes many of the same problems. Keep this document under two pages.

1. Revisit your predictions. For each difficulty predicted in phase1-stack.md: is it confirmed, refuted, or not tested yet? Give the evidence from your VM deployment in one or two sentences.
2. What you did not predict. Describe at least two problems you hit in phase 2 that you did not predict, and why you think you missed them.

3. Container readiness. Fill in this table, with one row for each of the prompts from phase 1: where state lives, what is hardcoded, how configuration is received, how many processes run and what starts them, what is written to disk at runtime, where logs go and in which format, cold start time, and what breaks if two backend copies run at once.

   Prompt	Current state on the VM	What has to change to run in a container	Effort (S / M / L)

   Be concrete: name the file path, the environment variable, the port, or the line of code. You will need this when migrating to Kubernetes.

You will come back to this table in phase 3.

Tasks¶

• Secure your application:
  • Prevent XSS vulnerabilities.
• Protect your secrets:
  • Implement secret detection and ensure secrets are not exposed in pipelines or logs.
• Automate your deployment:
  • Configure the pipeline to deploy the application automatically to a virtual machine.
• Update your documentation:
  • Include detailed explanations of the VM setup, deployment process, and any system changes from previous phases.