Black lives matter.

We stand in solidarity with the Black community.

Racism is unacceptable.

It conflicts with the core values of the Kubernetes project and our community does not tolerate it.

Grafana Plugin (grafana/v1-alpha)

The Grafana plugin is an optional plugin that can be used to scaffold Grafana Dashboards to allow you to check out the default metrics which are exported by projects using controller-runtime.

When to use it ?

How to use it ?

Prerequisites:

Basic Usage

The Grafana plugin is attached to the init subcommand and the edit subcommand:

# Initialize a new project with grafana plugin
kubebuilder init --plugins grafana.kubebuilder.io/v1-alpha

# Enable grafana plugin to an existing project
kubebuilder edit --plugins grafana.kubebuilder.io/v1-alpha

The plugin will create a new directory and scaffold the JSON files under it (i.e. grafana/controller-runtime-metrics.json).

Show case:

See an example of how to use the plugin in your project:

output

Now, let’s check how to use the Grafana dashboards

  1. Copy the JSON file
  2. Visit <your-grafana-url>/dashboard/import to import a new dashboard.
  3. Paste the JSON content to Import via panel json, then press Load button Screen Shot 2022-06-28 at 3 40 22 AM
  4. Select the data source for Prometheus metrics Screen Shot 2022-06-28 at 3 41 26 AM
  5. Once the json is imported in Grafana, the dashboard is ready.

Grafana Dashboard

Controller Runtime Reconciliation total & errors

  • Metrics:
    • controller_runtime_reconcile_total
    • controller_runtime_reconcile_errors_total
  • Query:
    • sum(rate(controller_runtime_reconcile_total{job=”$job”}[5m])) by (instance, pod)
    • sum(rate(controller_runtime_reconcile_errors_total{job=”$job”}[5m])) by (instance, pod)
  • Description:
    • Per-second rate of total reconciliation as measured over the last 5 minutes
    • Per-second rate of reconciliation errors as measured over the last 5 minutes
  • Sample:

Controller CPU & Memory Usage

  • Metrics:
    • process_cpu_seconds_total
    • process_resident_memory_bytes
  • Query:
    • rate(process_cpu_seconds_total{job=”$job”, namespace=”$namespace”, pod=”$pod”}[5m]) * 100
    • process_resident_memory_bytes{job=”$job”, namespace=”$namespace”, pod=”$pod”}
  • Description:
    • Per-second rate of CPU usage as measured over the last 5 minutes
    • Allocated Memory for the running controller
  • Sample:

Seconds of P50/90/99 Items Stay in Work Queue

  • Metrics
    • workqueue_queue_duration_seconds_bucket
  • Query:
    • histogram_quantile(0.50, sum(rate(workqueue_queue_duration_seconds_bucket{job=”$job”, namespace=”$namespace”}[5m])) by (instance, name, le))
  • Description
    • Seconds an item stays in workqueue before being requested.
  • Sample:

Seconds of P50/90/99 Items Processed in Work Queue

  • Metrics
    • workqueue_work_duration_seconds_bucket
  • Query:
    • histogram_quantile(0.50, sum(rate(workqueue_work_duration_seconds_bucket{job=”$job”, namespace=”$namespace”}[5m])) by (instance, name, le))
  • Description
    • Seconds of processing an item from workqueue takes.
  • Sample:

Add Rate in Work Queue

  • Metrics
    • workqueue_adds_total
  • Query:
    • sum(rate(workqueue_adds_total{job=”$job”, namespace=”$namespace”}[5m])) by (instance, name)
  • Description
    • Per-second rate of items added to work queue
  • Sample:

Retries Rate in Work Queue

  • Metrics
    • workqueue_retries_total
  • Query:
    • sum(rate(workqueue_retries_total{job=”$job”, namespace=”$namespace”}[5m])) by (instance, name)
  • Description
    • Per-second rate of retries handled by workqueue
  • Sample:

Subcommands

The Grafana plugin implements the following subcommands:

  • edit ($ kubebuilder edit [OPTIONS])

  • init ($ kubebuilder init [OPTIONS])

Affected files

The following scaffolds will be created or updated by this plugin:

  • grafana/*.json

Further resources