Add FAQ/troubleshooting: metrics discrepancies + loki timeout (#663)

* Add FAQ/troubleshooting: metrics discrepancies

* new FAQ item: loki timeout; + update old stuff

* update toc

Author: jotak

FAQ.md

@@ -8,15 +8,15 @@ If you can't find help here, don't hesitate to open [an issue](https://github.co
   * [Is it for OpenShift only?](#is-it-for-openshift-only)
   * [Which version of Kubernetes / OpenShift is supported?](#which-version-of-kubernetes--openshift-is-supported)
 * How-to
-  * [To run the eBPF agent](#to-run-the-ebpf-agent)
-  * [To use IPFIX exports](#to-use-ipfix-exports)
-  * [To get the OpenShift Console plugin](#to-get-the-openshift-console-plugin)
+  * [How do I visualize flows and metrics?](#how-do-i-visualize-flows-and-metrics)
   * [How can I make sure everything is correctly deployed?](#how-can-i-make-sure-everything-is-correctly-deployed)
 * Troubleshooting
   * [Everything seems correctly deployed but there isn't any flow showing up](#everything-seems-correctly-deployed-but-there-isnt-any-flow-showing-up)
   * [There is no Network Traffic menu entry in OpenShift Console](#there-is-no-network-traffic-menu-entry-in-openshift-console)
   * [I first deployed flowcollector, and then kafka. Flowlogs-pipeline is not consuming any flow from Kafka](#i-first-deployed-flowcollector-and-then-kafka-flowlogs-pipeline-is-not-consuming-any-flow-from-kafka)
+  * [I get a Loki error / timeout, when trying to run a large query, such as querying for the last month of data](#i-get-a-loki-error--timeout-when-trying-to-run-a-large-query-such-as-querying-for-the-last-month-of-data)
   * [I don't see flows from either the `br-int` or `br-ex` interfaces](#i-dont-see-flows-from-either-the-br-int-or-br-ex-interfaces)
+  * [I'm finding discrepancies in metrics](#im-finding-discrepancies-in-metrics)
 
 ## Q&A
 
@@ -28,25 +28,28 @@ And if something is not working as hoped with your setup, you are welcome to con
 
 ### Which version of Kubernetes / OpenShift is supported?
 
-It depends on which `agent` you want to use: `ebpf` or `ipfix`, and whether you want to get the OpenShift Console plugin.
+All versions of Kubernetes since 1.22 should work, although there is no official support (best effort).
 
-## How to
+All versions of OpenShift currently supported by Red Hat are supported. Older version, greater than 4.10, should also work although not being officially supported (best effort).
 
-### To run the eBPF agent
+Some features depend on the Linux kernel version in use. It should be at least 4.18 (earlier versions have never been tested). More recent kernels (> 5.14) are better, for agent feature completeness and improved performances.
 
-What matters is the version of the Linux kernel: 4.18 or more is supported. Earlier versions are not tested.
+### How do I visualize flows and metrics?
 
-Other than that, there are no known restrictions on the Kubernetes version.
+For OpenShift users, a visualization tool is integrated in the OpenShift console. Just open the console in your browser, and you will see new menu items (such as Network Traffic under Observe) once NetObserv is installed and configured.
 
-### To use CNI's IPFIX exports
+Without OpenShift, you can still access the data (Loki logs, Prometheus metrics) in different ways:
 
-This feature has been deprecated and is not available anymore. Flows are now always generated by the eBPF agent.
+- Querying Loki (or Prometheus) directly
+- Using the Prometheus console
+- Using and configuring Grafana
 
-Note that NetObserv itself is still able to export its enriched flows as IPFIX: that can be done by configuring `spec.exporters`.
+All these options depend on how you installed these components.
 
-### To get the OpenShift Console plugin
-
-OpenShift 4.10 or above is required.
+If you feel ready for hacking, there is also a way to view the Test Console, used by the development team, which is similar to the OpenShift console plugin and can work without OpenShift. You need to:
+- Build the console plugin in "standalone" mode: https://github.com/netobserv/network-observability-console-plugin?tab=readme-ov-file#standalone-frontend (you can just build the image, no need to run it locally).
+- Configure the Operator to use this build: `kubectl set env deployment/netobserv-controller-manager -c "manager" RELATED_IMAGE_CONSOLE_PLUGIN="<your build image here>"`
+- Configure the Operator to deploy the Test Console: in `FlowCollector` yaml, set `spec.consolePlugin.advanced.env.TEST_CONSOLE` to `true`.
 
 ### How can I make sure everything is correctly deployed?
 
@@ -174,6 +177,22 @@ This is a [known bug](https://github.com/segmentio/kafka-go/issues/1044) in one
 
 Please recreate the flowlogs-pipeline pods by either killing them maunally or deleting and recreating the flow collector object.
 
+### I get a Loki error / timeout, when trying to run a large query, such as querying for the last month of data
+
+There are several ways to mitigate this issue, although there is no silver bullet. As a rule of thumb, be aware that Prometheus is a better fit than Loki to query on large time ranges.
+
+With Loki queries, a first thing to understand is that, while Loki allows to query both on indexed and non-indexed fields (aka. labels), **queries that contain filters on labels will perform much better**. So, perhaps you can adapt your query to add an indexed filter. For instance if you were querying for a particular Pod (this isn't indexed), add its Namespace to the query. The list of indexed fields [is documented here](https://docs.openshift.com/container-platform/4.15/observability/network_observability/json-flows-format-reference.html#network-observability-flows-format_json_reference), in the `Loki label` column.
+
+Depending on what you are trying to get, you may as well **consider querying Prometheus rather than Loki**. Queries on Prometheus are much faster than on Loki, it should not struggle with large time ranges, hence should be favored whenever possible. But Prometheus metrics do not contain as much information as flow logs in Loki, so whether or not you can do that really depends on the use case. When you use the NetObserv console plugin, it will try automatically to favor Prometheus over Loki if the query is compatible; else it falls back to Loki. If your query does't run against Prometheus, changing some filters or aggregations can make the switch. In the console plugin, you can force the use of Prometheus. Incompatible queries will fail, and the error message displayed should help you figure out which labels you can try to change to make the query compatible (for instance, changing a filter or an aggregation from Resource/Pods to Owner).
+
+If the data that you need isn't available as a Prometheus metric, you may also **consider using the [FlowMetrics API](https://github.com/netobserv/network-observability-operator/blob/main/docs/Metrics.md#custom-metrics-using-the-flowmetrics-api)** to create your own metric. You need to be careful about the metrics cardinality, as explained in this link.
+
+If the problem persists, there are ways to **configure Loki to improve the query performance**. Some options depend on the installation mode you used for Loki (using the Operator and `LokiStack`, or `Monolithic` mode, or `Microservices` mode):
+
+- In `LokiStack` or `Microservices` modes, try [increasing the number of querier replicas](https://loki-operator.dev/docs/api.md/#loki-grafana-com-v1-LokiComponentSpec)
+- Increase the [query timeout](https://loki-operator.dev/docs/api.md/#loki-grafana-com-v1-QueryLimitSpec). You will also need to increase NetObserv read timeout to Loki accordingly, in `FlowCollector` `spec.loki.readTimeout`.
+
+
 ### I don't see flows from either the `br-int` or `br-ex` interfaces
 
 [`br-ex` and `br-int` are virtual bridge devices](https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/networking_guide/bridge-mappings),
@@ -183,3 +202,19 @@ by the agent when it is processed by other interfaces (e.g. physical host or vir
 
 This means that, if you restrict the agent interfaces (using the `interfaces` or `excludeInterfaces`
 properties) to attach only to `br-int` and/or `br-ex`, you won't be able to see any flow.
+
+### I'm finding discrepancies in metrics
+
+1. NetObserv metrics (such as `netobserv_workload_ingress_bytes_total`) show *higher values* than cadvisor metrics (such as `container_network_receive_bytes_total`)
+
+This can be caused when traffic goes through Kubernetes Services: when a Pod talks to another Pod via a Service, two flows are generated: one against the service and one against the pod. To avoid querying duplicated counts, you can refine your promQL to ignore traffic targeting Services: e.g: `sum(rate(netobserv_workload_ingress_bytes_total{DstK8S_Namespace="my-namespace",SrcK8S_Type!="Service",DstK8S_Type!="Service"}[2m]))`
+	
+2. NetObserv metrics (such as `netobserv_workload_ingress_bytes_total`) show *lower values* than cadvisor metrics (such as `container_network_receive_bytes_total`)
+
+There are several possible causes:
+
+- Sampling is being used. Check your `FlowCollector` `spec.agent.ebpf.sampling`: a value greater than 1 means not every flows are sampled. NetObserv metrics aren't normalized automatically, but you can do so in your promQL by multiplying with the sampling rate, for instance: `sum(rate(netobserv_workload_ingress_bytes_total{DstK8S_Namespace="my-namespace"}[2m])) * avg(netobserv_agent_sampling_rate > 0)`. Be aware that, the higher the sampling, the less accurate the metrics.
+
+- Filters are configured in the agent, resulting in ignoring some of the traffic. Check your `FlowCollector` `spec.agent.ebpf.flowFilter`, `spec.agent.ebpf.interfaces`, `spec.agent.ebpf.excludeInterfaces` and make sure it doesn't filter out some of the traffic that you are looking at.
+
+- Flows may also be dropped due to constraints on resources. Monitor the eBPF agent health in the `NetObserv / Health` dashboard: there are graphs showing drops. Increasing `spec.agent.ebpf.cacheMaxSize` can help to avoid these drops, at the cost of an increased memory usage.