PromQL in Google Cloud Monitoring produces unexpected results

Question

Disclaimer: I'm new to PromQL, and might have constructed a query that is completely wrong. I'm using PromQL as the built-in monitoring of Google Cloud does not produce expected results (see this post).

When I use PromQL for a custom metric in Google Cloud I'm facing several issues:

• my query generates too many data points (most likely something wrong with the query below?)
• when I zoom out, all data disappears, or some aggregates are gone. The counts go from 1k to 50 for some reason
• when I use a 1m interval, the data disappears
• it is unclear how user-based distribution metrics should be queried, the docs are missing examples (it seems 3 submetrics are generated: bucket, count and sum, but I have no clue on how to use them)

I mainly want to have a sum of log-based metrics that contains a number of API hits like:

hits: 14
hits: 56
...

an interval should just show the sum of these numbers.

The query:

sum by (dag_id) (
sum_over_time(logging_googleapis_com:user_metric_sum[30m])
)

Example

Zoomed in version:

Data is gone when zooming out:

Multiple data points when zoomed in:

Sums are wrong when zooming out (note the y-axis changing):