Grafana shows metrics value after removal in victoria metrics

[tonus01]

Is your question request related to a specific component?

VictroiaMetrics with grafana

Describe the question in detail

Hi. I use victoria metrics with promethues scrape config. After I changed job name(from servs to hosts) grafana started showing 2 jobs(old and new).

Then I deleted the time series for the servs job.
http://victoria-metrics:8428/api/v1/admin/tsdb/delete_series?match[]={job="servs"}
And victoria metrics explorer return me only 1 job(hosts). But grafana still return 2 jobs(hosts and servs). Why does grafana return something that doesn't exist?

Troubleshooting docs

• General - https://docs.victoriametrics.com/troubleshooting/
• vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[tonus01]

This issue is rise when i choose prometheus version >= 2.24.x When i choose 2.23.x my grafana work fine and not show removed time series.

[hagen1778]

Why does grafana return something that doesn't exist?

It is likely Grafana caches the previous response. Will it show the old job value if you create a new datasource with identical settings?

[tonus01]

Why does grafana return something that doesn't exist?

It is likely Grafana caches the previous response. Will it show the old job value if you create a new datasource with identical settings?

Yes, grafana show old job if i create new datasource with identical settings. And i use cache level: none for datasource to avoid cache influence. I try last releases grafana(10.4.{1-3} and 11.0.0) and this problem is everywhere. I use vm single install(try v1.101.0, v1.100.1,v1.93.14) Can you ask QA team to check this? And I repeat, everything works correctly if i select prometheus verison 2.23.X or less in datasource settings.