some metrics were missing when vm-storage crashes

[jinlongwang]

Describe the bug

we found vm-storage node crashed at 2024-05-16 12:32, and then we found that many metrics were missing.
We were pretty sure that these metrics existed before, and we found them in a different store（hive）.

for example:

we found that all the missing metrics are hashed to the same node, which is the node where the panic occurred

vm-storage panic logs

thanks for helping

To Reproduce

This problem has occurred twice in our production environment

sorry we have no way to reproduce this problem so far

Version

vmstorage-20230313-063739-tags-v1.87.5-cluster-0-g4793665c4

we build this version from cluster-v1.87.5 (there is no code changed)

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

No response

[hagen1778]

This problem has occurred twice in our production environment

Hey @jinlongwang! Do you know why the node has crashed? Have you tried following these recommendations https://docs.victoriametrics.com/troubleshooting/#unexpected-query-results ?

vmstorage-20230313-063739-tags-v1.87.5-cluster-0-g4793665c4

This is pretty old version. A lot has changed in indexdb and in the way of storing data. I'd recommend updating to the usptream version. Otherwise, it is unlikely I can help besides pointing to troubleshooting docs. In general, runtime crashes shouldn't corrupt indexdb (data structure which helps to locate time series) and affects searchability of time series. It is unexpected that you lost the data.