Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vmagent keeps retrying and destroys vminsert and entrance #5486

Open
1 of 3 tasks
perfect-duo opened this issue Dec 19, 2023 · 1 comment · May be fixed by #6289
Open
1 of 3 tasks

vmagent keeps retrying and destroys vminsert and entrance #5486

perfect-duo opened this issue Dec 19, 2023 · 1 comment · May be fixed by #6289
Assignees
@dmitryk-dk
Labels
question The question issue

Comments

@perfect-duo
Copy link

Is your question request related to a specific component?

vmagent vminsert

Describe the question in detail

2023-12-19T07:43:29.157Z warn VictoriaMetrics/app/vmagent/remotewrite/client.go:385 couldn't send a block with size 70070 bytes to "1:https://xxxxx/insert/236/prometheus": Post "https://xxxx/insert/236/prometheus": context deadline exceeded (Client.Timeout exceeded while awaiting headers); re-sending the block in 2.000 seconds

There is a problem with vminsert configuration. Therefore, vmagent keeps failing to send data if there are too many vmagent nodes. This will lead to constant retries, which in turn increases the pressure on our SLB entrance and vminsert. So is there a way to control the retry interval of vmagent?

Troubleshooting docs
@perfect-duo perfect-duo added the question The question issue label Dec 19, 2023
@dmitryk-dk
Copy link
Contributor

dmitryk-dk commented Dec 19, 2023 •
edited

Hi @perfect-duo ! This values is limited inside the function and growth twice per each error, but can't be more than 1 minute

retryDuration := time.Second
	retriesCount := 0

again:
	startTime := time.Now()
	resp, err := c.doRequest(c.remoteWriteURL, block)
	c.requestDuration.UpdateDuration(startTime)
	if err != nil {
		c.errorsCount.Inc()
		retryDuration *= 2
		if retryDuration > time.Minute {
			retryDuration = time.Minute
		}
		logger.Warnf("couldn't send a block with size %d bytes to %q: %s; re-sending the block in %.3f seconds",
			len(block), c.sanitizedURL, err, retryDuration.Seconds())
		t := timerpool.Get(retryDuration)

So, at that time, it can't be set via any flag.
If you need this flag please create a feature request and describe your problem and team will decide should it be implemented

@yorik yorik linked a pull request May 16, 2024 that will close this issue
Open
1 task
@AndrewChubatiuk AndrewChubatiuk mentioned this issue May 27, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmitryk-dk dmitryk-dk

Labels
question The question issue
Projects
None yet
Milestone
No milestone
2 participants
@dmitryk-dk @perfect-duo