Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ETCD: linstor-proxmox can't connect after migrating from H2 to ETCD #366

Open
ggzengel opened this issue Aug 8, 2023 · 1 comment
Open

ETCD: linstor-proxmox can't connect after migrating from H2 to ETCD #366

ggzengel opened this issue Aug 8, 2023 · 1 comment

Comments

@ggzengel
Copy link

ggzengel commented Aug 8, 2023

Linstor and DRBD is fully working after migration (export-db/import-db) except proxmox can't access the linstor-controller.

pvestatd[7861]: could not connect to any LINSTOR controller at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 214

It looks like REST-API does not respond:

# wget 172.19.1.4:3370/health
--2023-08-08 17:04:22--  http://172.19.1.4:3370/health
Connecting to 172.19.1.4:3370... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2023-08-08 17:04:22 ERROR 500: Internal Server Error.

2023-08-08T17:04:09.953648+00:00 px4 Controller[1694726]: LINSTOR, Module Controller
2023-08-08T17:04:09.997696+00:00 px4 Controller[1694726]: Version:            1.24.0 (326986a4d1834d970e03f26594f3198af6ad47a6)
2023-08-08T17:04:09.997798+00:00 px4 Controller[1694726]: Build time:         2023-08-07T11:51:43+00:00
2023-08-08T17:04:09.997821+00:00 px4 Controller[1694726]: Java Version:       17
2023-08-08T17:04:09.997840+00:00 px4 Controller[1694726]: Java VM:            Debian, Version 17.0.8+7-Debian-1deb12u1
2023-08-08T17:04:09.997859+00:00 px4 Controller[1694726]: Operating system:   Linux, Version 6.2.16-6-pve
2023-08-08T17:04:09.997881+00:00 px4 Controller[1694726]: Environment:        amd64, 16 processors, 30688 MiB memory reserved for allocations
2023-08-08T17:04:09.997913+00:00 px4 Controller[1694726]: System components initialization in progress
2023-08-08T17:04:10.008121+00:00 px4 Controller[1694726]: WARNING: A terminally deprecated method in java.lang.System has been called
2023-08-08T17:04:10.008242+00:00 px4 Controller[1694726]: WARNING: System::setSecurityManager has been called by com.linbit.linstor.core.Controller (file:/usr/share/linstor-server/lib/controller-1.24.0.jar)
2023-08-08T17:04:10.008277+00:00 px4 Controller[1694726]: WARNING: Please consider reporting this to the maintainers of com.linbit.linstor.core.Controller
2023-08-08T17:04:10.008296+00:00 px4 Controller[1694726]: WARNING: System::setSecurityManager will be removed in a future release
2023-08-08T17:04:10.130876+00:00 px4 Controller[1694726]: Loading configuration file "/etc/linstor/linstor.toml"
2023-08-08T17:04:10.362399+00:00 px4 Controller[1694726]: 17:04:10,232 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.3.8
2023-08-08T17:04:10.362516+00:00 px4 Controller[1694726]: 17:04:10,238 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
2023-08-08T17:04:10.362552+00:00 px4 Controller[1694726]: 17:04:10,238 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/usr/share/linstor-server/lib/conf/logback.xml]
2023-08-08T17:04:10.362589+00:00 px4 Controller[1694726]: 17:04:10,337 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
2023-08-08T17:04:10.362622+00:00 px4 Controller[1694726]: 17:04:10,337 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
2023-08-08T17:04:10.362654+00:00 px4 Controller[1694726]: 17:04:10,343 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
2023-08-08T17:04:10.362685+00:00 px4 Controller[1694726]: 17:04:10,358 |-WARN in ch.qos.logback.core.model.processor.AppenderModelHandler - Appender named [FILE] not referenced. Skipping further processing.
2023-08-08T17:04:10.362714+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [okhttp3] to OFF
2023-08-08T17:04:10.362739+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http] to INFO
2023-08-08T17:04:10.362777+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [cron] to INFO
2023-08-08T17:04:10.362807+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [LINSTOR/Controller] to INFO
2023-08-08T17:04:10.362839+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [LINSTOR/Controller] to false
2023-08-08T17:04:10.362870+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[LINSTOR/Controller]
2023-08-08T17:04:10.362902+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [LINSTOR/Satellite] to INFO
2023-08-08T17:04:10.362944+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [LINSTOR/Satellite] to false
2023-08-08T17:04:10.362975+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[LINSTOR/Satellite]
2023-08-08T17:04:10.363005+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [LINSTOR/TESTS] to OFF
2023-08-08T17:04:10.363033+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [LINSTOR/TESTS] to false
2023-08-08T17:04:10.363061+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[LINSTOR/TESTS]
2023-08-08T17:04:10.363089+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to INFO
2023-08-08T17:04:10.363119+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
2023-08-08T17:04:10.363150+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@1b2abca6 - End of configuration.
2023-08-08T17:04:10.363180+00:00 px4 Controller[1694726]: 17:04:10,360 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@6392827e - Registering current configuration as safe fallback point
2023-08-08T17:04:10.656103+00:00 px4 Controller[1694726]: 17:04:10.655 [main] INFO  LINSTOR/Controller - SYSTEM - ErrorReporter DB version 1 found.
2023-08-08T17:04:10.656621+00:00 px4 Controller[1694726]: 17:04:10.656 [main] INFO  LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
2023-08-08T17:04:10.682661+00:00 px4 Controller[1694726]: 17:04:10.682 [main] INFO  LINSTOR/Controller - SYSTEM - Database type is etcd
2023-08-08T17:04:10.682881+00:00 px4 Controller[1694726]: 17:04:10.682 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading API classes started.
2023-08-08T17:04:11.126970+00:00 px4 Controller[1694726]: 17:04:11.126 [Main] INFO  LINSTOR/Controller - SYSTEM - API classes loading finished: 443ms
2023-08-08T17:04:11.127123+00:00 px4 Controller[1694726]: 17:04:11.126 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection started.
2023-08-08T17:04:11.146791+00:00 px4 Controller[1694726]: 17:04:11.146 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule"
2023-08-08T17:04:11.147267+00:00 px4 Controller[1694726]: 17:04:11.147 [Main] INFO  LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule" is not installed
2023-08-08T17:04:11.147358+00:00 px4 Controller[1694726]: 17:04:11.147 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule"
2023-08-08T17:04:11.166204+00:00 px4 Controller[1694726]: 17:04:11.165 [Main] INFO  LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule" was successful
2023-08-08T17:04:11.166459+00:00 px4 Controller[1694726]: 17:04:11.166 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
2023-08-08T17:04:11.167412+00:00 px4 Controller[1694726]: 17:04:11.167 [Main] INFO  LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" was successful
2023-08-08T17:04:12.160229+00:00 px4 Controller[1694726]: 17:04:12.159 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection finished: 1033ms
2023-08-08T17:04:12.160389+00:00 px4 Controller[1694726]: 17:04:12.160 [Main] INFO  LINSTOR/Controller - SYSTEM - Cryptography provider: Using default cryptography module
2023-08-08T17:04:12.457393+00:00 px4 Controller[1694726]: 17:04:12.457 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
2023-08-08T17:04:12.707377+00:00 px4 Controller[1694726]: 17:04:12.707 [Main] INFO  LINSTOR/Controller - SYSTEM - SpaceTrackingService: Instance added as a system service
2023-08-08T17:04:12.708368+00:00 px4 Controller[1694726]: 17:04:12.708 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService
2023-08-08T17:04:12.708716+00:00 px4 Controller[1694726]: 17:04:12.708 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing the etcd database
2023-08-08T17:04:12.708792+00:00 px4 Controller[1694726]: 17:04:12.708 [Main] INFO  LINSTOR/Controller - SYSTEM - etcd connection URL is "etcd://px1.cc.private:2379,px2.cc.private:2379,px3.cc.private:2379,px4.cc.private:2379"
2023-08-08T17:04:13.344203+00:00 px4 Controller[1694726]: 17:04:13.343 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'ETCDDatabaseService' of type ETCDDatabaseService
2023-08-08T17:04:13.348404+00:00 px4 Controller[1694726]: 17:04:13.348 [Main] INFO  LINSTOR/Controller - SYSTEM - Security objects load from database is in progress
2023-08-08T17:04:13.504371+00:00 px4 Controller[1694726]: 17:04:13.504 [Main] INFO  LINSTOR/Controller - SYSTEM - Security objects load from database completed
2023-08-08T17:04:13.504515+00:00 px4 Controller[1694726]: 17:04:13.504 [Main] INFO  LINSTOR/Controller - SYSTEM - Core objects load from database is in progress
2023-08-08T17:04:13.741625+00:00 px4 Controller[1694726]: 17:04:13.741 [Main] INFO  LINSTOR/Controller - SYSTEM - Core objects load from database completed
2023-08-08T17:04:13.830935+00:00 px4 Controller[1694726]: 17:04:13.830 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'TaskScheduleService' of type TaskScheduleService
2023-08-08T17:04:13.833775+00:00 px4 Controller[1694726]: 17:04:13.833 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing network communications services
2023-08-08T17:04:13.834306+00:00 px4 Controller[1694726]: 17:04:13.834 [Main] WARN  LINSTOR/Controller - SYSTEM - The SSL network communication service 'DebugSslConnector' could not be started because the keyStore file (/etc/linstor/ssl/keystore.jks) is missing
2023-08-08T17:04:13.839026+00:00 px4 Controller[1694726]: 17:04:13.838 [Main] INFO  LINSTOR/Controller - SYSTEM - Created network communication service 'PlainConnector'
2023-08-08T17:04:13.839247+00:00 px4 Controller[1694726]: 17:04:13.839 [Main] WARN  LINSTOR/Controller - SYSTEM - The SSL network communication service 'SslConnector' could not be started because the keyStore file (/etc/linstor/ssl/keystore.jks) is missing
2023-08-08T17:04:13.839307+00:00 px4 Controller[1694726]: 17:04:13.839 [Main] INFO  LINSTOR/Controller - SYSTEM - Created network communication service 'SslConnector'
2023-08-08T17:04:13.904847+00:00 px4 Controller[1694726]: 17:04:13.904 [Main] INFO  LINSTOR/Controller - SYSTEM - Reconnecting to previously known nodes
2023-08-08T17:04:13.922775+00:00 px4 Controller[1694726]: 17:04:13.922 [Main] INFO  LINSTOR/Controller - SYSTEM - Reconnect requests sent
2023-08-08T17:04:13.923410+00:00 px4 Controller[1694726]: 17:04:13.923 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'SpaceTrackingService' of type SpaceTrackingService
2023-08-08T17:04:13.923742+00:00 px4 Controller[1694726]: 17:04:13.923 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'ScheduleBackupService' of type ScheduleBackupService
2023-08-08T17:04:13.924750+00:00 px4 Controller[1694726]: 17:04:13.924 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'EbsStatusPoll' of type EbsStatusPoll
2023-08-08T17:04:14.061903+00:00 px4 Controller[1694726]: 17:04:14.061 [SpaceTrackingService] ERROR LINSTOR/Controller - SYSTEM - Uncaught exception in j [Report number 64D2758A-00000-000000]
2023-08-08T17:04:14.223450+00:00 px4 Controller[1694726]: Aug 08, 2023 5:04:14 PM org.glassfish.jersey.server.wadl.WadlFeature configure
2023-08-08T17:04:14.223568+00:00 px4 Controller[1694726]: WARNING: JAX-B API not found . WADL feature is disabled.
2023-08-08T17:04:14.740898+00:00 px4 Controller[1694726]: 17:04:14.740 [Main] ERROR LINSTOR/Controller - SYSTEM - Unsupported address type [Report number 64D2758A-00000-000001]
2023-08-08T17:04:14.741076+00:00 px4 Controller[1694726]: 17:04:14.740 [Main] INFO  LINSTOR/Controller - SYSTEM - Trying to start grizzly http server on fallback ipv4: 0.0.0.0
2023-08-08T17:04:14.761325+00:00 px4 Controller[1694726]: Aug 08, 2023 5:04:14 PM org.glassfish.jersey.server.wadl.WadlFeature configure
2023-08-08T17:04:14.761417+00:00 px4 Controller[1694726]: WARNING: JAX-B API not found . WADL feature is disabled.
2023-08-08T17:04:14.961809+00:00 px4 Controller[1694726]: Aug 08, 2023 5:04:14 PM org.glassfish.grizzly.http.server.NetworkListener start
2023-08-08T17:04:14.961957+00:00 px4 Controller[1694726]: INFO: Started listener bound to [0.0.0.0:3370]
2023-08-08T17:04:14.970258+00:00 px4 Controller[1694726]: Aug 08, 2023 5:04:14 PM org.glassfish.grizzly.http.server.HttpServer start
2023-08-08T17:04:14.970501+00:00 px4 Controller[1694726]: INFO: [HttpServer-1] Started.
2023-08-08T17:04:14.973940+00:00 px4 Controller[1694726]: 17:04:14.973 [Main] INFO  LINSTOR/Controller - SYSTEM - Controller initialized

I use:

# apt list | grep linstor

linstor-client/unknown,now 1.19.0-1 all [installed]
linstor-common/unknown,now 1.24.0-1 all [installed,automatic]
linstor-controller/unknown,now 1.24.0-1 all [installed]
linstor-proxmox/unknown,now 7.0.1-1 all [installed]
linstor-satellite/unknown,now 1.24.0-1 all [installed]
python-linstor/unknown,now 1.19.0-1 all [installed,automatic]

# lsb_release -a

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm
@ggzengel
Copy link
Author

ggzengel commented Aug 8, 2023

After deleting /LINSTOR/SPACE_HISTORY/ and setting /LINSTOR/TRACKING_DATE/ENTRY_DATE the REST-API works.

etcdctl --user=root:$pwd del /LINSTOR/SPACE_HISTORY/ --prefix
etcdctl --user=root:$pwd put /LINSTOR/TRACKING_DATE/ENTRY_DATE 2023-08-08

this is related to: #365

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ggzengel