Unable to connect external client to CrateDB - exception caught on transport layer, closing connection #10841

Qloppa · 2020-12-02T10:03:04Z

Qloppa
Dec 2, 2020

Hello there,

im using CrateDB with docker-compose and i have trouble connecting external clients.
It works just fine, using the machine-name within the docker network. But as soon as i try to connect via ip-adress:port im not able to connect to the CrateDB.

The CrateDB-log displays the following message:

[2020-11-30T15:41:27,370][WARN ][o.e.t.n.Netty4Transport  ] [Monte Cernera] exception caught on transport layer [org.elasticsearch.transport.netty4.NettyTcpChannel@6f701d8f], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (0,0,0,49)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:528) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:482) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (0,0,0,49)
        at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1281) ~[crate-app-3.1.2.jar:3.1.2]
        at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[crate-app-3.1.2.jar:3.1.2]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.29.Final.jar:4.1.29.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.29.Final.jar:4.1.29.Final]
        ... 23 more

My docker-compose.yml looks like this:

version: "3.5"
services:

  orion:
    image: fiware/orion:latest
    depends_on:
      - mongo-db
    networks:
      - default
    expose:
      - "1026"
    ports:
      - "1026:1026"
    command: -dbhost mongo-db -logLevel DEBUG -corsOrigin __ALL
    healthcheck:
      test: curl --fail -s http://orion:1026/version || exit 1

  cratedb:
    image: crate:3.1.2
    networks:
      - default
    ports:
      - "4200:4200"
      - "4300:4300"
    environment:
      - CRATE_HEAP_SIZE=1g
    command: crate -Clicense.enterprise=false -Cauth.host_based.enabled=false -Ccluster.name=democluster -Chttp.cors.enabled=true -Chttp.cors.allow-origin="*"
    volumes:
      - cratedata:/data

  quantumleap:
    image: smartsdk/quantumleap
    networks:
      - default
    expose:
      - "9002"
    ports:
      - "9002:8668"
    depends_on:
      - cratedb
    environment:
      - CRATE_HOST=cratedb

  grafana:
    image: grafana/grafana
    networks:
      - default
    depends_on:
      - cratedb
    expose:
      - "9000"
    ports:
      - "9000:3000"
    volumes:
      - grafanadata:/var/lib/grafana

  mongo-db:
    image: mongo:3.6
    expose:
      - "27017"
    ports:
      - "27017:27017"
    networks:
      - default
    command: --bind_ip_all --smallfiles
    volumes:
      - mongo-db:/data

  iot-agent-lorawan:
    image: fiware/iotagent-lorawan
    depends_on:
      - mongo-db
    expose:
      - "4061"
    ports:
      - "4061:4061"
    networks:
      - default
    volumes:
      - loraiotdata:/data
    environment:
      - "IOTA_CB_HOST=orion"
      - "IOTA_CB_PORT=1026"
      - "IOTA_NORTH_PORT=4061"
      - "IOTA_REGISTRY_TYPE=mongodb"
      - "IOTA_MONGO_HOST=mongo-db"
      - "IOTA_MONGO_PORT=27017"
      - "IOTA_MONGO_DB=iotagent-lorawan"
      - "IOTA_PROVIDER_URL=http://iot-agent:4061"

volumes:
  mongo-db:
  cratedata:
  grafanadata:
  loraiotdata:

I don´t suspect an issue with the docker-network, since the webinterface is reachable and the connection attempt is visible in the CrateDB-logs.
Can anyone help me with this issue?

With kind regards,
Qloppa

Answered by seut

Dec 2, 2020

The issue looks like you are connecting with a client which is using PostgreSQL protocol (e.g. PgAdmin) to the HTTP port of CrateDB.
CrateDB listen to PostgreSQL protocol by default on port 5432 which must be then exposed/mapped additionally when using docker.

View full answer

amotl · 2020-12-02T11:06:06Z

amotl
Dec 2, 2020
Maintainer

Dear @Qloppa,

thanks for writing in. While I don't know about the background of the specific error you are observing, I want to add some suggestions.

According to [1], QuantumLeap already uses CrateDB 4.1.4 so it might be a good idea to follow that already. At [2], @c0c0n3 mentioned that they are already using CrateDB 4.1.8 on production. On the other hand, [3] apparently already is based on CrateDB 4.2.2. In general, we just released CrateDB 4.3.2 including many improvements and stability fixes, so it might also be worth to try that version already.

Maybe @chicco785 or @c0c0n3 can add some more thoughts here?

With kind regards,
Andreas.

[1] https://github.com/smartsdk/ngsi-timeseries-api/blob/0.7.6/docker/docker-compose.yml#L41
[2] orchestracities/ngsi-timeseries-api#306 (comment)
[3] https://hub.docker.com/r/orchestracities/crate

P.S.: Apart from my suggestion, I can only find references to exception caught on transport layer or invalid internal transport message format within very old issues on our issue tracker, see #2057, #3786 and #6085.

3 replies

chicco785 Dec 2, 2020

Did you try to reach the web interface on yourip:4200 ? The issue you may have may be on the client sending the wrong "format" but not knowing exactly which query toward crate you are running, hard to say.

Qloppa Dec 2, 2020
Author

The webinterface is reachable and works just fine (with the ip-adress of the server). Im not really sure about the "format" of the messages from the client, but i can say that i tried with grafana (7.2.0) and with pgAdmin (4.28).
As mentioned, the connection with grafana works but only if i use the docker container-name. Grafana and CrateDB are in the same Docker network.

amotl Dec 2, 2020
Maintainer

So, the error message/stacktrace referenced above appears when accessing CrateDB with recent versions of either Grafana or pgAdmin?

Regarding compatibility with Grafana through PostgreSQL-connectivity, it should work in general but CrateDB still lacks auto-completion support which we are tracking at #9320.

However, I still believe it is some networking issue as Grafana actually is able to talk to CrateDB successfully within the Docker network, right? Would you mind trying this again with a more recent version of CrateDB just to verify the issue is still there?

seut · 2020-12-02T11:57:16Z

seut
Dec 2, 2020
Maintainer

The issue looks like you are connecting with a client which is using PostgreSQL protocol (e.g. PgAdmin) to the HTTP port of CrateDB.
CrateDB listen to PostgreSQL protocol by default on port 5432 which must be then exposed/mapped additionally when using docker.

2 replies

amotl Dec 2, 2020
Maintainer

Ah, good catch. This really sounds like it might match the issue @Qloppa is observing. Thanks!

Qloppa Dec 2, 2020
Author

that seemed to be the problem.
Thank you very much!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to connect external client to CrateDB - exception caught on transport layer, closing connection #10841

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Unable to connect external client to CrateDB - exception caught on transport layer, closing connection #10841

Qloppa Dec 2, 2020

Replies: 2 comments · 5 replies

amotl Dec 2, 2020 Maintainer

chicco785 Dec 2, 2020

Qloppa Dec 2, 2020 Author

amotl Dec 2, 2020 Maintainer

seut Dec 2, 2020 Maintainer

amotl Dec 2, 2020 Maintainer

Qloppa Dec 2, 2020 Author

Qloppa
Dec 2, 2020

Replies: 2 comments 5 replies

amotl
Dec 2, 2020
Maintainer

Qloppa Dec 2, 2020
Author

amotl Dec 2, 2020
Maintainer

seut
Dec 2, 2020
Maintainer

amotl Dec 2, 2020
Maintainer

Qloppa Dec 2, 2020
Author