Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use of INTERPOLATE triggers "Not found column" in 24.3 in some cases #62464

Closed
vincentbernat opened this issue Apr 9, 2024 · 1 comment · Fixed by #64096
Closed

Use of INTERPOLATE triggers "Not found column" in 24.3 in some cases #62464

vincentbernat opened this issue Apr 9, 2024 · 1 comment · Fixed by #64096
Assignees
@yakov-olkhovskiy
Labels
analyzer Issues and pull-requests related to new analyzer bug Confirmed user-visible misbehaviour in official release

Comments

@vincentbernat
Copy link
Contributor

vincentbernat commented Apr 9, 2024 •
edited

Describe the issue

A request using WITH FILL ... INTERPOLATE triggers a "Not found column" error for a column that exists. This works with 23.12. This does not work with 24.3.2. This works with 23.12.

How to reproduce

Using this table:

CREATE TABLE default.flows_1m0s
(
    `TimeReceived` DateTime CODEC(DoubleDelta, LZ4),
    `SamplingRate` UInt64,
    `ExporterAddress` LowCardinality(IPv6),
    `ExporterName` LowCardinality(String),
    `ExporterGroup` LowCardinality(String),
    `ExporterRole` LowCardinality(String),
    `ExporterSite` LowCardinality(String),
    `ExporterRegion` LowCardinality(String),
    `ExporterTenant` LowCardinality(String),
    `SrcAS` UInt32,
    `DstAS` UInt32,
    `SrcNetName` LowCardinality(String),
    `DstNetName` LowCardinality(String),
    `SrcNetRole` LowCardinality(String),
    `DstNetRole` LowCardinality(String),
    `SrcNetSite` LowCardinality(String),
    `DstNetSite` LowCardinality(String),
    `SrcNetRegion` LowCardinality(String),
    `DstNetRegion` LowCardinality(String),
    `SrcNetTenant` LowCardinality(String),
    `DstNetTenant` LowCardinality(String),
    `SrcCountry` FixedString(2),
    `DstCountry` FixedString(2),
    `SrcGeoCity` LowCardinality(String),
    `DstGeoCity` LowCardinality(String),
    `SrcGeoState` LowCardinality(String),
    `DstGeoState` LowCardinality(String),
    `Dst1stAS` UInt32,
    `Dst2ndAS` UInt32,
    `Dst3rdAS` UInt32,
    `InIfName` LowCardinality(String),
    `OutIfName` LowCardinality(String),
    `InIfDescription` LowCardinality(String),
    `OutIfDescription` LowCardinality(String),
    `InIfSpeed` UInt32,
    `OutIfSpeed` UInt32,
    `InIfConnectivity` LowCardinality(String),
    `OutIfConnectivity` LowCardinality(String),
    `InIfProvider` LowCardinality(String),
    `OutIfProvider` LowCardinality(String),
    `InIfBoundary` Enum8('undefined' = 0, 'external' = 1, 'internal' = 2),
    `OutIfBoundary` Enum8('undefined' = 0, 'external' = 1, 'internal' = 2),
    `EType` UInt32,
    `Proto` UInt32,
    `Bytes` UInt64 CODEC(T64, LZ4),
    `Packets` UInt64 CODEC(T64, LZ4),
    `PacketSize` UInt64 ALIAS intDiv(Bytes, Packets),
    `PacketSizeBucket` LowCardinality(String) ALIAS multiIf(PacketSize < 64, '0-63', PacketSize < 128, '64-127', PacketSize < 256, '128-255', PacketSize < 512, '256-511', PacketSize < 768, '512-767', PacketSize < 1024, '768-1023', PacketSize < 1280, '1024-1279', PacketSize < 1501, '1280-1500', PacketSize < 2048, '1501-2047', PacketSize < 3072, '2048-3071', PacketSize < 4096, '3072-4095', PacketSize < 8192, '4096-8191', PacketSize < 10240, '8192-10239', PacketSize < 16384, '10240-16383', PacketSize < 32768, '16384-32767', PacketSize < 65536, '32768-65535', '65536-Inf'),
    `ForwardingStatus` UInt32
)
ENGINE = SummingMergeTree((Bytes, Packets))
PARTITION BY toYYYYMMDDhhmmss(toStartOfInterval(TimeReceived, toIntervalSecond(12096)))
PRIMARY KEY (TimeReceived, ExporterAddress, EType, Proto, InIfName, SrcAS, ForwardingStatus, OutIfName, DstAS, SamplingRate)
ORDER BY (TimeReceived, ExporterAddress, EType, Proto, InIfName, SrcAS, ForwardingStatus, OutIfName, DstAS, SamplingRate, SrcNetName, DstNetName, SrcNetRole, DstNetRole, SrcNetSite, DstNetSite, SrcNetRegion, DstNetRegion, SrcNetTenant, DstNetTenant, SrcCountry, DstCountry, SrcGeoCity, DstGeoCity, SrcGeoState, DstGeoState, Dst1stAS, Dst2ndAS, Dst3rdAS)
TTL TimeReceived + toIntervalSecond(604800)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1

And this query:

SELECT
    TimeReceived AS time,
    [toString(SrcAS)] AS dimensions
FROM flows_1m0s
WHERE (TimeReceived >= toDateTime('2024-04-09 14:27:00', 'UTC')) AND (TimeReceived <= toDateTime('2024-04-09 20:27:00', 'UTC'))
GROUP BY
    time,
    dimensions
ORDER BY time ASC WITH FILL FROM toDateTime('2024-04-09 14:27:00', 'UTC') TO toDateTime('2024-04-09 20:27:00', 'UTC') + toIntervalSecond(1) STEP 60
INTERPOLATE ( dimensions AS ['Other'] )

This triggers the following error:

Received exception from server (version 24.3.2):
Code: 10. DB::Exception: Received from localhost:9000. DB::Exception: Not found column __table1.SrcAS: in block __table1.TimeReceived DateTime UInt32(size = 0), array(__table1.SrcAS) Array(UInt32) Array(size = 0, UInt64(size = 0), UInt32(size = 0)). (NOT_FOUND_COLUMN_IN_BLOCK)

Removing INTERPOLATE makes it work. Also, not using an array for dimensions also works:

SELECT
    TimeReceived AS time,
    toString(SrcAS) AS dimensions
FROM flows_1m0s
WHERE (TimeReceived >= toDateTime('2024-04-09 14:27:00', 'UTC')) AND (TimeReceived <= toDateTime('2024-04-09 20:27:00', 'UTC'))
GROUP BY
    time,
    dimensions
ORDER BY time ASC WITH FILL FROM toDateTime('2024-04-09 14:27:00', 'UTC') TO toDateTime('2024-04-09 20:27:00', 'UTC') + toIntervalSecond(1) STEP 60
INTERPOLATE ( dimensions AS 'Other' )
@yakov-olkhovskiy yakov-olkhovskiy added the analyzer Issues and pull-requests related to new analyzer label Apr 12, 2024
@yakov-olkhovskiy yakov-olkhovskiy self-assigned this Apr 12, 2024
@yakov-olkhovskiy
Copy link
Member

seems to be analyzer issue
use set allow_experimental_analyzer=0 as a workaround until we fix it...

vincentbernat added a commit to akvorado/akvorado that referenced this issue Apr 19, 2024
@vincentbernat
common/clickhousedb: disable experimental analyzer in ClickHouse
12967c8 
See ClickHouse/ClickHouse#62464
and #1165

This setting was introduced in 22.3.16.1190-lts. See ClickHouse/ClickHouse@e6bfff3
@Algunenano Algunenano added bug Confirmed user-visible misbehaviour in official release and removed backward compatibility labels May 9, 2024
@yakov-olkhovskiy yakov-olkhovskiy mentioned this issue May 18, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yakov-olkhovskiy yakov-olkhovskiy

Labels
analyzer Issues and pull-requests related to new analyzer bug Confirmed user-visible misbehaviour in official release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix analyzer: only interpolate expression should be used for DAG ClickHouse/ClickHouse
3 participants
@vincentbernat @Algunenano @yakov-olkhovskiy