You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, I'm having some slow querys with dynamic tables so I wanted to add some generic pre-aggs, I'm a little lost on finding good examples and also those I created are not working, and I think is because of the cube.js we have, since we are adding an account_uuid filter to have each database to just have data related to them (we use metabase cloud).
Tools:
Cube Cloud Latest (0.35.22)
Metabase Cloud v1.49.6
Google Cloud (have the prod_pre_aggregations dataset already saving this aggregations I did)
Related Cube.js schema
const{memoizedFetchAccountPassword}=require("./sql-auth")functiondecodeBase64(data){letbuff=Buffer.from(data,'base64');returnbuff.toString('ascii');}constdeconstructGlobalId=(globalId)=>{constdecoded=decodeBase64(globalId)constglobalIdParts=decoded.split(":")return{node: globalIdParts[0],uuid: globalIdParts[1]}}constcontextToAppId=(context)=>`CUBEJS_APP_${context.cacct}`constextendContext=(req)=>{// For SQL API (context is then handled by checkSqlAuth)if(req.headers===undefined||req.headers.cacct===undefined){return;}try{constdeconstructed=deconstructGlobalId(req.headers.cacct);return{cacct: deconstructed["uuid"]};}catch(err){console.log("Error extending context: "+err)}}constqueryRewrite=(query,request)=>{console.log("Rewriting for",JSON.stringify(request))constaccountId=request.cacct!==undefined ? request.cacct : request.securityContext.cacct;if(query["dimensions"].length>0){try{vartable_name=query["dimensions"][0].substring(0,query["dimensions"][0].indexOf("."));}catch{vartable_name=query["dimensions"][0]["cubeName"]}}// measures in queryelseif(query["measures"].length>0){try{vartable_name=query["measures"][0].substring(0,query["measures"][0].indexOf("."))}catch{vartable_name=query["measures"][0]["cubeName"]}}elseif(query["timeDimensions"].length>0){try{vartable_name=query["timeDimensions"][0]["dimension"].substring(0,query["timeDimensions"][0]["dimension"].indexOf("."))}catch{vartable_name=query["measures"][0]["cubeName"]}}else{console.log("Using account id:",accountId)console.log("The query:",query)}console.log("Using account id:",accountId);query.filters.push({member: `${table_name}.accountUuid`,operator: 'equals',values: [accountId],});returnquery;}constcheckSqlAuth=async(req,username)=>{constdeconstructed=deconstructGlobalId(username);// const password = await memoizedFetchAccountPassword(deconstructed['uuid']);constpassword="stagingaccess";if(password!==null){return{
password,securityContext: {cacct: deconstructed['uuid']}}}}// To test sql auth locally, uncomment the following line.// checkSqlAuth({}, "QWNjb3VudDo4NDUyZjg4MS0wYWM1LTRmMjYtOWFhYi0xNmUzN2IyZWM0ZjI=").then((ctx) => console.log("X>", ctx));constqueueOptions={concurrency: 2,executionTimeout: 600,orphanedTimeout: 120,heartBeatInterval: 120,};module.exports={http: {cors: {origin: '*',methods: 'GET,HEAD,PUT,PATCH,POST,DELETE',preflightContinue: false,allowedHeaders: ['Content-Type','Authorization','cacct'],optionsSuccessStatus: 204,},},scheduledRefreshTimer: 120,
checkSqlAuth,
contextToAppId,
extendContext,
queryRewrite,orchestratorOptions: {queryCacheOptions: {refreshKeyRenewalThreshold: 120,backgroundRenew: true,
queueOptions,},preAggregationsOptions: {queueOptions},},// Figure out ScheduledRefreshContexts later! (probably needed for QueryRewrite)// Placeholder to prevent the error message:scheduledRefreshContexts: ()=>[{securityContext: {cacct: '00000000-0000-0000-0000-000000000000'}}],};
So this is one of the dynamic files we have:
{% set account = "accountUuid" %}
{% set interaction = "interactionUuid" %}
{% set time = "time" %}
cubes:
{%- for cube in load_interaction_tags()["cubes"] %}
- name: {{ cube.name }}
sql: {{ cube.sql }}
dataSource: default
{%- if cube.dimensions is not none and cube.dimensions|length > 0 %}
pre_aggregations:
- name: {{cube.name + "_table" }}
dimensions:
{%- for dimension in cube.dimensions %}
- {{ dimension.name }}
{%- endfor %}
refresh_key:
every: 12 hour
- name: {{cube.name + "_table2" }}
dimensions:
{%- for dimension in cube.dimensions %}
{%- if dimension.name != account and dimension.name != interaction %}
- {{ dimension.name }}
{%- endif %}
{%- endfor %}
refresh_key:
every: 12 hour
{%- for dimension in cube.dimensions %}
- name: {{ dimension.name + "_filter" }}
dimensions:
- {{ dimension.name }}
refresh_key:
every: 12 hour
- name: {{ dimension.name + "_by_distinctUuid" }}
dimensions:
- {{ dimension.name }}
measures:
- distinctUuid
refresh_key:
every: 12 hour
{%- if dimension.type == time %}
- name: {{ dimension.name + "uuid_by_month" }}
measures:
- distinctUuid
time_dimension: {{ dimension.name }}
granularity: month
- name: {{ dimension.name + "uuid_by_day" }}
measures:
- distinctUuid
time_dimension: {{ dimension.name }}
granularity: day
- name: {{ dimension.name + "uuid_by_year" }}
measures:
- distinctUuid
time_dimension: {{ dimension.name }}
granularity: year
{%- endif %}
{%- endfor %}
{%- endif %}
{%- if cube.joins is not none and cube.joins|length > 0 %}
joins:
{%- for join in cube.joins %}
- name: {{ join.name }}
relationship: {{ join.relationship }}
sql: {{ join.sql }}
{%- endfor %}
{%- endif %}
{%- if cube.measures is not none and cube.measures|length > 0 %}
measures:
{%- for measure in cube.measures %}
- name: {{ measure.name }}
type: {{ measure.type }}
{%- if measure.sql %}
sql: {{ measure.sql }}
{%- endif %}
{%- endfor %}
{%- endif %}
{%- if cube.dimensions is not none and cube.dimensions|length > 0 %}
dimensions:
{%- for dimension in cube.dimensions %}
- name: {{ dimension.name }}
sql: {{ dimension.sql }}
type: {{ dimension.type }}
{% if dimension.primaryKey == True -%}
primaryKey: true
public: true
{% endif -%}
{%- endfor %}
{%- endif %}
{%- endfor %}
As you see I already did some pre-aggs for every table, column and some testing ones for date columns, but my problem is first, this is how the table gets to metabase:
And as you see it doesn't detects the pre-aggregation, I don't know if its because the query has user and cubejoin columns in them, or maybe we have too much pre-aggs?
The thing is I don't find how to do a pre-agg with joins and filters, all this dynamic tables are primarily used joined with two other tables. I want to do something like the two first pre-aggs but with the joins to the tables and filtering not null in a different column for each pre-agg (this is likely the most general useful pre-agg I can create).
Always videos and documentation are welcome, I did all this with what I could find but I cannot say I read them deeply, though I tried.
The text was updated successfully, but these errors were encountered:
Is it possible that it stayed bugged? I have the dataset created and last modified on 22, but I have been creating more pre-aggregations and deleting them:
It also seems like I have less than 100 pre-aggs from like 5-10 tables, but I should have more.
I disabled this two options and it seems that now the pre-aggs are working when metabase call those querys, I don't know if it was too much for the wam-up, it was like bugged and not creating the new pre-aggregations, now seems like its working.
Hello, I'm having some slow querys with dynamic tables so I wanted to add some generic pre-aggs, I'm a little lost on finding good examples and also those I created are not working, and I think is because of the cube.js we have, since we are adding an account_uuid filter to have each database to just have data related to them (we use metabase cloud).
Tools:
Related Cube.js schema
So this is one of the dynamic files we have:
As you see I already did some pre-aggs for every table, column and some testing ones for date columns, but my problem is first, this is how the table gets to metabase:
And this is the query generated in query history:
And as you see it doesn't detects the pre-aggregation, I don't know if its because the query has user and cubejoin columns in them, or maybe we have too much pre-aggs?
The thing is I don't find how to do a pre-agg with joins and filters, all this dynamic tables are primarily used joined with two other tables. I want to do something like the two first pre-aggs but with the joins to the tables and filtering not null in a different column for each pre-agg (this is likely the most general useful pre-agg I can create).
Always videos and documentation are welcome, I did all this with what I could find but I cannot say I read them deeply, though I tried.
The text was updated successfully, but these errors were encountered: