What's Changed

Exciting New Features ✨

• feat(query): support settings admin api for global level settings by @zhang2014 in #14901
• feat: add a new ratio config instead of table_data_deserialized_data_bytes by @chienguo in #14896
• feat(ast): add parser for script by @andylokandy in #14891
• feat: support cross join spilling by @xudong963 in #14887
• feat: add more function alias names by @sundy-li in #14924
• feat(query): support spill for new agg hashtable by @Freejww in #14905
• feat(query): Support refresh inverted index by @b41sh in #14912
• feat: ndjson support option null_if. by @youngsofun in #14938
• feat(query): add statement_queued_timeout_in_seconds setting for queries queue by @zhang2014 in #14945
• feat: change tracking enabled table support merge into by @zhyass in #14900
• feat: histogram aggregate function by @suimenno3002 in #14839
• feat(query): add metrics for session queue manager by @everpcpc in #14966
• feat(query): add error_or function by @sundy-li in #14980
• feat: purge duplicated files (copy into table) by @dantengsky in #14962
• feat(query): support parse mulity modify column? ty by @TracyZYJ in #14979
• feat: support prql by @ncuwaln in #14922
• chore(query): support domain contains in string type by @sundy-li in #15023
• feat(query): check view's inner table privilege when create or alter view by @TCeason in #14986
• feat: add buffer for spiller by @xudong963 in #15021
• feat: add show views & desc view by @blackstar-baba in #14926
• feat: support pagination on task_history by @ZhiHanZ in #15047
• feat(ast): add parser for template sql by @andylokandy in #15026
• refactor: SHOW GRANTS should not display drop on table/database by @TCeason in #14931
• feat(query): Support query match inverted index by @b41sh in #14997
• feat: Implement concurrent stat by @Xuanwo in #15083
• feat(metasrv): add network metrics rpc_delay_ms by @everpcpc in #15084
• feat: create stream at navigation point by @zhyass in #15062
• feat(query): implement st_makeline() function by @kkk25641463 in #15059
• feat(planner): improve push down filter join by @Dousir9 in #14872
• feat: support otel context propagation for databend-query by @ZhiHanZ in #15096
• feat(query): add query_result_cache_min_execute_secs by @sundy-li in #15101
• feat(query): Support hook refresh inverted index after insert by @b41sh in #15094
• feat: time travel support at stream by @zhyass in #15116
• feat(query): implement st_geometryfromewkb function by @kkk25641463 in #15118
• feat: refactor ndjson input format. by @youngsofun in #14943
• feat(script): add script compiler by @andylokandy in #15113
• feat: support multi-table insert by @SkyFan2002 in #15002
• feat(query): to_timestamp support parser string to ts without timezone by @TCeason in #15131
• feat: Enhance show views by @blackstar-baba in #15109
• feat: support expr in values clause of multi table insert by @SkyFan2002 in #15147
• feat(query): implement ST_GEOMFROMGEOHASH by @kkk25641463 in #15156
• feat(query): show create table support display inverted index by @b41sh in #15169
• feat(query): support CHANGES clause by @zhyass in #15163
• feat: add inverted index pruning status by @BohuTANG in #15175
• feat: Support to add random seed on random engine by @blackstar-baba in #15167
• feat(planner): push down not in filter expression by @Dousir9 in #15168
• feat: time travel support at offset by @zhyass in #15177

Thoughtful Bug Fix 🔧

• fix(ci): fix internal benchmark by @TCeason in #14897
• fix: Display of merge into .. insert * by @dantengsky in #14892
• fix: check_and_upgrade_to_pb() should return MetaError, add quota by @drmingdrmer in #14899
• fix: Display of compact AST by @SkyFan2002 in #14914
• fix(query): fix native topk_sorter by @sundy-li in #14939
• fix: stream source can't get the same tablemeta by @zhyass in #14935
• fix: cluster table with change tracking enabled append panic by @zhyass in #14956
• fix: enable runtime filter when join_spilling_memory_ratio !=0 by @xudong963 in #14959
• fix: fix stream explain bug for merge into by @JackTan25 in #14969
• fix(query): Need return error udf_name when derialize from pb failed. by @TCeason in #14964
• fix: try fix compute stddev for decimal type overflow by @ariesdevil in #14952
• fix: csv read error with skip_header. by @youngsofun in #14981
• fix: fix incorrect agg spill in new agg hashtable by @Freejww in #14995
• fix: wrong row id by @youngsofun in #15018
• fix: trim leading '/'s for files specified in copy opt by @dantengsky in #15056
• fix: stream get wrong result if change tracking is changed by @zhyass in #15072
• fix(ast): fix incorrect display for ast (part 1) by @andylokandy in #15069
• fix: privilege check for insert statement by @SkyFan2002 in #15075
• fix(query): fix attach_query_str by @sundy-li in #15066
• fix: fix plan_sql error can not write query_log by @BohuTANG in #15077
• fix: improve error hints when select * from xx.yy by @ZhengLin-Li in #14663
• fix: Support udf for copy table by @JackTan25 in #15078
• fix(ast): fix incorrect display for ast (part 2) by @andylokandy in #15076
• fix: make block data type be consistent with hash method by @xudong963 in #15089
• fix: add prql description by @ncuwaln in #15091
• fix: sqllogicaltest datatype mismatch in expr_in_window.test by @c8ef in #15090
• fix: schema mismatch in named_window_basic.test by @c8ef in #15099
• fix: test duplicate in window_basic.test by @c8ef in #15100
• fix(query): verify_privilege need check privilege one by one by @TCeason in #15105
• fix(query):user option DEFAULT_ROLE ignore ascii_case by @TCeason in #15137
• fix(query): tuple field name can not contain special char by @TCeason in #15126
• fix(query): new agg hashtable hung with little data on singleton by @Freejww in #15140
• fix(query): show query with name consider quote by @TCeason in #15138
• fix: forbid end at stream syntax by @zhyass in #15173
• fix(storage): Don't return empty credential while loading failed by @Xuanwo in #15181

Code Refactor 🎉

• refactor: upgrade to openraft 0.9.0 by @drmingdrmer in #14907
• refactor: use kvapi::Key to define meta-service key for QuotaMgr by @drmingdrmer in #14915
• refactor: reduce hash join build spilling files by @xudong963 in #14919
• refactor: define kvapi::Key for PasswordPolicyMgr by @drmingdrmer in #14920
• refactor: reduce meta call by @TCeason in #14917
• refactor: add NetworkPolicyIdent as NetworkPolicy meta-service key by @drmingdrmer in #14933
• refactor: http handler plan sql async. by @youngsofun in #14908
• refactor: Add TIdent for all meta-service key that share the same form of <PREFIX>/<TENANT>/<NAME> by @drmingdrmer in #14940
• refactor: add generic CrudMgr to implement various simple CRUD manager by @drmingdrmer in #14988
• refactor(query): use project plan for copy into select by @zhang2014 in #14977
• refactor: use CrudMgr to impl PasswordPolicyMgr by @drmingdrmer in #14994
• refactor: Add ExistError and UnknownError, TenantResource does not need to define error type any more by @drmingdrmer in #14996
• refactor: Use CrudMgr to impl FileFormatMgr by @drmingdrmer in #14999
• refactor: Use CrudMgr to impl ConnectionMgr by @drmingdrmer in #15003
• refactor(query): add copy into location physical plan by @zhang2014 in #15010
• refactor(query): csv reader support prefetch by @youngsofun in #14983
• refactor: alter view only need to modify table option by @TCeason in #15012
• refactor: truncate support build pipeline by @zhyass in #15027
• refactor: move id-generator from crate meta-api to meta-app by @drmingdrmer in #15036
• refactor: remove innecessary serde by @drmingdrmer in #15037
• refactor(query): use scan table physical plan for copy into table from stage by @zhang2014 in #15016
• refactor: remove innecessary serde by @drmingdrmer in #15040
• refactor: remove innecessary serde by @drmingdrmer in #15042
• refactor: spead up csv reader. by @youngsofun in #15043
• refactor(query): remove metrics if moved to the profile by @zhang2014 in #15054
• refactor: reduce hash join probe spilled files by @xudong963 in #14942
• refactor: separate key and value: add BackgroundTaskCreator by @drmingdrmer in #15063
• refactor: remove serde from CatalogId and CatalogNameIdent by @drmingdrmer in #15068
• refactor: remove serde from kvapi::Key types by @drmingdrmer in #15070
• refactor: use type Tenant for BackgroundJobIdent.tenant by @drmingdrmer in #15074
• refactor: BackgroundTaskIdent use Tenant by @drmingdrmer in #15080
• refactor: make the block pruning more clear by @BohuTANG in #15079
• refactor: change PartInfo is_lazy to LazyLevel by @BohuTANG in #15081
• refactor: QueryFragmentsPlanPacket should include just setting changes by @drmingdrmer in #15095
• refactor(executor): refactor queries executor to prepare by @dqhl76 in #14965
• refactor: show grants support display ObjectId by @TCeason in #15064
• refactor: replace NonEmptyString with Tenant by @drmingdrmer in #15103
• refactor: replace String with Tenant by @drmingdrmer in #15119
• refactor: use TIdent to unify kvapi key def for MaskPolicyTableIdListIdent and DataMaskNameIdent by @drmingdrmer in #15123
• refactor: remove count-table for tenant. It is not used at all by @drmingdrmer in #15134
• refactor: use Tenat in VirtualColumnNameIdent by @drmingdrmer in #15135
• refactor(query): record physical plan level metrics into profiling log by @zhang2014 in #15092
• refactor: define ShareEndpointIdent with TIdent by @drmingdrmer in #15136
• refactor: add peer info to network error context by @drmingdrmer in #15139
• refactor: add Tenant to ShareConsumer by @drmingdrmer in #15142
• chore: client should not depend on the final response. by @youngsofun in #15146
• refactor(executor): globalize QueriesPipelineExecutor creation method by @dqhl76 in #15129
• refactor(planner): remove statistics collection from bind phase by @Dousir9 in #15112
• refactor(query): only intercept dml and queries in queries queue by @zhang2014 in #15143
• refactor: user can execute show grants query itself by @TCeason in #15149
• refactor: generate inverted indexs for each blocks by @b41sh in #15150
• refactor: remove dependency on parquet2: Part I by @SkyFan2002 in #15158
• refactor: use Tenant for ListBackgroundTasksReq by @drmingdrmer in #15176
• refactor: move TIdent into separate files by @drmingdrmer in #15179

Others 📒

• chore(storage): split mutation_log file by @zhyass in #14903
• chore(query): create user/role name can not contain single/double quote by @TCeason in #14902
• chore(ci): benchmark local need set meta client timeout by @TCeason in #14910
• chore: extend time tolerance for meta-service test by @drmingdrmer in #14927
• chore: add step logs for hash join build side by @xudong963 in #14930
• chore: use shorter name for management API by @drmingdrmer in #14937
• chore(ci): add pr summary check by @everpcpc in #14944
• chore(ci): fix set failure for pr check by @everpcpc in #14946
• chore(ci): fix description check by @everpcpc in #14947
• chore(ci): fix event for pr assistant by @everpcpc in #14948
• chore(query): improve push_column_with_selection by @Dousir9 in #14954
• chore(query): add disable system tables by @zhang2014 in #14958
• chore(planner): fix distributed query plan by @Dousir9 in #14951
• chore: tweak txn syntax by @SkyFan2002 in #14960
• chore(query): add settings level for admin api by @zhang2014 in #14967
• chore(ast): CreatOption::None -> CreateOption::Create by @andylokandy in #14941
• chore: improve histogram‘s implement and comments by @suimenno3002 in #14976
• chore: number range for settings by @BohuTANG in #14985
• chore: group limit optimization in new hashtable by @sundy-li in #14989
• chore: add integration test for runtime filter by @xudong963 in #14992
• chore: upgrade openraft to 0.9.1 by @drmingdrmer in #15006
• chore: max_running_queries from 8 to 0, disable the max running queries by @BohuTANG in #15013
• chore: tweak deletion batch size & log messages by @dantengsky in #15005
• chore: rollback partial tests by @JackTan25 in #15015
• chore: add benchmark for loading data on cloud by @everpcpc in #15017
• chore: add a setting for inlist to join by @xudong963 in #15024
• chore: unify TIdent key format by @drmingdrmer in #15025
• chore: the fist http resp wait longer to tolerant old bendsql clients. by @youngsofun in #15022
• chore: add log about presign elapsed by @flaneur2020 in #14990
• chore: fix sql seq for benchmark report by @everpcpc in #15030
• chore: print query detail for bench & fix prepare database for load bench by @everpcpc in #15032
• chore: default enalbe merge into by @BohuTANG in #15034
• chore: run load test with new label by @everpcpc in #15035
• chore(query): add spill profile for join by @zhang2014 in #15044
• chore(http): tune log on getting /v1/query/:id/kill by @flaneur2020 in #14987
• chore: default enable aggregate/sort/join spill by @BohuTANG in #15038
• chore: do not drop database first for bench by @everpcpc in #15041
• chore(query): improve group by @sundy-li in #15031
• chore(planner): fix invalid aggregate physical plan by @Dousir9 in #15046
• chore: adjust the default value for table lock timeout by @zhyass in #15055
• chore: use Tenant::new_literal() and new_nonempty() to avoid creating empty Tenant by @drmingdrmer in #15061
• chore(ci): fix benchmark job dependency by @everpcpc in #15087
• chore: remove rebinding merge into for insert-only by @xudong963 in #15086
• chore: clean z3 and meta dir by @zhyass in #15098
• chore(executor): remove duplicate profile statistics by @zhang2014 in #15115
• chore(query): errorcode use snake shape format by @TCeason in #15122
• chore: treat seq as place holder when navigate by @zhyass in #15124
• chore(planner): make optimize function async by @Dousir9 in #15121
• chore: adjust stack size of flaky ut by @SkyFan2002 in #15130
• chore: to avoid UT stackoverflow, disable async_backtrace of TypeChecker::resolve by @dantengsky in #15133
• chore(query): refactor new transform partition bucket for new aggregation hashtable by @Freejww in #15093
• chore: enable inlist to join in cluster by @xudong963 in #15108
• chore: remove no longer used message encoding benchmark by @dantengsky in #15148
• chore(query): fix order for scalar cmp with null by @sundy-li in #15145
• chore: fix typos by @Dousir9 in #15153
• chore(query): use i64 min max value as histogram bucket min max limit by @zhang2014 in #15152
• chore(query): turn on new agg hashtable by @Freejww in #15155
• chore(query): row-fetcher-cache by @sundy-li in #15159
• chore: add CopyInto to query queue by @BohuTANG in #15160
• chore: refine the query/02_function to query/functions by @BohuTANG in #15161
• chore: refine the query queue is_heavy_action func by @BohuTANG in #15166
• chore: fix window plan in lazy topn by @sundy-li in #15165
• chore(deps): bump h2 from 0.3.24 to 0.3.26 by @dependabot in #15172
• chore(query): fix boolean type comparison by @Dousir9 in #15171
• chore: return inserted rows for each table in multi table insert by @SkyFan2002 in #15178

New Contributors

• @suimenno3002 made their first contribution in #14839
• @ncuwaln made their first contribution in #14922
• @blackstar-baba made their first contribution in #14926
• @kkk25641463 made their first contribution in #15059
• @c8ef made their first contribution in #15090

Full Changelog: https://github.com/datafuselabs/databend/releases/tag/v1.2.410