Struct SqlBatchView 

pub struct SqlBatchView { /* private fields */ }

SQL-defined view updated in batch

Implementations

impl SqlBatchView

pub async fn new( runtime: Arc<RuntimeEnv>, view_set_name: Arc<String>, min_event_time_column: Arc<String>, max_event_time_column: Arc<String>, count_src_query: Arc<String>, extract_query: Arc<String>, merge_partitions_query: Arc<String>, lake: Arc<DataLakeConnection>, view_factory: Arc<ViewFactory>, session_configurator: Arc<dyn SessionConfigurator>, update_group: Option<i32>, max_partition_delta_from_source: TimeDelta, max_partition_delta_from_merge: TimeDelta, ) -> Result<SqlBatchView, Error>

Arguments

• runtime - datafusion runtime
• view_set_name - name of the table
• min_event_time_column - min(column) should result in the first timestamp in a dataframe
• max_event_time_column - max(column) should result in the last timestamp in a dataframe
• count_src_query - used to count the rows of the underlying data to know if a cached partition is up to date
• extract_query - used to extract the source data into a cached partition
• merge_partitions_query - used to merge multiple partitions into a single one (and user queries which are one multiple partitions by default)
• lake - data lake
• view_factory - all views accessible to the count_src_query
• session_configurator - configurator for registering custom tables (e.g., JSON files)
• update_group - tells the daemon which view should be materialized and in what order

pub fn with_merge_sort_order( self, columns: Vec<Arc<String>>, ) -> Result<SqlBatchView, Error>

Declares that this view’s partitions are internally sorted, ascending, by columns in order. columns must be non-empty and every name must exist in this view’s schema.

This is a three-item view-author contract:

1. Every declared sort column must appear among the merge query’s GROUP BY keys (order within GROUP BY is irrelevant; extra keys degrade to PartiallySorted, not a blocking sort).
2. Any enrichment join in the merge query must put the dimension table on the left and the ordered stream on the right – CollectLeft buffers its left (build) input and inherits its output ordering from its right (probe) input, so the natural phrasing (<ordered agg>) a LEFT JOIN dim d reinstates a blocking sort.
3. The merge query’s aggregates must be composable over already-aggregated rows (e.g. sum(count), not count(*); no bare avg – carry sum and count and divide at read time). log_stats is the in-repo model.

Neither query needs an author-written ORDER BY any more: this builder forwards columns to both sides, QueryMerger applies the sort to the merge query as a DataFusion logical-plan node, and the extract path applies it to the extract query the same way (see sql_partition_spec::plan_sorted_extract) – no SQL text is derived or rewritten on either side.

pub fn get_extract_query(&self) -> Arc<String>

The extract query as given to SqlBatchView::new, {begin}/{end} placeholders included.

pub fn get_merge_partitions_query(&self) -> Arc<String>

The merge query as given to SqlBatchView::new, {source} placeholder included.

Trait Implementations

impl Debug for SqlBatchView

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl View for SqlBatchView

fn get_view_set_name(&self) -> Arc<String>

name of the table from the user’s perspective

fn get_view_instance_id(&self) -> Arc<String>

get_view_instance_id can be a process_id, a stream_id or ‘global’.

fn make_batch_partition_spec<'life0, 'async_trait>( &'life0 self, lakehouse: Arc<LakehouseContext>, existing_partitions: Arc<PartitionCache>, insert_range: TimeRange, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn PartitionSpec>, Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, SqlBatchView: 'async_trait,

make_batch_partition_spec determines what should be found in an up to date partition. The resulting PartitionSpec can be used to validate existing partitions are create a new one.

fn get_file_schema_hash(&self) -> Vec<u8>

get_file_schema_hash returns a hash (can be a version number, version string, etc.) that allows to identify out of date partitions.

fn get_file_schema(&self) -> Arc<Schema>

get_file_schema returns the schema of the partition file in object storage

fn jit_update<'life0, 'async_trait>( &'life0 self, _lakehouse: Arc<LakehouseContext>, _query_range: Option<TimeRange>, ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, SqlBatchView: 'async_trait,

jit_update creates or updates process-specific partitions before a query

fn make_time_filter( &self, begin: DateTime<Utc>, end: DateTime<Utc>, ) -> Result<Vec<Expr>, Error>

make_time_filter returns a set of expressions that will filter out the rows of the partition outside the time range requested.

fn get_time_bounds(&self) -> Arc<dyn DataFrameTimeBounds>

fn register_table<'life0, 'life1, 'async_trait>( &'life0 self, ctx: &'life1 SessionContext, table: MaterializedView, ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, SqlBatchView: 'async_trait,

register the table in the SessionContext

fn merge_partitions<'life0, 'async_trait>( &'life0 self, lakehouse: Arc<LakehouseContext>, partitions_to_merge: Arc<Vec<Partition>>, partitions_all_views: Arc<PartitionCache>, insert_range: TimeRange, ) -> Pin<Box<dyn Future<Output = Result<MergeQueryResult, Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, SqlBatchView: 'async_trait,

fn get_merged_partition_sort_order( &self, partitions_to_merge: &[Partition], ) -> Option<Vec<String>>

Returns the sort guarantee a merge of partitions_to_merge will actually produce, to be recorded as the resulting partition’s Partition::sort_order (see merge::create_merged_partition).

fn get_scan_output_ordering(&self) -> ScanOrdering

Declares an ordering the view’s partition scan already emits, letting DataFusion elide redundant Sort nodes for queries that ORDER BY these columns.

fn get_update_group(&self) -> Option<i32>

tells the daemon which view should be materialized and in what order

fn get_max_partition_time_delta( &self, strategy: &PartitionCreationStrategy, ) -> TimeDelta

allow the view to subdivide the requested partition