Struct SqlBatchView 

pub struct SqlBatchView { /* private fields */ }

SQL-defined view updated in batch

Implementations

impl SqlBatchView

pub async fn new( runtime: Arc<RuntimeEnv>, view_set_name: Arc<String>, min_event_time_column: Arc<String>, max_event_time_column: Arc<String>, count_src_query: Arc<String>, extract_query: Arc<String>, merge_partitions_query: Arc<String>, lake: Arc<DataLakeConnection>, view_factory: Arc<ViewFactory>, session_configurator: Arc<dyn SessionConfigurator>, update_group: Option<i32>, max_partition_delta_from_source: TimeDelta, max_partition_delta_from_merge: TimeDelta, merger_maker: Option<&(dyn Fn(Arc<RuntimeEnv>, Arc<Schema>) -> Arc<dyn PartitionMerger> + Send + Sync + 'static)>, ) -> Result<SqlBatchView, Error>

Arguments

• runtime - datafusion runtime
• view_set_name - name of the table
• min_event_time_column - min(column) should result in the first timestamp in a dataframe
• max_event_time_column - max(column) should result in the last timestamp in a dataframe
• count_src_query - used to count the rows of the underlying data to know if a cached partition is up to date
• extract_query - used to extract the source data into a cached partition
• merge_partitions_query - used to merge multiple partitions into a single one (and user queries which are one multiple partitions by default)
• lake - data lake
• view_factory - all views accessible to the count_src_query
• session_configurator - configurator for registering custom tables (e.g., JSON files)
• update_group - tells the daemon which view should be materialized and in what order

Trait Implementations

impl Debug for SqlBatchView

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl View for SqlBatchView

fn get_view_set_name(&self) -> Arc<String>

name of the table from the user’s perspective

fn get_view_instance_id(&self) -> Arc<String>

get_view_instance_id can be a process_id, a stream_id or ‘global’.

fn make_batch_partition_spec<'life0, 'async_trait>( &'life0 self, lakehouse: Arc<LakehouseContext>, existing_partitions: Arc<PartitionCache>, insert_range: TimeRange, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn PartitionSpec>, Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, SqlBatchView: 'async_trait,

make_batch_partition_spec determines what should be found in an up to date partition. The resulting PartitionSpec can be used to validate existing partitions are create a new one.

fn get_file_schema_hash(&self) -> Vec<u8>

get_file_schema_hash returns a hash (can be a version number, version string, etc.) that allows to identify out of date partitions.

fn get_file_schema(&self) -> Arc<Schema>

get_file_schema returns the schema of the partition file in object storage

fn jit_update<'life0, 'async_trait>( &'life0 self, _lakehouse: Arc<LakehouseContext>, _query_range: Option<TimeRange>, ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, SqlBatchView: 'async_trait,

jit_update creates or updates process-specific partitions before a query

fn make_time_filter( &self, begin: DateTime<Utc>, end: DateTime<Utc>, ) -> Result<Vec<Expr>, Error>

make_time_filter returns a set of expressions that will filter out the rows of the partition outside the time range requested.

fn get_time_bounds(&self) -> Arc<dyn DataFrameTimeBounds>

fn register_table<'life0, 'life1, 'async_trait>( &'life0 self, ctx: &'life1 SessionContext, table: MaterializedView, ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, SqlBatchView: 'async_trait,

register the table in the SessionContext

fn merge_partitions<'life0, 'async_trait>( &'life0 self, lakehouse: Arc<LakehouseContext>, partitions_to_merge: Arc<Vec<Partition>>, partitions_all_views: Arc<PartitionCache>, insert_range: TimeRange, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch, DataFusionError>> + Send>>, Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, SqlBatchView: 'async_trait,

fn get_update_group(&self) -> Option<i32>

tells the daemon which view should be materialized and in what order

fn get_max_partition_time_delta( &self, strategy: &PartitionCreationStrategy, ) -> TimeDelta

allow the view to subdivide the requested partition