Chapter 3 discussed designing partition strategies. Remember the two primary reasons for partitioning data. The first reason is to improve the speed at which a query for data responds with [...]
Chapter 7, “Design and Implement a Data Stream Processing Solution,” discusses partitioning data within one partition and across partitions. Exercise 7.5 features the hands‐on implementation of partitioning streaming workloads. Partitioning [...]
In Exercise 4.7 you performed a conversion of brain waves stored in multiple CSV files using the following PySpark code snippet: %%pysparkdf = spark.read.option(“header”,”true”) \ .csv(‘abfss://*@*.dfs.core.windows.net/EMEA/brainjammer/in/2022/04/01/18/*’)display(df.limit(10)) Then you wrote that [...]
Unlike physical data storage structures, you cannot actually touch logical storage structures. That is because logical storage structures are arrangements like blocks, extents, segments, and tablespaces that reside on the [...]
The serving layer is a component of the lambda architecture (refer to Figure 3.13). The serving layer is the place into which both the speed layer, which processes data incrementally, [...]
A flowlet is a container that holds reusable activities. Consider the activity you created in Exercise 4.8, which retrieves brain wave data and converts the epoch date into a more [...]
When you select the Source item, six tabs are rendered in the configuration panel: Source Settings, Source Options, Projection, Optimize, Inspect, and Data Preview. Figure 4.25 shows the Source Settings [...]
df = spark.read.option(“header”,”true”) \ .csv(‘abfss://@.dfs.core.windows.net/EMEA/brainjammer/in/2022/04/01/18/*’) display(df.limit(10)) FIGURE 4.21 Azure Synapse Analytics Develop hub load Notebook FIGURE 4.22 Azure Synapse Analytics Develop hub write Notebook Parquet files In this exercise you [...]
Your Azure Databricks workspace includes an associated Azure storage account and blob container. Two lifecycle management policies that delete temporary and log files are created by default. You can view [...]
FIGURE 4.19 Azure Synapse Analytics Data hub ADLS directory FROM OPENROWSET( ) AS [result] FIGURE 4.20 Azure Synapse Analytics Data hub ADLS directory Notice in Figure 4.20 that the SQL [...]