This section will help you increase your knowledge of two of Azure open‐source products and features, as well as provide a summary of the storage of data along the Big [...]
Congratulations on creating your first Azure Synapse Analytics pipeline. Reflect a bit on all the features it relied on and how much you have learned about them so far. You [...]
When you begin to brainstorm the storage of data for an analytical workload, terms such as hybrid transaction/analytical processing (HTAP) and online analytical processing (OLAP) might come to mind. Both [...]
DROP TABLE brainwaves.DimELECTRODE 2. Create an SCD table, and then execute the following SQL script, which is located in the folder Chapter04/Ch04Ex09 on GitHub at https://github.com/benperk/ADE and named createSlowlyChangingDimensionTable.sql: CREATE [...]
COLLATE Latin1_General_100_BIN2_UTF8 WITH (LOCATION = ‘abfss://@.dfs.core.windows.net’) WITH (FORMAT_TYPE = PARQUET) ([Timestamp] NVARCHAR(50),[AF3theta] NVARCHAR(50),[AF3alpha] NVARCHAR(50),[AF3betaL] NVARCHAR(50),…) WITH(LOCATION = ‘EMEA/brainjammer/out/2022/04/03//.parquet/*’,DATA_SOURCE = SampleBrainwavesSource,FILE_FORMAT = SampleBrainwavesParquet) FIGURE 4.33 Building an external table You might [...]
Unlike physical data storage structures, you cannot actually touch logical storage structures. That is because logical storage structures are arrangements like blocks, extents, segments, and tablespaces that reside on the [...]
When you select the Source item, six tabs are rendered in the configuration panel: Source Settings, Source Options, Projection, Optimize, Inspect, and Data Preview. Figure 4.25 shows the Source Settings [...]
df = spark.read.option(“header”,”true”) \ .csv(‘abfss://@.dfs.core.windows.net/EMEA/brainjammer/in/2022/04/01/18/*’) display(df.limit(10)) FIGURE 4.21 Azure Synapse Analytics Develop hub load Notebook FIGURE 4.22 Azure Synapse Analytics Develop hub write Notebook Parquet files In this exercise you [...]
Your Azure Databricks workspace includes an associated Azure storage account and blob container. Two lifecycle management policies that delete temporary and log files are created by default. You can view [...]
FIGURE 4.19 Azure Synapse Analytics Data hub ADLS directory FROM OPENROWSET( ) AS [result] FIGURE 4.20 Azure Synapse Analytics Data hub ADLS directory Notice in Figure 4.20 that the SQL [...]