Implement Efficient File and Folder Structures – The Storage of DataImplement Efficient File and Folder Structures – The Storage of Data
df = spark.read.load(‘abfss://@.dfs.core.windows.net/in-path/file.csv’, df.write.mode(“overwrite”) \ df = spark.read.load(‘abfss://@.dfs.core.windows.net/out-path/file.parquet’, print(df.count()) from pyspark.sql.functions import year, month, col df = spark.read \ .load(‘abfss://@.dfs.core.windows.net/out-path/file.parquet’, format=’parquet’, header=True) df_year_month_day = (df.withColumn(“year”, year(col(“SESSION_DATETIME”)))) \ .withColumn(“month”, month(col(“SESSION_DATETIME”))) from [...]