RELX:
delta tables quetions :
Q) query to merge data from a table to b:
merge source a table into target b
on [Link] = [Link]
when matched
update
[Link] = [Link]
[Link] = [Link]
Not Matched
insert into target
Q) create external table with externam location path :
create external table employee (first_name varchar(10) last_name varchar(10)
address varchar(100) date (date) )
partion by date
format(parquet)
location(adls/path/of/file)
Q) merge source a table into target b
on [Link] = [Link] and a.first_name = b.first_name
when matched
update
[Link] = [Link]
[Link] = [Link]
Not Matched
insert into target
Q)append new csv file data into table
source_df = [Link](csv).path(path/of/source_file)
result_df = [Link](parquet).path(path/of/result_file)
desti_df = source_df.union(result_df)
dest_df.saveastable(tablename)
Q) append new csv file data into table using append mode :
[Link]("append).format(parquet).option(path,"path/of/
file").saveasatable(tablename)
Therory Quetions :
save data in only one file (ans use coalese(1) while saving dataframe )
How to optimize the performance of PySpark jobs?
techniques I use to handle skewed data in PySpark?
concepts of transformations and actions in PySpark?
how to check data quality and integrity when processing large datasets with PySpark
invoke one notebook from another in Databricks?
Query for c reating internal and external tables in Databricks?
Dataflow end to end :
Agile methodology process :
About team : How many member are in team and work of each member and workflow