Writing Data to Amorphic Datalake

Following are some examples on how to read using amorphicutils.

1. Python shell

1.1. Writing CSV data

1.1.1 Writing full dataset

2. Spark

2.1. Writing CSV data

2.1.1 Writing to dataset

2.2. Writing Parquet data

2.2.1 Writing to dataset

from amorphicutils.pyspark.write import Write
from amorphicutils.pyspark.infra.gluespark. import GlueSpark

lz_bucket=""
domain=""
dataset=""
user=""

glue_spark = GlueSpark()
glue_context = glue_spark.get_glue_context()

csv_writer = Write(bucket_name=lz_bucket,
                   spark=glue_context)
result = csv_writer.write_parquet(df, domain, dataset, user)

If the dataset is of reload type then pass full_reload=True as parameter to the write_parquet method