CODEWITHSUNDEEP
jdbcamazon-redshiftaws-gluecatalog
Abhi5421
Abhi5421

Reputation: 11

writing data from aws glue catalog to redshift

scenario

I have glue catalog and I need to load this data from catalog to redshift.

dyf = glueContext.create_dynamic_frame.from_catalog(
            database=catalog_db,
            table_name=f"{table}",
            push_down_predicate=f"day = {day}"
        )
glueContext.write_dynamic_frame.from_jdbc_conf(
            frame=DynamicFrame.fromDF(df, glueContext, "trx_df"),
            catalog_connection="redshift_connection",
            connection_options={
                "database": database,
                "dbtable": f"{schema}.{table}"
            },
            redshift_tmp_dir= "s3://"

using this i am writing, redshift_connection this my redshift connection which is my redshift established connection on aws connection.it loading data daily

Requirement:

  1. I have some dimensions tables and daily before loading need to truncate table and load (consider it like some operation i need to do before loading and after loading).
  2. Instead of using this established connection can i use password, user with jdbc url to load, (which is better)
  3. Even i need see one more option like schema creation upfront before loading if possible, as of now above code it loading table and creating table schema also

Upvotes: 0

Views: 53

Answers (0)

Related Questions