CODEWITHSUNDEEP
amazon-redshiftsqoop
DataDog
DataDog

Reputation: 525

Sqoop import from redshift

Just as the title says, trying to move some data from Redshift to S3 via Sqoop:

sqoop-import -Dmapreduce.job.user.classpath.first=true --connect "jdbc:redshift://redshiftinstance.us-east-1.redshift.amazonaws.com:9999/stuffprd;database=ourDB;user=username;password=password;" --table ourtable -m 1 --as-avrodatafile --target-dir s3n://bucket/folder/folder1/

All drivers are in the proper folders however the error being throw is:

 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: No manager for connect string:

Upvotes: 1

Views: 1263

Answers (2)

Dave Whittingham
Dave Whittingham

Reputation: 338

Not sure if you already have got the answer to this, but you need to add the following to your sqoop command:

--driver com.amazon.redshift.jdbc42.Driver

--connection-manager org.apache.sqoop.manager.GenericJdbcManager

Upvotes: 1

Joe Harris
Joe Harris

Reputation: 14035

I can't help with the error but I recommend you not do it this way. Sqoop will try retrieve the table as SELECT * and all results will have to pass through the leader node. This will be much slower than using UNLOAD to export the data directly to S3 in parallel. You can then convert the unloaded text files to Avro using Sqoop.

Upvotes: 0

Related Questions