CODEWITHSUNDEEP
hiveamazon-emrpresto
nish
nish

Reputation: 7280

Setting hive configuration properties for Presto in EMR

I'm trying to update hive connector properties inside /etc/presto/conf/catalog/hive.properties. I added the following:

hive.s3.max-client-retries=50
hive.s3.max-error-retries=50
hive.s3.max-connections=500
hive.s3.connect-timeout=5m
hive.s3.socket-timeout=5m

But for some reason these are not accepted. The presto server restart fails with the following error:

2018-04-16T19:34:26.369Z    WARN    main    Bootstrap   UNUSED PROPERTIES
2018-04-16T19:34:26.369Z    WARN    main    Bootstrap   hive.s3.connect-timeout
2018-04-16T19:34:26.369Z    WARN    main    Bootstrap   hive.s3.max-client-retries
2018-04-16T19:34:26.369Z    WARN    main    Bootstrap   hive.s3.max-connections
2018-04-16T19:34:26.369Z    WARN    main    Bootstrap   hive.s3.max-error-retries
2018-04-16T19:34:26.369Z    WARN    main    Bootstrap   hive.s3.socket-timeout
2018-04-16T19:34:26.369Z    WARN    main    Bootstrap   
2018-04-16T19:34:26.587Z    ERROR   main    com.facebook.presto.server.PrestoServer Unable to create injector, see the following errors:

1) Configuration property 'hive.s3.connect-timeout' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

2) Configuration property 'hive.s3.max-client-retries' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

3) Configuration property 'hive.s3.max-connections' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

4) Configuration property 'hive.s3.max-error-retries' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

5) Configuration property 'hive.s3.socket-timeout' was not used
  at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:234)

5 errors

Upvotes: 1

Views: 2918

Answers (2)

zyork
zyork

Reputation: 65

As Piotr mentioned, EMR changed the default FileSystem implementation starting with emr-5.12.0 from PrestoS3FileSystem to EmrFS. PrestoS3FileSystem properties will be unavailable, but you can configure EmrFS as you would for any other application. See 'EMRFS and PrestoS3FileSystem Configuration' on https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-presto-considerations.html for more information.

Upvotes: 2

Piotr Findeisen
Piotr Findeisen

Reputation: 20770

These properties are applicable when hive.s3-file-system-type is set to PRESTO. When it's eg. EMRFS, these configuration properties are not applicable and setting them causes the error you observe.

Upvotes: 3

Related Questions