D. R
D. R

Reputation: 11

How does GPG Encryption/Decryption work on Spark?

I am doing GPG/PGP encryption/Decryption on spark. I have some large files. Sizes up to 30 GB . I am using a spark cluster with following configuration -

  1. Worker Node - 16 GB , 4 Core . Min 2 , Max 8 Workers
  2. Driver Node - 16 GB , 4 Core

MY Question is -> How does spark handle encryption decryption when the file is large and can not fit in single nodes memory ? Spark is Optimised for parallel processing , especially for tasks that can be divided in chunks , Whereas Encryption/Decryption tends to be sequential operation and cant be optimised with parallel process. So how does spark manage to decrypt a file larger than its memory ?

  1. I looked into some online articles but did not get any good answer

Upvotes: 1

Views: 52

Answers (0)

Related Questions