Hadoop Remote Exception Error on minReplication of datanodes

Question

We are running our data processing Spark/Scala application on a 12 node EMR cluster. There are multiple stages during processing where we read and write the files to and from HDFS. Most of the time(99%) the data is being processed smoothly without any issues. But for some specific set of files pertaining some specific days ( like May5th or Apr 26th etc.) we are getting the following error:

2018-10-25 19:21:16,958 [task-result-getter-3] WARN  org.apache.spark.scheduler.TaskSetManager:66 - Lost task 221.0 in stage 31.0 (TID 4910, ip-00-00-00-00.companyname.net, executor 4): org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/hadoop/working/fileType/CBO_33019.txt.2ea3ae69-8efd-eeba-4526-6511b0752fdb.fileType.xml.__temp could only be replicated to 0 nodes instead of minReplication (=1).  There are 11 datanode(s) running and no node(s) are excluded in this operation.

at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1735)
    at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:265)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2561)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:829)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)

    at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
    at org.apache.hadoop.ipc.Client.call(Client.java:1435)
    at org.apache.hadoop.ipc.Client.call(Client.java:1345)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
    at com.sun.proxy.$Proxy38.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:444)
    at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
    at com.sun.proxy.$Proxy39.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1838)
    at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1638)
    at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:704)

Can someone please help me decode this error? Most of the online help are falling into following 2 categories:

1. There are 0 datanode(s) running and no node(s) are excluded in this operation. or
2. There are 3 datanode(s) running and 3 node(s) are excluded in this operation.

But ours is a different scenario (see below) for which there is no such example anywhere in online forums.

There are 11 datanode(s) running and no node(s) are excluded in this operation.

What can we do to fix this error. Any help is much appreciated because this looks like a one of kind issue and might help many others out there. Thanks.