randomSplit datatype related error splitting dataframe

Question

I have a pyspark dataframe named frozen_df. I'm trying to split it into two dataframes without any overlapping records. I'm using randomSplit() to split the dataframe. When I run it though I'm getting the error below. I have the schema of the dataframe below, along with the code and error. Does anyone see what the issue is and can you suggest how to fix it?

code:

frozen_df.schema

output:

StructType(List(StructField(product_num,StringType,true),StructField(product1,StringType,true),StructField(product2,StringType,true),StructField(product3,StringType,true),StructField(product4,StringType,true),StructField(productbrand,StringType,true),StructField(producttype,StringType,true),StructField(weight,StringType,true),StructField(unitofmeasure,StringType,true),StructField(productname,StringType,true),StructField(tz_brandname,StringType,true),StructField(size,StringType,true),StructField(package_size,StringType,true),StructField(classification,StringType,true),StructField(id,IntegerType,true)))

code:

splits=list(frozen_df.randomSplit([70,30],13))

error:

An error was encountered:
An error occurred while calling o102.randomSplit.
: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Double
    at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:114)
    at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:93)
    at scala.collection.IterableLike$class.copyToArray(IterableLike.scala:254)
    at scala.collection.AbstractIterable.copyToArray(Iterable.scala:54)
    at scala.collection.TraversableOnce$class.copyToArray(TraversableOnce.scala:278)
    at scala.collection.AbstractTraversable.copyToArray(Traversable.scala:104)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:286)
    at scala.collection.AbstractTraversable.toArray(Traversable.scala:104)
    at org.apache.spark.sql.Dataset.randomSplit(Dataset.scala:2130)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 918, in randomSplit
    rdd_array = self._jdf.randomSplit(_to_list(self.sql_ctx._sc, weights), long(seed))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o102.randomSplit.
: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Double
    at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:114)
    at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:93)
    at scala.collection.IterableLike$class.copyToArray(IterableLike.scala:254)
    at scala.collection.AbstractIterable.copyToArray(Iterable.scala:54)
    at scala.collection.TraversableOnce$class.copyToArray(TraversableOnce.scala:278)
    at scala.collection.AbstractTraversable.copyToArray(Traversable.scala:104)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:286)
    at scala.collection.AbstractTraversable.toArray(Traversable.scala:104)
    at org.apache.spark.sql.Dataset.randomSplit(Dataset.scala:2130)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

Answer 1

The weights you pass to randomSplit should add to 1.0 not 100 Pass [0.7, 0.3] instead of [70, 30].