SparkNLP messages disabled

Question

When I run the code "spark = sparknlp.start(), it always returns such a message to the terminal, which is very annoying.

23/07/13 14:23:33 WARN Utils: Your hostname, ---------- resolves to a loopback address: -------; using 192.168.28.181 instead (on interface eth0)
23/07/13 14:23:33 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
:: loading settings :: url = jar:file:/home/------/venv/lib/python3.8/site-packages/pyspark/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/-----/.ivy2/cache
The jars for the packages stored in: /home/-----/.ivy2/jars
com.johnsnowlabs.nlp#spark-nlp_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-f555c3ee-fa03-4bec-81c2-51839abf785a;1.0
        confs: [default]
        found com.johnsnowlabs.nlp#spark-nlp_2.12;4.4.4 in central
        found com.typesafe#config;1.4.2 in central
        found org.rocksdb#rocksdbjni;6.29.5 in central
        found com.amazonaws#aws-java-sdk-bundle;1.11.828 in central
        found com.github.universal-automata#liblevenshtein;3.0.0 in central
        found com.google.protobuf#protobuf-java-util;3.0.0-beta-3 in central
        found com.google.protobuf#protobuf-java;3.0.0-beta-3 in central
        found com.google.code.gson#gson;2.3 in central
        found it.unimi.dsi#fastutil;7.0.12 in central
        found org.projectlombok#lombok;1.16.8 in central
        found com.google.cloud#google-cloud-storage;2.16.0 in central
        found com.google.guava#guava;31.1-jre in central
        found com.google.guava#failureaccess;1.0.1 in central
        found com.google.guava#listenablefuture;9999.0-empty-to-avoid-conflict-with-guava in central
        found com.google.errorprone#error_prone_annotations;2.16 in central
        found com.google.j2objc#j2objc-annotations;1.3 in central
        found com.google.http-client#google-http-client;1.42.3 in central
        found io.opencensus#opencensus-contrib-http-util;0.31.1 in central
        found com.google.http-client#google-http-client-jackson2;1.42.3 in central
        found com.google.http-client#google-http-client-gson;1.42.3 in central
        found com.google.api-client#google-api-client;2.1.1 in central
        found commons-codec#commons-codec;1.15 in central
        found com.google.oauth-client#google-oauth-client;1.34.1 in central
        found com.google.http-client#google-http-client-apache-v2;1.42.3 in central
        found com.google.apis#google-api-services-storage;v1-rev20220705-2.0.0 in central
        found com.google.code.gson#gson;2.10 in central
        found com.google.cloud#google-cloud-core;2.9.0 in central
        found com.google.auto.value#auto-value-annotations;1.10.1 in central
        found com.google.cloud#google-cloud-core-http;2.9.0 in central
        found com.google.http-client#google-http-client-appengine;1.42.3 in central
        found com.google.api#gax-httpjson;0.105.1 in central
        found com.google.cloud#google-cloud-core-grpc;2.9.0 in central
        found io.grpc#grpc-core;1.51.0 in central
        found com.google.api#gax;2.20.1 in central
        found com.google.api#gax-grpc;2.20.1 in central
        found io.grpc#grpc-alts;1.51.0 in central
        found io.grpc#grpc-grpclb;1.51.0 in central
        found org.conscrypt#conscrypt-openjdk-uber;2.5.2 in central
        found io.grpc#grpc-protobuf;1.51.0 in central
        found com.google.auth#google-auth-library-credentials;1.13.0 in central
        found com.google.auth#google-auth-library-oauth2-http;1.13.0 in central
        found com.google.api#api-common;2.2.2 in central
        found javax.annotation#javax.annotation-api;1.3.2 in central
        found io.opencensus#opencensus-api;0.31.1 in central
        found io.grpc#grpc-context;1.51.0 in central
        found com.google.api.grpc#proto-google-iam-v1;1.6.22 in central
        found com.google.protobuf#protobuf-java;3.21.10 in central
        found com.google.protobuf#protobuf-java-util;3.21.10 in central
        found com.google.api.grpc#proto-google-common-protos;2.11.0 in central
        found org.threeten#threetenbp;1.6.4 in central
        found com.google.api.grpc#proto-google-cloud-storage-v2;2.16.0-alpha in central
        found com.google.api.grpc#grpc-google-cloud-storage-v2;2.16.0-alpha in central
        found com.google.api.grpc#gapic-google-cloud-storage-v2;2.16.0-alpha in central
        found com.fasterxml.jackson.core#jackson-core;2.14.1 in central
        found com.google.code.findbugs#jsr305;3.0.2 in central
        found io.grpc#grpc-api;1.51.0 in central
        found io.grpc#grpc-auth;1.51.0 in central
        found io.grpc#grpc-stub;1.51.0 in central
        found org.checkerframework#checker-qual;3.28.0 in central
        found com.google.api.grpc#grpc-google-iam-v1;1.6.22 in central
        found io.grpc#grpc-protobuf-lite;1.51.0 in central
        found com.google.android#annotations;4.1.1.4 in central
        found org.codehaus.mojo#animal-sniffer-annotations;1.22 in central
        found io.grpc#grpc-netty-shaded;1.51.0 in central
        found io.perfmark#perfmark-api;0.26.0 in central
        found io.grpc#grpc-googleapis;1.51.0 in central
        found io.grpc#grpc-xds;1.51.0 in central
        found io.opencensus#opencensus-proto;0.2.0 in central
        found io.grpc#grpc-services;1.51.0 in central
        found com.google.re2j#re2j;1.6 in central
        found com.navigamez#greex;1.0 in central
        found dk.brics.automaton#automaton;1.11-8 in central
        found com.johnsnowlabs.nlp#tensorflow-cpu_2.12;0.4.4 in central
:: resolution report :: resolve 1603ms :: artifacts dl 638ms
        :: modules in use:
        com.amazonaws#aws-java-sdk-bundle;1.11.828 from central in [default]
        com.fasterxml.jackson.core#jackson-core;2.14.1 from central in [default]
        com.github.universal-automata#liblevenshtein;3.0.0 from central in [default]
        com.google.android#annotations;4.1.1.4 from central in [default]
        com.google.api#api-common;2.2.2 from central in [default]
        com.google.api#gax;2.20.1 from central in [default]
        com.google.api#gax-grpc;2.20.1 from central in [default]
        com.google.api#gax-httpjson;0.105.1 from central in [default]
        com.google.api-client#google-api-client;2.1.1 from central in [default]
        com.google.api.grpc#gapic-google-cloud-storage-v2;2.16.0-alpha from central in [default]
        com.google.api.grpc#grpc-google-cloud-storage-v2;2.16.0-alpha from central in [default]
        com.google.api.grpc#grpc-google-iam-v1;1.6.22 from central in [default]
        com.google.api.grpc#proto-google-cloud-storage-v2;2.16.0-alpha from central in [default]
        com.google.api.grpc#proto-google-common-protos;2.11.0 from central in [default]
        com.google.api.grpc#proto-google-iam-v1;1.6.22 from central in [default]
        com.google.apis#google-api-services-storage;v1-rev20220705-2.0.0 from central in [default]
        com.google.auth#google-auth-library-credentials;1.13.0 from central in [default]
        com.google.auth#google-auth-library-oauth2-http;1.13.0 from central in [default]
        com.google.auto.value#auto-value-annotations;1.10.1 from central in [default]
        com.google.cloud#google-cloud-core;2.9.0 from central in [default]
        com.google.cloud#google-cloud-core-grpc;2.9.0 from central in [default]
        com.google.cloud#google-cloud-core-http;2.9.0 from central in [default]
        com.google.cloud#google-cloud-storage;2.16.0 from central in [default]
        com.google.code.findbugs#jsr305;3.0.2 from central in [default]
        com.google.code.gson#gson;2.10 from central in [default]
        com.google.errorprone#error_prone_annotations;2.16 from central in [default]
        com.google.guava#failureaccess;1.0.1 from central in [default]
        com.google.guava#guava;31.1-jre from central in [default]
        com.google.guava#listenablefuture;9999.0-empty-to-avoid-conflict-with-guava from central in [default]
        com.google.http-client#google-http-client;1.42.3 from central in [default]
        com.google.http-client#google-http-client-apache-v2;1.42.3 from central in [default]
        com.google.http-client#google-http-client-appengine;1.42.3 from central in [default]
        com.google.http-client#google-http-client-gson;1.42.3 from central in [default]
        com.google.http-client#google-http-client-jackson2;1.42.3 from central in [default]
        com.google.j2objc#j2objc-annotations;1.3 from central in [default]
        com.google.oauth-client#google-oauth-client;1.34.1 from central in [default]
        com.google.protobuf#protobuf-java;3.21.10 from central in [default]
        com.google.protobuf#protobuf-java-util;3.21.10 from central in [default]
        com.google.re2j#re2j;1.6 from central in [default]
        com.johnsnowlabs.nlp#spark-nlp_2.12;4.4.4 from central in [default]
        com.johnsnowlabs.nlp#tensorflow-cpu_2.12;0.4.4 from central in [default]
        com.navigamez#greex;1.0 from central in [default]
        com.typesafe#config;1.4.2 from central in [default]
        commons-codec#commons-codec;1.15 from central in [default]
        dk.brics.automaton#automaton;1.11-8 from central in [default]
        io.grpc#grpc-alts;1.51.0 from central in [default]
        io.grpc#grpc-api;1.51.0 from central in [default]
        io.grpc#grpc-auth;1.51.0 from central in [default]
        io.grpc#grpc-context;1.51.0 from central in [default]
        io.grpc#grpc-core;1.51.0 from central in [default]
        io.grpc#grpc-googleapis;1.51.0 from central in [default]
        io.grpc#grpc-grpclb;1.51.0 from central in [default]
        io.grpc#grpc-netty-shaded;1.51.0 from central in [default]
        io.grpc#grpc-protobuf;1.51.0 from central in [default]
        io.grpc#grpc-protobuf-lite;1.51.0 from central in [default]
        io.grpc#grpc-services;1.51.0 from central in [default]
        io.grpc#grpc-stub;1.51.0 from central in [default]
        io.grpc#grpc-xds;1.51.0 from central in [default]
        io.opencensus#opencensus-api;0.31.1 from central in [default]
        io.opencensus#opencensus-contrib-http-util;0.31.1 from central in [default]
        io.opencensus#opencensus-proto;0.2.0 from central in [default]
        io.perfmark#perfmark-api;0.26.0 from central in [default]
        it.unimi.dsi#fastutil;7.0.12 from central in [default]
        javax.annotation#javax.annotation-api;1.3.2 from central in [default]
        org.checkerframework#checker-qual;3.28.0 from central in [default]
        org.codehaus.mojo#animal-sniffer-annotations;1.22 from central in [default]
        org.conscrypt#conscrypt-openjdk-uber;2.5.2 from central in [default]
        org.projectlombok#lombok;1.16.8 from central in [default]
        org.rocksdb#rocksdbjni;6.29.5 from central in [default]
        org.threeten#threetenbp;1.6.4 from central in [default]
        :: evicted modules:
        com.google.protobuf#protobuf-java-util;3.0.0-beta-3 by [com.google.protobuf#protobuf-java-util;3.21.10] in [default]
        com.google.protobuf#protobuf-java;3.0.0-beta-3 by [com.google.protobuf#protobuf-java;3.21.10] in [default]
        com.google.code.gson#gson;2.3 by [com.google.code.gson#gson;2.10] in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   73  |   0   |   0   |   3   ||   70  |   0   |
        ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-f555c3ee-fa03-4bec-81c2-51839abf785a
        confs: [default]
        0 artifacts copied, 70 already retrieved (0kB/37ms)
23/07/13 14:23:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

I thought it is the log level issues, so I set the log level to "OFF" by the following commands:

# Set the app name
conf = SparkConf().setAppName("MyApp")

# Create the SparkContext
sc = SparkContext(conf=conf)

sc.setLogLevel("ALL")

It then invokes the error message:

# File "/home/-----/base_extractor.py", line 167, in sparknlp_zero_shot_ner
    document_assembler = DocumentAssembler().setInputCol("text").setOutputCol("document")
  File "/home/-----/venv/lib/python3.8/site-packages/pyspark/__init__.py", line 135, in wrapper
    return func(self, **kwargs)
  File "/home/-----/venv/lib/python3.8/site-packages/sparknlp/base/document_assembler.py", line 96, in __init__
    super(DocumentAssembler, self).__init__(classname="com.johnsnowlabs.nlp.DocumentAssembler")
  File "/home/-----/venv/lib/python3.8/site-packages/pyspark/__init__.py", line 135, in wrapper
    return func(self, **kwargs)
  File "/home/-------/venv/lib/python3.8/site-packages/sparknlp/internal/annotator_transformer.py", line 36, in __init__
    self._java_obj = self._new_java_obj(classname, self.uid)
  File "/home/-------/venv/lib/python3.8/site-packages/pyspark/ml/wrapper.py", line 86, in _new_java_obj
    return java_obj(*java_args)
TypeError: 'JavaPackage' object is not callable

I am trying to utilize the zero_shot_ner model to extract some words, but after changing the logging level it completely crashes and I have no idea what is happening. May I know if there are any ways to disable the annoying without any error. Thank you.

I expect to disable all the messages returned by the start-up of spark-nlp without altering the spark-nlp setup and causing any error in the main program.