Elastalert Blacklist alert giving traceback errors

Question

In our ES, we have an alias for several indexes that we search regularly. One of the indexes covered holds our DNS logs.

If I search in Kibana or ES for:

Query: www.testbad.domain

I can get hits whether I'm searching in the aliased indexes or in the DNS index.

So I created the following rule:

# Test alert to check function

use_ssl: False

es_username: me
es_password: 12345

type: blacklist
    - "www.testbad.domain"
    - "!file /opt/elastalert/black_lists/bad_domains.txt"

index: alias-*

name: Detect-bad-domains

compare_key: Query

ignore_null: true

alert:
    - command

command: send_alert.sh

But when I test the alert, I get the following:

 Successfully loaded Detect-bad-domains

 Got 12345325 hits from the last 1 day

 Available terms in first hit: <list of terms which does NOT include Query>

 Warning: compare key %s is either missing or null! 
 Included term Query may be missing or null

 Traceback (most recent call last):  
 File "/usr/bin/elastalert-test-rule", line 11 in <module>
 load_entry_point('elastalert==0.1.29', 'console_scripts','elastalert-test-rule')()  
 File "/usr/lib/python2.7/site-packages/elastalert-0.1.29.py2.7.egg/elastalert/test_rule.py", line 378, in main  
 test_instance.run_rule_test()  
 File "/usr/lib/python2.7/site-packages/elastalert-0.1.29.py2.7.egg/elastalert/test_rule.py", line 373, in run_rule_test  
 self.run_elastalert(rule_yaml, conf, args_
 File "/usr/lib/python2.7/site-packages/elastalert-0.1.29.py2.7.egg/elastalert/test_rule.py", line 211, in run_elastalert  
 load_modules(rule, load_modules_args) 
 File "/usr/lib/python2.7/site-packages/elastalert-0.1.29.py2.7.egg/elastalert/config.py", line 341, in load_modules  
 rule['type']=get_module(rule['type']) 
 File "/usr/lib/python2.7/site-packages/elastalert-0.1.29.py2.7.egg/elastalert/config.py", line 100, in get_module  
 base_module = __import__(module_path, globals(), locals(), [module_class])  
 elastalert.util.EAException:
 Could not import module blacklist: - "www.testbad.domain": No module named blacklist = "www.testbad

If I delete the value www.testbad.domain from the list under the type: blacklist, I get the a differnet traceback error:

 Traceback (most recent call last): 
 File "/usr/bin/elastalert-test-rule", line 11 in <module>
 load_entry_point('elastalert==0.1.29', 'console_scripts','elastalert-test-rule')() 
 File "/usr/lib/python2.7/site-packages/elastalert-0.1.29.py2.7.egg/elastalert/test_rule.py", line 378, in main 
 test_instance.run_rule_test() 
 File "/usr/lib/python2.7/site-packages/elastalert-0.1.29.py2.7.egg/elastalert/test_rule.py", line 357, in run_rule_test 
 rule_yaml=load_rule_yaml(arg.file) 
 File "/usr/lib/python2.7/site-packages/elastalert-0.1.29.py2.7.egg/elastalert/config.py", line 128, in load_rule_yaml 
 loaded = yaml_loader(filename) 
 File "build/bdist.linux-x86_64/egg/staticconf/loader.py",line 93, in yaml_loader 
 File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 71, in load return 
 loader.get_single_data() 
 File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data 
 node = self.get_single_node() 
 File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node 
 document = self.compose_document() 
 File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 55, in compose_document 
 node = self.compose_document(None, None) 
 File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 84, in compose_node 
 node = self.compose_mapping_node(anchor) 
 File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 127, in compose_mapping_node 
 while not self.check_event(MappingEndEvent): 
 File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 98, in check_event 
 self.current_event = self.state() 
 File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 439, in parse_block_mapping_key 
 "expected <block end>, but found %r" % token.id, token.start_mark) 
 yaml.parser.ParserError: while parsing a block mapping 
 in "test-detect.yaml", line 3, column 1 
 expected <block end, but found '<block sequence start>' 
 in "test-detect.yaml", line 10, column 5

Then I tried changing to type any and specifying the query.

type: any

filter:
    - query:
        query_string:
            query: "Query: www.testbad.domain"

That produced no results, BUT "Query" was listed in the "Available terms in first hit"

So I tried: query: "Query: *www.testbad.domain*". Still no hits.

Then I tried: query: "Query: *testbad*". Now I get hits.

(I can't get it to show right, but both queries should be surrounded by asterisks with no spaces)

So my questions are these:

1. What is going on with the blacklist search? Why does python error out?
2. Why do I not get a hit when I search for the FQDN in the specific index for the query string?
3. Why isn't Query listed as a field I can search using the alias?

Answer 1

Gaaah!!! syntax, syntax, syntax.

type: blacklist
- "www.testbad.domain"
- "!file /opt/elastalert/black_lists/bad_domains.txt"

Should be:

type: blacklist

blacklist:
- "www.testbad.domain"
- "!file /opt/elastalert/black_lists/bad_domains.txt"