CODEWITHSUNDEEP
pythonsolrckan
user3366016
user3366016

Reputation: 1312

How to add custom dataset facet / solr filter in CKAN's Web UI?

Question:

How am I able to reproduce the result from the url below from the package_search API endpoint with CKAN's web UI?

https://demo.ckan.org/api/3/action/package_search?fq=num_resources:[1%20TO%20*]

I want to let a user filter by Packages with resources or without (0 num_resources or 1 to * num_resources).

What I've tried:

I've looked at and tried adding some facets and sorting already.

The facet allows filtering by Packages with X number of resources (e.g. 1). The sorting allows you to Sort all data sets by order of number of resources e.g. Packages with 10 resources first then 9 resources, then 8, etc...

I've tried to duplicate the API URI with the below

https://demo.ckan.org/dataset?num_resources=[1%20TO%20*]

If I add the fq portion it does not work either. The search() action will grab num_resources=[1 TO *] and append it to the solr fq param (can be seen here with log statements).

However, after trouble shooting this I've found that the CKAN Package controller's search() action DOES allow you to add values to the solr filter option fq like in the API call BUT that it first converts the params to a string num_resources:"[1 TO *]". This works fine to get a single value but not the range I'm after. If I use this exact param (with the quotes and not like the above url/api endpoint) with the API I get the incorrect result as well.

General Info:

UPDATES:

2018-12-20:

I have since found that q=num_resources:[1%20TO%20*] as a query string works as this is not escaped in the search() action. The q parameters are extracted before the encoding takes place.

However, this isn't ideal as it updates the search input and overrides any existing query strings unless you append to the query string and adding this to the filters so far is a pain.

# I've switched spaces to + to help with readability.
https://demo.ckan.org/dataset?q=num_resources:[1+TO+*]+Test

2018-12-21:

Working on implementing IPackageController from within an extension. This seems to be the proper way to go about this given the circumstances. Will add implementation after.

However, I feel an updated implementation of the params could be done in ckan's package_search

Turns out the search index and organization read are implemented quite a bit differently so the exact same implementation wont work. extra params are actually included as part of the q parameters instead of fq like search.

Upvotes: 4

Views: 1204

Answers (1)

user3366016
user3366016

Reputation: 1312

As per my last update, it appears the best way to approach this is through IPackageController from an extension and use before_search() to modify the search params.

This works well however, it would be nice if CKAN allowed a way to pass in additional fq filters on it's main search pages (dataset and organization ?fq=num_resources:[1 TO *] and append to fq). Also, it seems that datasets is slightly different in assigning params to fq than organizations. You can see this in these 2 lines from their actions (dataset search vs organization read). In my case I decided to only handle this for dataset searching.

Main pieces

# In plugin class implement IPackageController.
class CustomPlugin(plugins.SingletonPlugin, toolkit.DefaultDatasetForm):
    ...
    plugins.implements(plugins.IPackageController)


    # Lower in plugin class add needed functions from IPackageController, 
    # I decided to add them all and leave them untouched to avoid various 
    # errors I was getting.
    def before_search(self, search_params):
        u'''Extensions will receive a dictionary with the query parameters,
        and should return a modified (or not) version of it.

        Basically go over all search_params and look for values that contain my 
        additional filter and remove the double quotes. All fq values are a 
        single string, so do exact match to not remove other escaping / quotes.

        In query string in URL if you do `?num_resources=0` you get datasets with no
        resources (the opposite of this query string).
        '''
        for (param, value) in search_params.items():
            if param == 'fq' and 'num_resources:"[' in value:
                v = value.replace('num_resources:"[1 TO *]"', 'num_resources:[1 TO *]')
                search_params[param] = v

        return search_params

    def after_search(self, search_results, search_params):
        return search_results

    def before_index(self, pkg_dict):
        return pkg_dict

    def before_view(self, pkg_dict):
        return pkg_dict

    def read(self, entity):
        return entity

    def create(self, entity):
        return entity

    def edit(self, entity):
        return entity

    def delete(self, entity):
        return entity

    def after_create(self, context, pkg_dict):
        return pkg_dict

    def after_update(self, context, pkg_dict):
        return pkg_dict

    def after_delete(self, context, pkg_dict):
        return pkg_dict

    def after_show(self, context, pkg_dict):
        return pkg_dict

Then for the UI I added a custom facet list on the search.html template.

<div>
  <section class="module module-narrow module-shallow">
    {% block facet_list_heading %}
      <h2 class="module-heading">
        <i class="fa fa-filter"></i>
        {% set title = 'Resources (data)' %}
        {{ title }}
      </h2>
    {% endblock %}
    {% block facet_list_items %}
      {% set title = 'Has Resources (data)' %}
      <nav>
        <ul class="{{ nav_class or 'list-unstyled nav nav-simple nav-facet' }}">
          {% set href = h.remove_url_param('num_resources',
                                           extras=extras,
                                           alternative_url=alternative_url) 
                        if c.fields_grouped['num_resources'] 
                        else h.add_url_param(new_params={'num_resources': '[1 TO *]' },
                                            alternative_url=alternative_url) %}
          <li class="{{ nav_item_class or 'nav-item' }}{% if c.fields_grouped['num_resources'] %} active{% endif %}">
            <a href="{{ href }}" title="{{ title }}">
              <span>{{ title }}</span>
            </a>
          </li>
        </ul>
      </nav>
    {% endblock %}
  </section>
</div>

Doing it this way it doesn't add a new facet using IFacets as this adds a facet list for num_resources that gives the filter options of 0, 1, 2, 3, ... (or whatever makes sense for your setup e.g. if a dataset has 15 resources it shows that as an option).

I also did some modifications to the search_form.html snippet to get the facet filter to display how I wanted it but that was just extra.

Upvotes: 1

Related Questions