<<

NAME

Koha::SearchEngine::Elasticsearch - Base module for things using elasticsearch

ACCESSORS

index

The name of the index to use, generally 'biblios' or 'authorities'.

index_name

The Elasticsearch index name with Koha instance prefix.

FUNCTIONS

get_elasticsearch

    my $elasticsearch_client = $self->get_elasticsearch();

Returns a Search::Elasticsearch client. The client is cached on a Koha::SearchEngine::ElasticSearch instance level and will be reused if method is called multiple times.

get_elasticsearch_params

    my $params = $self->get_elasticsearch_params();

This provides a hashref that contains the parameters for connecting to the ElasicSearch servers, in the form:

    {
        'nodes' => ['127.0.0.1:9200', 'anotherserver:9200'],
        'index_name' => 'koha_instance_index',
    }

This is configured by the following in the config block in koha-conf.xml:

    <elasticsearch>
        <server>127.0.0.1:9200</server>
        <server>anotherserver:9200</server>
        <index_name>koha_instance</index_name>
    </elasticsearch>

get_elasticsearch_settings

    my $settings = $self->get_elasticsearch_settings();

This provides the settings provided to Elasticsearch when an index is created. These can do things like define tokenization methods.

A hashref containing the settings is returned.

get_elasticsearch_mappings

    my $mappings = $self->get_elasticsearch_mappings();

This provides the mappings that get passed to Elasticsearch when an index is created.

raw_elasticsearch_mappings

Return elasticsearch mapping as it is in database. marc_type: marc21|unimarc

$raw_mappings = raw_elasticsearch_mappings( $marc_type )

_get_elasticsearch_field_config

Get the Elasticsearch field config for the given purpose and data type.

$mapping = _get_elasticsearch_field_config('search', 'text');

_load_elasticsearch_mappings

Load Elasticsearch mappings in the format of mappings.yaml.

$indexes = _load_elasticsearch_mappings();

_process_mappings($mappings, $data, $record_document, $meta)

    $self->_process_mappings($mappings, $marc_field_data, $record_document, 0)

Process all $mappings targets operating on a specific MARC field $data. Since we group all mappings by MARC field targets $mappings will contain all targets for $data and thus we need to fetch the MARC field only once. $mappings will be applied to $record_document and new field values added. The method has no return value.

$mappings

Arrayref of mappings containing arrayrefs in the format [$target, $options] where $target is the name of the target field and $options is a hashref containing processing directives for this particular mapping.

$data

The source data from a MARC record field.

$record_document

Hashref representing the Elasticsearch document on which mappings should be applied.

$meta

A hashref containing metadata useful for enforcing per mapping rules. For example for providing extra context for mapping options, or treating mapping targets differently depending on type (sort, search, facet etc). Combining this metadata with the mapping options and metadata allows us to mutate the data per mapping, or even replace it with other data retrieved from the metadata context.

Current properties are:

altscript: A boolean value indicating whether an alternate script presentation is being processed.

data_source: The source of the $<data> argument. Possible values are: 'leader', 'control_field', 'subfield' or 'subfields_group'.

code: The code of the subfield $data was retrieved, if data_source is 'subfield'.

codes: Subfield codes of the subfields group from which $data was retrieved, if data_source is 'subfields_group'.

field: The original MARC::Record object.

marc_records_to_documents($marc_records)

    my $record_documents = $self->marc_records_to_documents($marc_records);

Using mappings stored in database convert $marc_records to Elasticsearch documents.

Returns array of hash references, representing Elasticsearch documents, acceptable as body payload in Search::Elasticsearch requests.

$marc_documents

Reference to array of MARC::Record objects to be converted to Elasticsearch documents.

_marc_to_array($record)

    my @fields = _marc_to_array($record)

Convert a MARC::Record to an array modeled after MARC-in-JSON (see https://github.com/marc4j/marc4j/wiki/MARC-in-JSON-Description)

$record

A MARC::Record object

_array_to_marc($data)

    my $record = _array_to_marc($data)

Convert an array modeled after MARC-in-JSON to a MARC::Record

$data

An array modeled after MARC-in-JSON (see https://github.com/marc4j/marc4j/wiki/MARC-in-JSON-Description)

_field_mappings($facet, $suggestible, $sort, $search, $filter, $target_name, $target_type, $range)

    my @mappings = _field_mappings( $facet, $suggestible, $sort, $search, $filter, $target_name, $target_type, $range )

Get mappings, an internal data structure later used by "_process_mappings($mappings, $data, $record_document, $meta)" to process MARC target data for a MARC mapping.

The returned $mappings is not to to be confused with mappings provided by _foreach_mapping, rather this sub accepts properties from a mapping as provided by _foreach_mapping and expands it to this internal data structure. In the caller context (_get_marc_mapping_rules) the returned @mappings is then applied to each MARC target (leader, control field data, subfield or joined subfields) and integrated into the mapping rules data structure used in marc_records_to_documents to transform MARC records into Elasticsearch documents.

$facet

Boolean indicating whether to create a facet field for this mapping.

$suggestible

Boolean indicating whether to create a suggestion field for this mapping.

$sort

Boolean indicating whether to create a sort field for this mapping.

$search

Boolean indicating whether to create a search field for this mapping.

$target_name

Elasticsearch document target field name.

$target_type

Elasticsearch document target field type.

$range

An optional range as a string in the format "<START>-<END>" or "<START>", where "<START>" and "<END>" are integers specifying a range that will be used for extracting a substring from MARC data as Elasticsearch field target value.

The first character position is "0", and the range is inclusive, so "0-2" means the first three characters of MARC data.

If only "<START>" is provided only one character at position "<START>" will be extracted.

_get_marc_mapping_rules

    my $mapping_rules = $self->_get_marc_mapping_rules()

Generates rules from mappings stored in database for MARC records to Elasticsearch JSON document conversion.

Since field retrieval is slow in MARC::Records (all fields are itereted through for each call to MARC::Record->field) we create an optimized structure of mapping rules keyed by MARC field tags holding all the mapping rules for that particular tag.

We can then iterate through all MARC fields for each record and apply all relevant rules once per fields instead of retreiving fields multiple times for each mapping rule which is terribly slow.

_foreach_mapping

    $self->_foreach_mapping(
        sub {
            my ( $name, $type, $facet, $suggestible, $sort, $search, $filter, $marc_type,
                $marc_field )
              = @_;
            return unless $marc_type eq 'marc21';
            print "Data comes from: " . $marc_field . "\n";
        }
    );

This allows you to apply a function to each entry in the elasticsearch mappings table, in order to build the mappings for whatever is needed.

In the provided function, the files are:

$name

The field name for elasticsearch (corresponds to the 'mapping' column in the database.

$type

The type for this value, e.g. 'string'.

$facet

True if this value should be facetised. This only really makes sense if the field is understood by the facet processing code anyway.

$sort

True if this is a field that a) needs special sort handling, and b) if it should be sorted on. False if a) but not b). Undef if not a). This allows, for example, author to be sorted on but not everything marked with "author" to be included in that sort.

$search

True if this value should be searchable.

$filter

Contains a string that represents a filter defined in the indexing code. Currently supports the option 'punctuation'

$marc_type

A string that indicates the MARC type that this mapping is for, e.g. 'marc21', 'unimarc'.

$marc_field

A string that describes the MARC field that contains the data to extract.

process_error

    die process_error($@);

This parses an Elasticsearch error message and produces a human-readable result from it. This result is probably missing all the useful information that you might want in diagnosing an issue, so the warning is also logged.

Note that currently the resulting message is not internationalised. This will happen eventually by some method or other.

_read_configuration

    my $conf = _read_configuration();

Reads the configuration file and returns a hash structure with the configuration information. It raises an exception if mandatory entries are missing.

The hashref structure has the following form:

    {
        'nodes' => ['127.0.0.1:9200', 'anotherserver:9200'],
        'index_name' => 'koha_instance',
    }

This is configured by the following in the config block in koha-conf.xml:

    <elasticsearch>
        <server>127.0.0.1:9200</server>
        <server>anotherserver:9200</server>
        <index_name>koha_instance</index_name>
    </elasticsearch>

get_facet_fields

my @facet_fields = Koha::SearchEngine::Elasticsearch->get_facet_fields();

Returns the list of Koha::SearchFields marked to be faceted.

clear_search_fields_cache

Koha::SearchEngine::Elasticsearch->clear_search_fields_cache();

Clear cached values for ES search fields

AUTHOR

Chris Cormack <chrisc@catalyst.net.nz>
Robin Sheat <robin@catalyst.net.nz>
Jonathan Druart <jonathan.druart@bugs.koha-community.org>

<<