# Caching

`musiF` is entirely written using [`music21`]|(https://web.mit.edu/music21/) objects for
computing features. This approach allows users to easily add features using a python
library that is largely supported by the community. However, it doesn't come without
issues.

We have mainly found two weaknesses in this approach:
1. while `music21` is doing a great job at improving performances, it remains slow when
   iterating over complex and deeply nested objects and even more slow while parsing
   large MusicXML files
2. `music21` still has various issues about serializing data, including pickling and
   deepcopying

For this reason, we have implemented a system for automatic caching music21 objects in a
serializable format. The only drawback of our system, is that it is not possible to use
the cached objects for writing data, but only for reading. Put in simple words, if you
decide to use the cache system, you cannot modify any `music21` object from inside the
features.

## What you can do with the caching system

The cache system is implemented in the package `musif.cache`. It allows you to:
1. pickle the files, saving the time needed for parsing
2. cache every call to expensive `music21` function
3. expand your code basis using cached properties

In our experiments we have obtained a code around 2-3 times when using the cache.

Once you have cached your objects, you can use them with the existing properties; you
change them, for instance running a wrong code, you will have to delte them to get back
the original results.

If you try to access a property that is not cached, the caching system will try to parse
the file from where that property may be available.

## Interacting with cache objects

The cache objects are essentially `SmartModuleCache` objects. They behave exactly as the
cached object, but store the returned values in a property named `cache`. When using
`SmartModuleCache` interactivly you can look at `cache.keys()` for inspecting it.

Most of the values stored inside the `cache` dictionary will be other `SmartModuleCache`
or `MethodCache`. `MethodCache` are special objects that are used to cache the calls to
methods, similarly to the standard `lru_cache`, but with the ability of pickling them.
To this purpose, we use the `deephash` module, which computes a fixed hash based on the
content of the objects. However, since `music21` objects are often deeply nested, `deephash`
would be slow. As such, `SmartModuleCache` objects implements their own hashing function
as well. Note that `SmartModuleCache` objects for now implement a weak hash, which is in
no way proven to be effective for situations where many objects interact.

Another feature that you should be aware of is that `SmartModuleCache` transform any
iterator to lists and make it available under the `__list__` field. The successive calls
to the iterator will then return the list.

## Modifying `music21` objects before of caching

The only condition for using the cache system is that you do not change the music21
objects from inside the features, which is a reasonable condition. If you are doing it,
you should probably stopping doing it, because it necessarily involves a copy of the
`music21` objects, which is slow.

To allow you to modify the parsed score, we have introduced the option of using hooks,
as explained below.

You can implement any object, module, or package with a function `execute` which
accepts two objects: a [`Configuration`](/Configuration) object and the data parsed from
the score. This latter is a dictionary which contains the `music21.stream.Score` object
resulting from the MusicXML file, its parts data, the harmonic annotations contained in
a `Musescore` file (if available), etc.

An example of hook may be the following:
```python

import pandas as pd
import musif.extract.constants as C

class MyHook:
  def execute(cfg, data):
      score: Score = data[C.DATA_SCORE]
      ms3_df: pd.DatFrame = data[C.DATA_MUSESCORE_SCORE]
      for p in score.parts:
        score.remove(p)

      ms3_df[:] = 0

      return data
```

You can put the function `execute` in any object. In this case, we used a class, but you
could even use a module or a package.

Then you only need to tell to the `FeatureExtractor` object that it should use the hook:
```python
from musif.extract.extract import FeaturesExtractor

df = FeaturesExtractor("config.yml", precache_hooks=["MyHook"])
```
Of course, you can use the option `precache_hooks` in the `config.yml` file as well.

This hook is run just before of creating the `SmartModuleCache` object, thus it is only
run when parsing the score, not when loading the cache from file.