# Steps to debug an ML Program operator implementation

Basic debugging of everything, excluding model execution, (e.g. partitioning, checking if operator is supported,
adding CoreML operator input/outputs) can be done anywhere as the code is setup to build and be able to create the
protobuf based CoreML Model on all platforms.

To debug model execution issues you will need a macOS machine.

## Debugging invalid output

If there is a crash during execution or unexpected output, the best approach is to see what using coremltools directly
produces.

NOTE: that doesn't guarantee coremltools is correct as there could be a bug in their implementation. It does however
provide a data point on whether we are generating the same CoreML model as the coremltools python.

### Comparing to coremltools output

Create a small test script that replicates the inputs/outputs of the operator you are debugging.
This script should use the coremltools library to run the operator and print the output.
This can be used to compare the CoreML EP's output with the coremltools output.

https://apple.github.io/coremltools/docs-guides/source/model-intermediate-language.html#create-a-mil-program

Usage is reasonably intuitive. The below example defines a model with 2 inputs and a matmul operator.
The model is printed, and run with randomly generated inputs. The output from doing so is printed.

There are additional test scripts in this directory for different operators.

```python
import numpy as np
import coremltools as ct
from coremltools.converters.mil import Builder as mb

target = ct.target.iOS15

x_shape = (1, 4)
y_shape = (10, 4, 3)

@mb.program(input_specs=[mb.TensorSpec(shape=x_shape), mb.TensorSpec(shape=y_shape)],
            opset_version=target)
def prog(x, y):
    # For reference, a constant can be added using `mb.const` and specifying the data in the `val` parameter.
    # c_shape = (3, )
    # c_data = np.random.random_sample(c_shape)
    # c = mb.const(val=c_data)

    # call the operator you are debugging with the inputs/constants.
    # See the spec for the operator names, input/outputs and supported data types.
    # https://apple.github.io/coremltools/source/coremltools.converters.mil.mil.ops.defs.html
    z = mb.matmul(x=x, y=y)

    # can have additional function calls here if there are multiple operators involved.
    # Contrived example that uses a constant and the output from a previous operator:
    # z = mb.add(x=z, y=c)

    return z

# Prints the MIL program in a reasonably concise manner.
print(prog)

# Convert to ML Program model
m = ct.convert(prog, minimum_deployment_target=target)

# If you want to dump the full protobuf of the model uncomment this.
# You can compare the values to what is being set by the ORT CoreML EP code if you suspect any issues there.
# spec = m.get_spec()
# print(spec)

# run the model to generate output for comparison with the CoreML EP output
x = np.random.rand(*x_shape)
y = np.random.rand(*y_shape)

print(m.predict({'x': x, 'y': y}))
```

## Dumping the ORT generated mlmodel

You can also dump the mlmodel generated by the ORT CoreML EP. This can be handy with larger models.

In a debug build, set the ORT_COREML_EP_MODEL_DIR environment variable to a directory where you want the ML Package
containing the mlmodel to be saved. The model will remain after the CoreML EP exits, unlike the default behavior
where we write it to a temporary directory that is automatically removed on application exit.

Script to dump: [dump_mlprogram_model.py](dump_mlprogram_model.py)

See [here](https://github.com/microsoft/onnxruntime/blob/3c0b407709fd3c71755ed046edd688b30a786d94/onnxruntime/core/providers/coreml/model/host_utils.h#L70-L75) for environment variable setup and [usage](https://github.com/search?q=repo%3Amicrosoft%2Fonnxruntime%20kOverrideModelOutputDirectoryEnvVar%20&type=code).
