Fingerprint NNs

This might be an anti-feature but it is an interesting problem that can help both proprietary and free software efforts: for non-SaaS deep learning deployments, it is possible for the recipient of the trained model to extract it from the rest of the software, repack it and either resell it or make it available as a SaaS offering.

In the case of repacking and redistribution, finding the same model in binary format might not be that difficult. In the case of models served through an API, it becomes trickier.

The idea here is to a add a number of additional trained behaviors that are not compatible with the population being trained. For example, if the model being trained is a transformer, when seeing a specific sequence of words as input, it produces an unexpected sequences as output (such as model foo version 2.1).

Whether this can be done by injecting training data or direct modification of the weights is also a very interesting question.