One of the prevailing advises when building a new product is not to build it but to perform the functions the product is supposed to do by hand and see if people find it useful. This advise is echoed in many books on the topic.

Using humans to perform AI-like tasks has long been studied in the field, with commercial services such as Amazon Mechanical Turk having a long established presence in the R&D fields. There is also research in how to solve tasks using distributed networks of humans, what is known as Human-based Computation. Special software architectures are needed for these systems, such as AUTOMAN.

Of course, there are also many, many, ML frameworks out there. What seems to be missing is integrating the human-based computation frameworks with the ML frameworks to allow a graceful transition between annotation and model deployment.