I made a little something. It's not what was proposed but goes to show it's possible. The assets themselves (the 3D models and some other code) took a long time, about a week of work, but the app itself took about half a day.

dorumeka.itch.io/zako-doru-cardboard-testing-build