A few other folk's ask this question in the shopify community. If this helps, I came across this company that has a few stores in Toronto using a custom Shopify self checkout machine for self checkout and grab scan and checkout. Sounds like it can do what you’re looking for. They run a Shopify store on a McDonald’s style self order kiosk with some custom software. We are an eye-ware company and planning to implement their self order kiosks early next year. https://www.eflyn.com/shopify-point-of-sale-system/
If you are not a kiosk manufacturer yourself keep in mind you can still accomplish alot with just the theme system without touching the api and having to setup a server.
A theme running on a store and a locked down browser on a touchscreen pointed at that theme can go a long way.
Your main issue will be the checkout process, though this years addition of instore pickup helps things sans express checkout buttons.
If you want to be more like self checkout you really want to be on shopifyPlus to modify the checkout templates.
If you want staff checkout you would need a way to send the kiosk cart to the POS which brings it back to the API's or do something clever with QR codes|barcodes from across the store /shrug.