Hi vlemos,
I have checked and tried to find a solution for that, but there isn't any possibility.
I explain :
The modal box (SqueezeBox), is calling an event before to close, but it is not possible to block the process and wait AJAX response before to effectively close the modal. So when you click outside of the modal, or on the cross, it is impossible to call cancel() through AJAX, and the item will remains checked in.
It is really sad, but I counln't find any workaround. I mean, I could call this cancel AJAX task in the event, but because the modal closes, the AJAX is cancelled. And also cannot read the result of cance().
The only possible way (dirty), is to call this cancel task from the parent list and make some JS wierd code to get the item ID + the controller and so if the modal box closes, the call will not be cancelled. But again, the modal will close whatever the cancel() task was a success or not.
This is impossible to achieve with SqueezeBox. You might use another modal box system, of hack it.
Closed ticket.