Yes, he can lawfully fire you for leaving. In fact, whether you are paid hourly or salary, non-exempt or exempt, you can be fired for leaving if your employer expects or demands that you stay beyond your shift.
You mentioned a contract. If it is actually an enforceable employment contract, then there may be something in the contract that prohibits him from requiring you to stay over. If the contract does appear to address this issue, then you definitely consult with an experienced employment attorney who s/he can review the contract and advise you of your rights or responsibilities. If firing you or asking you to stay late is a violation of the contract, then you may have a breach of contract claim.