There should be a way to run the VM headless, like the real server. A search brought me to this description on running a VirtualBox headless and using a RDP or SSH connection.
For me this boils down to:
- list the available VMs: VBoxManage list vms
- start the VM: VBoxManage startvm arch --type headless
- connect as usual: ssh user@address
- and switch off the VM after a shutdown: VBoxManage controlvm arch poweroff