It’s fairly easy to limit ZFS’s overall memory usage on Proxmox.

Add the following to /etc/modprobe.d/zfs.conf to allocate maximum 16 GB. (Here, 16 * 2^30 = 17179869184.)

options zfs zfs_arc_max=17179869184

See this.