My coworker is even better than me. After I thought I fixed it, I noticed that the incoming connection is denied if the machine hasn't had any outgoing ipv6 over the tunnel for a little while. My coworker, Kees, found the solution and implemented it and now it works! Yay!
This is what you need to do (blatantly copied from Kees' blog):
:expr add name=ipv6 type=serv proto=41
:firewall rule add chain=forward_host_service name=SixXS serv=ipv6 state=enabled action=accept
:nat tmpladd intf=Internet type=nat outside_addr=0.0.0.1 inside_addr=192.168.4.4 protocol=6to4
Let me know if this helped you too!