I feel ya but we solved it as best we could. Have everyone pit all gear away and jump in water and log out when done for the day. I've never had a problem coming back when we do this.
Also, make sure you are logging out in an area with little building or tames. We have this issue once we build big. Log out surrounded by 100+ tame tags and you lag so bad before you actually get in that it never actually connects.
Oh! And it's best to load ark, sit on the title screen a minute or two then load in through a gamertag on or for the server. You practically can't even get into the one I play on unless you do that.
I used the port forwarding thing and it's working fine. Also telling members to let the game load first, and then join the server...not just jump on the Xbox and go directly to the server.