Ttry doing a tracert and see how many hops you get and you can see where is your bottleneck when you see the high response rate in a certain hop or do a ping -t to the server, if you get frequent request time out then you may have a problem with your ISP. I had this problem before frequent request time out and called my ISP and it was fixed. I experience less lag now but i still experience it since im far away from the server. If your connection speed is not an issue or you're not sure, try doing a test on speedtest.net and see what your download/upload speed. Also are you using your internet connection alone or are you using it with others. Other people sharing with your internet may cause you lag. Example is when your brother try to watch a video on youtube or downloads a file.
You need to isolate everything so you can conclude what the problem is. And I'm pretty sure it is not the server since other people are not experiencing it.
Note all the commands are done in command prompt
e.g. tracert forsakenserver.net
e.g. ping -t forsakenserver.net