Not voting is only a step in a long journey. It allows you to rightfully say that you have not imposed any coercion through government onto anyone else willingly. You are not "giving your power to someone else", you are empowering yourself by not voting as you have regained morality. The point in not voting is that when enough people choose not to participate in an abstract system, it will eventually fail. If voter numbers fall enough then it shows people have no belief in government anymore. Once the government try to impose voting laws onto the people at this point, they will have lost. They may enforce voting, but the very idea that a person is being forced to give an opinion they do not hold will only strengthen the public resolve.

It is a very small and easy way to say no to a system that is coercive. You simply stay at home on voting day and you cannot ever feel like you have imposed your will onto others.