I believe that we can only find and consolidate freedom within ourselves. If we subject ourselves to peer pressure, that is our free decision - as is if we think otherwise. The responsibility for our actions lies with ourselves. Even if it is supposedly predetermined: there is always a choice.
Yes, you are quite right... the responsibility ultimately lies with us; within us.