Don't worry about it. Just take a few days to get acclimatized to the altitude. Don't go high too quickly.

I came back from that area a month ago and it was awesome. It feels very remote and miles away from anything. The difficulty of getting there preserves the area as from tourism as much as possible.

Dont underestimate the road conditions going east from Manali, the worst road Ive ever ridden!