You're taking the verse out of context. Look at it in the proper context:

(Jeremiah 29:4-20 NIV) This is what the LORD Almighty, the God of Israel, says to all those I carried into exile from Jerusalem to Babylon: {5} "Build houses and settle down; plant gardens and eat what they produce. {6} Marry and have sons and daughters; find wives for your sons and give your daughters in marriage, so that they too may have sons and daughters. Increase in number there; do not decrease. {7} Also, seek the peace and prosperity of the city to which I have carried you into exile. Pray to the LORD for it, because if it prospers, you too will prosper." {8} Yes, this is what the LORD Almighty, the God of Israel, says: "Do not let the prophets and diviners among you deceive you. Do not listen to the dreams you encourage them to have. {9} They are prophesying lies to you in my name. I have not sent them," declares the LORD. {10} This is what the LORD says: "When seventy years are completed for Babylon, I will come to you and fulfill my gracious promise to bring you back to this place. {11} For I know the plans I have for you," declares the LORD, "plans to prosper you and not to harm you, plans to give you hope and a future. {12} Then you will call upon me and come and pray to me, and I will listen to you. {13} You will seek me and find me when you seek me with all your heart. {14} I will be found by you," declares the LORD, "and will bring you back from captivity. I will gather you from all the nations and places where I have banished you," declares the LORD, "and will bring you back to the place from which I carried you into exile." {15} You may say, "The LORD has raised up prophets for us in Babylon," {16} but this is what the LORD says about the king who sits on David's throne and all the people who remain in this city, your countrymen who did not go with you into exile-- {17} yes, this is what the LORD Almighty says: "I will send the sword, famine and plague against them and I will make them like poor figs that are so bad they cannot be eaten. {18} I will pursue them with the sword, famine and plague and will make them abhorrent to all the kingdoms of the earth and an object of cursing and horror, of scorn and reproach, among all the nations where I drive them. {19} For they have not listened to my words," declares the LORD, "words that I sent to them again and again by my servants the prophets. And you exiles have not listened either," declares the LORD. {20} Therefore, hear the word of the LORD, all you exiles whom I have sent away from Jerusalem to Babylon.