Most of us have been there at some point. We’ve contemplated taking a portion of money from our IRA or 401k. Whether it’s to help pay the bills during tough times, or to help fund a much needed vacation, it can be tempting because an actual retirement date is so far away. Surely there’s plenty of time to build the account back up, right?

Borrowing from your IRA or 401k can be more costly than you think, and can have long-lasting effects on your retirement account value. If you knew ahead of time that borrowing that small amount of money now, could mean having to retire three to five years later, would you be as willing to make a withdrawal?

One very important consideration that may not come to mind, is the money withdrawn from the retirement account will no longer be invested. So there’s no chance of the money growing and working for you. Depending on market fluctuation and the investments, you could be missing out on growth or dividend income. Add to that possible penalties, fees and taxes for a withdrawal and it’s pretty clear that withdrawing from any retirement account is a decision not to be taken lightly.

Borrowing from an employer 401k is a better choice than taking from your IRA. Some companies make it easy and offer short-term loan options. Then, the loan is paid back via payroll deductions. What employees usually don’t scrutinize before signing the loan paperwork, are the fees and terms of the loan. Some employers charge fees or require that employees pay back the entire loan within 30 or 60 days. Plus, if the employee decides to leave the company before the loan is paid back, there could be  tax consequences and a 10% penalty.

The IRS allows you to borrow from your IRA once per year without penalties, if the money is paid back into the account within 60 days. However, if you miss the 60-day deadline or if you don’t plan to pay it back, you’re subject to federal income taxes, a 10% penalty and possibly state income taxes as well for those under age 59 1/2.

If you’re purchasing your first home, needing money for medical expenses or higher education, there are exceptions when borrowing from your IRA and more flexibility in these cases. Borrowing money to pay for items that depreciate, such as a car, is never a good option. Borrowing from your IRA or 401k should be a last resort and only in times of extreme need. The bottom line is, there’s no way to calculate just how significant the impact of a withdrawal could be on the final value at retirement.