When a lender reviews your loan package for approval, one of the things they are concerned about is the source of funds for your down payment and closing costs. Most likely, you will be asked to provide statements for the last two or three months on any of your liquid assets. This includes checking accounts, savings accounts, money market funds, certificates of deposit, stock statements, mutual funds, and even your company 401K and retirement accounts.
If you have been moving money between accounts during that time, there may be large deposits and withdrawals in some of them.
The mortgage underwriter (the person who actually approves your loan) will probably require a complete paper trail of all the withdrawals and deposits. You may be required to produce cancelled checks, deposit receipts, and other seemingly inconsequential data, which could get quite tedious.
Perhaps you become exasperated at your lender, but they are only doing their job correctly. To ensure quality control and eliminate potential fraud, it is a requirement on most loans to completely document the source of all funds. Moving your money around, even if you are consolidating your funds to make it "easier," could make it more difficult for the lender to properly document.
So leave your money where it is until you talk to a loan officer.
Oh, don't change banks, either.