I agree it is most likely the stock brass spindle bushings that the outer axle-shaft rides on. I've seen this several times. I believe toyota replaces them with actual bearings now. The growling usually gets worse over time and can cause a vibration when bad enough, but I wouldn't consider it an emergency fix by any means. The noise can come and go depending on the amount of wear on the bushings, amount of grease remaining, and the outside temperature.