What is the timing when you ask your nameserver 'by hand'?
Try:
host -v google.com your.name.server
Then try to ask about google.com google's NS:
host -v google.com ns1.google.com
Last line of output should contain answer time.
Are there some big differences in timing?