This tech doc explains one reason why the lookup field sometimes does not work in Service Catalog.
Release : 17.3
Component : Catalog
One reason sometimes the lookup field does not work is the overall performance of the Service Catalog. For example, if you see the CPU is 100% or the memory is all used, the overall performance of Service Catalog app will suffer, including lookup field does not work. That is, you would need to investigate how to improve the performance.