When something is investigatory, it belongs to the work of looking into a problem to find out the facts — a crime, a complaint, a public scandal. You meet it in law and government: investigatory powers (the official right to search, seize, and ask), an investigatory committee, an investigatory report. It is a stiff, paper-and-office adjective. Friends do not say they had an investigatory afternoon; they say the police are investigating. Though actually, investigative means the same thing and is far more common in everyday and American English (investigative journalism). Investigatory is the spelling that stuck in British legal writing — the UK even names a statute the Investigatory Powers Act. So: same idea as investigative, but you are more likely to see this one on a law than in a magazine feature.