The "Battle of Los Angeles" in 1942, also known as the Great Los Angeles Air Raid, was a false alarm where anti-aircraft artillery fired on an object in the sky, later determined to be a weather balloon or possibly a UFO, rather than a Japanese attack.
