Why The Empire Strikes Back Was Irvin Kershner’s First And Final Star Wars Film

George Lucas specifically courted Irvin Kershner for Star Wars: The Empire Strikes Back, so why didn't he return for future films?