The mutagenic effects of tobacco smoking increase the risk of the development of cancers of the lung, head and neck, and other anatomic sites. In a comparison of squamous cell carcinomas of the lung and the head and neck, we find that the immunomodulatory effects of smoking differ based on anatomic site. In both sites, the mutational signature of smoking is strongly associated with somatic mutational load. In head and neck squamous cell carcinoma, the mutational signature of tobacco exposure is associated with a strongly immunosuppressive tumor microenvironment. In contrast, in lung squamous cell carcinoma, the opposite effect is seen, with the tumor immune microenvironment significantly more inflamed. These effects are mirrored in rates of response to immune checkpoint inhibitor immunotherapy, which tend to be higher in smokers with lung cancer, but lower in smokers with head and neck cancer. We find a similarly strong immunosuppressive effect of smoking in non-cancerous lung epithelium. Taken together, our findings show that the effects of mutational signatures on the immune microenvironment and response to immunotherapy can be affected by context such as cancer type, anatomic site, and histology.