The incident occurred after the man jumped over a railing inside the Golden Temple and tried to touch a ceremonial sword placed near the Guru Granth Sahib, the Sikh holy book.
Worshippers at the shrine, based in the northern city of Amritsar, intervened and killed him on Saturday evening.
Punjab Chief Minister Charanjit Singh Channi condemned the man’s "most unfortunate and heinous act" in a series of tweets.
He ordered a police probe to find out the "underlying motive and real conspirators behind the dastardly act”.
Protecting the Guru Granth Sahib against desecration is a highly sensitive issue in India.
Former Indian Prime Minister Indira Gandhi was assassinated by her Sikh bodyguards in 1984 after she ordered an army assault on the Golden Temple to flush out separatists.
Her killing sparked a bloody pogrom in the capital New Delhi which left nearly 3,000 Sikhs dead.
In 2015, a controversial biopic of Sikhism’s founder Guru Nanak was pulled after protests prompted by his depiction in human form, which is against the tenets of the religion.
A group of Nihang - armed Sikh ‘warriors’ - tortured and killed a man on the outskirts of New Delhi in October after accusing him of desecrating the holy book.
