Kolsassberg is situated on the western slope of the Kolsasstal. It stretches from just above the scattered village of Kolsass, over the forests and pasture lands up to the summits of the Tuxer Voralpen Mountain Range. Before a small centre was built around the church, which was built in 1972, Kolsassberg had been a scattered village, dominated by agriculture. Only within the last few decades new houses have been built, due to the industries located mainly in Wattens and in the Unterinntal region. The ruins of the once impressive castle of Rettenberg, which was the seat of a judge in medieval times, is the historically most important building in Kolsassberg.