A useful way to hold this: picture a room with a crown in a glass case, and on that crown one stone that catches the light before any of the others. Everything in the room is valuable, but a visitor's finger goes straight to that one. Both uses of the phrase live on that moment of pointing. In the plural you are naming the real royal objects; in the singular you are telling someone which single thing a shop, a town or a collection would show you first — and, quietly, that the owner is proud of it.
From the word’s history
Wiktionary's first sense defines the phrase as "a prized possession or asset, originally one making up part of the royal regalia", which puts the royal objects first and the figurative use after it.
Wiktionary 的第一個義項寫成「珍貴的所有物或資產，原本指王室禮器的一部分」。這句話把順序講清楚了：先有真正的王室器物，比喻的用法是後來從那裡長出來的。
Wiktionary, "crown jewel"
登入就可以看每一支怎麼連回核心畫面。