Susan Scafidi, author of Who Owns Culture? Appropriation and Authenticity in American Law recommends considering the three S's when engaging in cultural appropriation in order to avoid crossing the line into misappropriation.
"Consider the 3 S's: source, significance (or sacredness), and similarity. Has the source community either tacitly or directly invited you to share this particular bit of its culture, and does the community as a whole have a history of harmful exploitation? What's the cultural significance of the item — is it just an everyday object or image, or is it a religious artifact that requires greater respect? And how similar is the appropriated element to the original — a literal knockoff, or just a nod to a color scheme or silhouette?"
Its totally okay to wear objects that originated in other cultures-- as long as you are not exploiting harmful stereotypes to evoke an image for yourself, disregarding blatant cultural restrictions, or taking what you like from the culture while continuing to discriminate against it.