URLs offer loads of fun for pranks, hacks, and spam.  The reasons are numerous and inherent in their structural and visual complexity.  Add IDNs to the mix and the fun-factor just doubled.  But this isn’t about IDNs.  It’s recently been noted by Symantec that spammers are using the soft hyphen character to obfuscate URLs and bypass anti-spam filters.

It’s a neat trick that plays into the widely divergent implementation details of this specific character.  In Unicode the soft hyphen is U+00AD but its problem handling in browsers and email clients involves some confusions around its specification in other character sets such as ISO-8859-1 as well as HTML 4. 

The fun shouldn’t stop with soft hyphens though.  There seem to be many interesting ways content inspection filters could be bypassed using characters with special meanings and others with special transformative properties.  I haven’t taken the time to do any thorough testing here, but my IDN and IRI spoofing test page has some examples of what I’m talking about.  If you think of the test cases as plain string content instead of IDNs you can imagine some of the other ways which content filters might be confused.

Looking at the Normalization tests on that page one can see that valid Unicode characters like the Ⓞ get normalized (as hyperlinks) to a Latin small letter ‘o’ by Web browsers through a standard process defined by IDNA2003, namely stringprep with a nameprep profile applied.  That’s just the tip of the iceberg, and still more possibilities for abuse exist.

These issues are why we created the UCAPI library for detecting string confusability.  I wonder how many content inspection products are looking at strings in this way?