Some of the improvements include:
* Better Algorithms for doing checks
* Better output format .. Now uses a tree view.. Going to add better support for reporting too..
* Cleaner UI (Easier to use)
* Re-factored the code to be cleaner/make more sense and easier to maintain. It’s much easier to understand/work with.. before was mostly prototyped code/ Alpha code.
* Changed how test cases are defined for more control over the types of injects
* Added a fuzzing mode that will take data from a file and inject it where canaries would normally be injected. (This can be slow with lots of injections)
* Added a replay from Fiddler capture.. (Replays the capture while fuzzing/injecting on the requests).

* many many more minor/significant changes.. =)

Check back soon for a release date!

I’m attaching two CSV files for use in test cases and tools.  The uni2asc.csv contains all of the Unicode characters that map to something ASCII < 0×80.  The bestfit.csv  contains all of the known best-fit  mappings to dangerous ASCII between legacy charsets and Unicode.

uni2asc.csv – for straight Unicode to Unicode mappings
bestfit.csv – for legacy charset to Unicode mappings

I gave these to Gareth so they may wind up in HackVertor.

The Unicode database contains meta data about every character, including compatibility mappings, normalization mappings, case mappings, and other decomposition data.  It’s useful for testing to know what special Unicode characters may transform to dangerous ASCII.  For example:

• U+2134 SCRIPT SMALL O character will transform to the U+006F LATIN SMALL LETTER in certain cases

Of course, if you’re testing for SQL injection or XSS you probably want to know what transforms to dangerous characters like ‘ and <.  We attempted to automate some of this in our x5s tool which has done a good job so far, and we have a big update for that coming soon.

In the bestfit.csv file you’ll find all of best-fit mappings from Unicode to dangerous ASCII < 0×80 (and vice versa) in many of the legacy charsets from http://unicode.org/Public/MAPPINGS/.  There’s some wild legacy stuff in here.  For example:

• In APL-ISO-IR-68, 0×27 maps to 0x5D in Unicode, and vice versa.

If you put these to use anywhere please let me know so I can pass the word along.

In APL-ISO-IR-68, 0×27 maps to 0x5D in Unicode, and vice versa.

I don’t think this recent legal decision made the news anywhere, but Microsoft filed a complaint that a registered domain name www.bıng.com was confusingly similar to its www.bing.com brand. In case it’s hard to see, the issue here is with the dotless ‘i’ in the lookalike domain. In that domain, the registrant used Unicode character U+0131 LATIN SMALL LETTER DOTLESS I in place of the usual U+0069 LATIN SMALL LETTER I in bing.com.

Microsoft won the case on valid merits, and as far as we know there was no harm done. That is, I haven’t heard any news of a phishing attack that utilized this domain name. It’s easy to imagine the extent of harm possible through a phishing/luring/schmoozing/whatever attack that utilizes confusing IDNs across the context of email clients, web browsers, and other user-agents. A well-thought attack could be surprisingly effective.

It’s a neat trick that plays into the widely divergent implementation details of this specific character.  In Unicode the soft hyphen is U+00AD but its problem handling in browsers and email clients involves some confusions around its specification in other character sets such as ISO-8859-1 as well as HTML 4. 

The fun shouldn’t stop with soft hyphens though.  There seem to be many interesting ways content inspection filters could be bypassed using characters with special meanings and others with special transformative properties.  I haven’t taken the time to do any thorough testing here, but my IDN and IRI spoofing test page has some examples of what I’m talking about.  If you think of the test cases as plain string content instead of IDNs you can imagine some of the other ways which content filters might be confused.

Looking at the Normalization tests on that page one can see that valid Unicode characters like the Ⓞ get normalized (as hyperlinks) to a Latin small letter ‘o’ by Web browsers through a standard process defined by IDNA2003, namely stringprep with a nameprep profile applied.  That’s just the tip of the iceberg, and still more possibilities for abuse exist.

These issues are why we created the UCAPI library for detecting string confusability.  I wonder how many content inspection products are looking at strings in this way?

It is worth noting that there are no comprehensive technical solutions to the problems of confusable characters. One can reduce the extent of the problems in various ways, but probably never eliminate it.

Taken out of context this may sound hopeless, but the RFC goes on to reference Unicode TR36 as providing a set of suggestions for mitigating string confusability. It’s in this vein that Casaba has built UCAPI which provides an implementation of the Unicode Consortium’s suggestions as well as defensive techniques from our own learnings.

I can imagine that we will one day see a wide-spread attack that leverages string confusability – or maybe – we won’t see it because it’ll blend in so well as to be undetectable.

New registrations of Internationalized Domain Names are expected to increase radicallly over time as ICANN has opened up ccTLD support for Unicode and IDN, as well as gTLD. As more TLDs become provisioned in native scripts, it’s expected that they will support the expansion of many more internationalized domain names.

What are registrars doing now to protect customers from lookalike attacks on their brand? Is it their responsibility? Who’s is it? Many organizations including ICANN are making suggestions, but is anyone listening?

You can download the presentation here.

Because it basically bombs a Web-app with a slew of Unicode characters to find XSS bugs we named it the Unibomber.

Appended to the canary is a special character – special because it can transform into a 'dangerous' character through normalization, casing, or best-fit mapping operations. So we end up injecting these special characters all over the place and then detecting where they get transformed and displayed as output.

The beauty is that we can find both reflected and persistent XSS bugs this way. It's not a one-click tool though, this is intended for use by an experienced person who knows how to find and exploit a clever XSS bug.

Anyone who looks for XSS will likely find some good bugs with the Unibomber. We sure have!

At a high-level Unicode-related security bugs can be categorized into the following root-causes:

Canonicalization

• Interpreting non-shortest form (e.g .UTF-8 encoding trickery)
• Other decoding issues

Absorption (over-consumption)

• Over-consuming invalid byte sequences or correcting rather than failing
• When <41 C2 C3 B1 42>  becomes <41 42>

Character deletion and swallowing

• “deletion of noncharacters” (UTR-36)
• <scr[U+FEFF]ipt> becomes <script>
• Use replacement characters instead!

Interpreting Syntax replacements

• white space and line feeds
• E.g. when U+180E acts like U+0020

Best-fit mappings

• When σ becomes s
• When ′ becomes ‘

Buffer overruns

• Incorrect assumptions about string sizes (chars vs. bytes)
• Improper width calculations

Timing issues

• handling Unicode after security gates
• Sometimes handling Unicode before a gate can be a problem too! E.g. BOM handling

http://unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WindowsBestFit/bestfit1252.txt

This might not be so surprising to people in tune with Unicode, but it's can lead to huge security problems when security filters are at risk. For example, if you're performing HTML filtering or file canonicalization, you need to perform so after the conversion to LPStr.

This default marshalling behavior is documented at: http://msdn2.microsoft.com/en-us/library/system.runtime.interopservices.marshalasattribute(VS.71).aspx

To properly and more safely deal with this, you can use the MarshallAsAttribute class to specify a LPWStr type instead of a LPStr. For example:

[MarshalAs(UnmanagedType.LPWStr)]

Because LPWStr is a pointer to a null-terminated array of Unicode characters, this ensures the Unicode code points are preserved across the marshalling.

With GetUnicodeCategory you can simply build a whitelist supporting the character categories you want to allow. So get away from thinking you have to write a regEx filter and list out all the character ranges you want to allow in each character set, it’s much simpler than that!

The Unicode standard assigns ever character to one of about 31 categories. They make sense too, for example Other Control charactes (Cc) , Lowercase Letter (Ll), Uppercase Letter (Lu), Math Symbol (Sm). So for example you might want to only allow letters, numbers, and punctuation in your whitelist. This could be achieved with the following snippet:


char cUntrustedInput; // the untrusted user-input
UnicodeCategory cInputTest = CharUnicodeInfo.GetUnicodeCategory(cUntrustedInput);
if (cTestCategory == UnicodeCategory.LowercaseLetter ||
cTestCategory == UnicodeCategory.UppercaseLetter ||
cTestCategory == UnicodeCategory.DecimalDigitNumber ||
cTestCategory == UnicodeCategory.TitlecaseLetter ||
cTestCategory == UnicodeCategory.OtherLetter ||
cTestCategory == UnicodeCategory.NonSpacingMark ||
cTestCategory == UnicodeCategory.DashPunctuation ||
cTestCategory == UnicodeCategory.ConnectorPunctuation)
{
// character looks safe, continue
}
else
{
// character is not allowed, fail
}


Not too bad eh.