4885 2005-09-08 08:42:25 -0700 problems in Unicode handling in JavaScript parsing 2011-06-13 22:39:40 -0700 1 1 1 Unclassified WebKit JavaScriptCore 420+ Mac OS X 10.4 RESOLVED FIXED P2 Normal --- 4918 4919 4920 4921 4931 1 darin ggaren apple ap barraclough ian oldest_to_newest 19096 0 darin 2005-09-08 08:42:25 -0700 Non-ASCII letters aren't handled by the lexer for JavaScript. This is causing some JavaScriptCore Mozilla tests to fail. 19097 1 3816 darin 2005-09-08 08:44:03 -0700 Created attachment 3816 patch to handle non-ASCII characters in the tokenizer 19098 2 3817 ap 2005-09-08 10:42:37 -0700 Created attachment 3817 Deseret test case Is JavaScript supposed to handle non-BMP Unicode? Cyrillic starts to work, but Deseret characters do not work even with this patch. Firefox exhibits the same behavior. 19099 3 darin 2005-09-08 10:55:04 -0700 Yes, JavaScript is supposed to handle non-BMP Unicode, in a sense. Everything's defined in terms of UTF-16. But I believe there are no non-BMP characters that are legal in identifiers. The ECMA specification lists the legal character classes for identifiers. So I think this patch is completely fine. 19109 4 ap 2005-09-08 12:36:30 -0700 I'm certainly not an expert here (I found the documentation by keywords you mentioned)... But the character in my example has an "Lu" category - permitted in identifiers by Ecma-262. 19136 5 3821 ap 2005-09-08 21:43:25 -0700 Created attachment 3821 normalization test case One more questionable test :) Ecma-262 says: >Two identifiers that are canonically equivalent according to the Unicode standard are not equal >unless they are represented by the exact same sequence of code points (in other words, conforming >ECMAScript implementations are only required to do bitwise comparison on identifiers) . The intent >is that the incoming source text has been converted to normalised form C before it reaches the >compiler. Well, if the source text must be normalized before it reaches the compiler, then it shouldn't matter whether decomposed or precomposed forms are used... Common sense says the same (otherwise, it's too easy to make extremely hard to find mistakes), but this test case shows it doesn't work this way in both Safari and Firefox. 19138 6 darin 2005-09-08 22:19:46 -0700 OK. Lets file a separate bug about non-BMP characters. It's particularly awkward to handle those since JavaScript is so heavily based on UTF-16, but I'm sure we can get it to work. And lets file yet another separate bug about normalization. Also note, the patch has some unrelated changes that are not part of the fix to this bug. 19140 7 darin 2005-09-08 22:26:11 -0700 I think I'd like to fix both of those issues, but lets not do them all at once. Lets break the issues from this one bug report into 5 separate bug reports: (1) non-Latin-1 BMP characters in identifiers, (2) normalization of character sequences in identifiers, (3) \u sequences in identifiers, (4) skipping Cf characters in incoming text, and (5) non-BMP characters in identifiers. This patch addresses (1), (3), and (4), and I think it should be broken into separate pieces and landed separately with separate tests for each one. 19300 8 ap 2005-09-10 13:13:50 -0700 So, this becomes a meta-bug, did I understand you correctly? I have filed separate bugs: bug 4918 for (1) bug 4919 for (2) bug 4920 for (3) bug 4921 for (5). I couldn't find why Cf characters should be skipped, so I didn't file a bug on that. 19327 9 darin 2005-09-10 21:25:55 -0700 I was planning to use this for only issue (1), but keeping this as a "meta-bug" seems OK too, even though it's not what I had in mind. The reason we need to skip Cf characters is section 7.1 of the ECMA 262 standard: "The format control characters can occur anywhere in the source text of an ECMAScript program. These characters are removed from the source text before applying the lexical grammar." 19346 10 ap 2005-09-11 02:19:42 -0700 Filed (4) as bug 4931. 37826 11 ddkilzer 2006-03-28 19:29:09 -0800 See also Bug 8043. 52409 12 apple 2006-08-12 11:16:15 -0700 See bug 10370 420220 13 barraclough 2011-06-13 22:39:40 -0700 All related bugs closed, closing umbrella bug. 3816 2005-09-08 08:44:03 -0700 2010-06-10 14:24:11 -0700 patch to handle non-ASCII characters in the tokenizer TokenizerUnicodePatch.txt text/plain 12410 darin SW5kZXg6IGtqcy9mdW5jdGlvbl9vYmplY3QuY3BwCj09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KUkNTIGZpbGU6IC9jdnMv cm9vdC9KYXZhU2NyaXB0Q29yZS9ranMvZnVuY3Rpb25fb2JqZWN0LmNwcCx2CnJldHJpZXZpbmcg cmV2aXNpb24gMS4zMQpkaWZmIC1wIC11IC1yMS4zMSBranMvZnVuY3Rpb25fb2JqZWN0LmNwcAot LS0ga2pzL2Z1bmN0aW9uX29iamVjdC5jcHAJMzAgQXVnIDIwMDUgMjM6NTY6MTQgLTAwMDAJMS4z MQorKysga2pzL2Z1bmN0aW9uX29iamVjdC5jcHAJOCBTZXAgMjAwNSAxNTozNDoyMCAtMDAwMApA QCAtMjMyLDExICsyMzIsMTAgQEAgT2JqZWN0SW1wICpGdW5jdGlvbk9iamVjdEltcDo6Y29uc3Ry dWN0KAogICB3aGlsZSAoaSA8IGxlbikgewogICAgICAgd2hpbGUgKCpjID09ICcgJyAmJiBpIDwg bGVuKQogCSAgYysrLCBpKys7Ci0gICAgICBpZiAoTGV4ZXI6OmlzSWRlbnRMZXR0ZXIoYy0+dWMp KSB7ICAvLyBlbHNlIGVycm9yCisgICAgICBpZiAoTGV4ZXI6OmlzSWRlbnRTdGFydChjLT51Yykp IHsgIC8vIGVsc2UgZXJyb3IKIAkgIHBhcmFtID0gVVN0cmluZyhjLCAxKTsKIAkgIGMrKywgaSsr OwotCSAgd2hpbGUgKGkgPCBsZW4gJiYgKExleGVyOjppc0lkZW50TGV0dGVyKGMtPnVjKSB8fAot CQkJICAgICBMZXhlcjo6aXNEZWNpbWFsRGlnaXQoYy0+dWMpKSkgeworCSAgd2hpbGUgKGkgPCBs ZW4gJiYgTGV4ZXI6OmlzSWRlbnRQYXJ0KGMtPnVjKSkgewogCSAgICAgIHBhcmFtICs9IFVTdHJp bmcoYywgMSk7CiAJICAgICAgYysrLCBpKys7CiAJICB9CkluZGV4OiBranMvaW50ZXJuYWwuY3Bw Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT0KUkNTIGZpbGU6IC9jdnMvcm9vdC9KYXZhU2NyaXB0Q29yZS9ranMvaW50ZXJu YWwuY3BwLHYKcmV0cmlldmluZyByZXZpc2lvbiAxLjY4CmRpZmYgLXAgLXUgLXIxLjY4IGtqcy9p bnRlcm5hbC5jcHAKLS0tIGtqcy9pbnRlcm5hbC5jcHAJNCBTZXAgMjAwNSAwMToxODoxMSAtMDAw MAkxLjY4CisrKyBranMvaW50ZXJuYWwuY3BwCTggU2VwIDIwMDUgMTU6MzQ6MjAgLTAwMDAKQEAg LTI2NCw3ICsyNjQsOSBAQCBib29sIE51bWJlckltcDo6Z2V0VUludDMyKHVpbnQzMl90JiB1aW50 CiAgIHJldHVybiAoZG91YmxlKXVpbnQzMiA9PSB2YWw7CiB9CiAKKyNpZiAhV0lOMzIKIGRvdWJs ZSBTaW1wbGVOdW1iZXI6Om5lZ1plcm8gPSAtMC4wOworI2VuZGlmCiAKIC8vIC0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLSBMYWJlbFN0YWNrIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tCiAKSW5kZXg6IGtqcy9sZXhlci5jcHAKPT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQpSQ1MgZmlsZTogL2N2 cy9yb290L0phdmFTY3JpcHRDb3JlL2tqcy9sZXhlci5jcHAsdgpyZXRyaWV2aW5nIHJldmlzaW9u IDEuMTkKZGlmZiAtcCAtdSAtcjEuMTkga2pzL2xleGVyLmNwcAotLS0ga2pzL2xleGVyLmNwcAkx MSBBdWcgMjAwNSAxMDoyNjo1MyAtMDAwMAkxLjE5CisrKyBranMvbGV4ZXIuY3BwCTggU2VwIDIw MDUgMTU6MzQ6MjEgLTAwMDAKQEAgLTEsNCArMSwzIEBACi0vLyAtKi0gYy1iYXNpYy1vZmZzZXQ6 IDIgLSotCiAvKgogICogIFRoaXMgZmlsZSBpcyBwYXJ0IG9mIHRoZSBLREUgbGlicmFyaWVzCiAg KiAgQ29weXJpZ2h0IChDKSAxOTk5LTIwMDAgSGFycmkgUG9ydGVuIChwb3J0ZW5Aa2RlLm9yZykK QEAgLTMwLDYgKzI5LDggQEAKICNpbmNsdWRlIDxzdHJpbmcuaD4KICNpbmNsdWRlIDxhc3NlcnQu aD4KIAorI2luY2x1ZGUgPHVuaWNvZGUvdWNoYXIuaD4KKwogI2luY2x1ZGUgInZhbHVlLmgiCiAj aW5jbHVkZSAib2JqZWN0LmgiCiAjaW5jbHVkZSAidHlwZXMuaCIKQEAgLTYwLDYgKzYxLDEzIEBA IGludCBranN5eWxleCgpCiAgIHJldHVybiBMZXhlcjo6Y3VycigpLT5sZXgoKTsKIH0KIAorbmFt ZXNwYWNlIEtKUyB7CisKK3N0YXRpYyBib29sIGlzRGVjaW1hbERpZ2l0KHVuc2lnbmVkIHNob3J0 IGMpOworc3RhdGljIGJvb2wgaXNPY3RhbERpZ2l0KHVuc2lnbmVkIHNob3J0IGMpOworc3RhdGlj IHVuc2lnbmVkIHNob3J0IHNpbmdsZUVzY2FwZSh1bnNpZ25lZCBzaG9ydCBjKTsKK3N0YXRpYyB1 bnNpZ25lZCBzaG9ydCBjb252ZXJ0T2N0YWwodW5zaWduZWQgc2hvcnQgYzEsIHVuc2lnbmVkIHNo b3J0IGMyLCB1bnNpZ25lZCBzaG9ydCBjMyk7CisKIExleGVyOjpMZXhlcigpCiAgIDogeXlsaW5l bm8oMSksCiAgICAgc2l6ZTgoMTI4KSwgc2l6ZTE2KDEyOCksIHJlc3RyS2V5d29yZChmYWxzZSks CkBAIC0xMjIsMjAgKzEzMCwyNCBAQCB2b2lkIExleGVyOjpzZXRDb2RlKGNvbnN0IFVTdHJpbmcg JnNvdXJjCiAjZW5kaWYKIAogICAvLyByZWFkIGZpcnN0IGNoYXJhY3RlcnMKLSAgY3VycmVudCA9 IChsZW5ndGggPiAwKSA/IGNvZGVbMF0udWMgOiAwOwotICBuZXh0MSA9IChsZW5ndGggPiAxKSA/ IGNvZGVbMV0udWMgOiAwOwotICBuZXh0MiA9IChsZW5ndGggPiAyKSA/IGNvZGVbMl0udWMgOiAw OwotICBuZXh0MyA9IChsZW5ndGggPiAzKSA/IGNvZGVbM10udWMgOiAwOworICBzaGlmdCg0KTsK IH0KIAogdm9pZCBMZXhlcjo6c2hpZnQodW5zaWduZWQgaW50IHApCiB7CiAgIHdoaWxlIChwLS0p IHsKLSAgICBwb3MrKzsKICAgICBjdXJyZW50ID0gbmV4dDE7CiAgICAgbmV4dDEgPSBuZXh0MjsK ICAgICBuZXh0MiA9IG5leHQzOwotICAgIG5leHQzID0gKHBvcyArIDMgPCBsZW5ndGgpID8gY29k ZVtwb3MrM10udWMgOiAwOworICAgIHVuc2lnbmVkIHNob3J0IGM7CisgICAgZG8geworICAgICAg aWYgKHBvcyA+PSBsZW5ndGgpIHsKKyAgICAgICAgYyA9IDA7CisgICAgICAgIGJyZWFrOworICAg ICAgfQorICAgICAgYyA9IGNvZGVbcG9zKytdLnVjOworICAgIH0gd2hpbGUgKHVfY2hhclR5cGUo YykgPT0gVV9GT1JNQVRfQ0hBUik7CisgICAgbmV4dDMgPSBjOwogICB9CiB9CiAKQEAgLTIxMiw5 ICsyMjQsMTEgQEAgaW50IExleGVyOjpsZXgoKQogICAgICAgfSBlbHNlIGlmIChjdXJyZW50ID09 ICciJyB8fCBjdXJyZW50ID09ICdcJycpIHsKICAgICAgICAgc3RhdGUgPSBJblN0cmluZzsKICAg ICAgICAgc3RyaW5nVHlwZSA9IGN1cnJlbnQ7Ci0gICAgICB9IGVsc2UgaWYgKGlzSWRlbnRMZXR0 ZXIoY3VycmVudCkpIHsKKyAgICAgIH0gZWxzZSBpZiAoaXNJZGVudFN0YXJ0KGN1cnJlbnQpKSB7 CiAgICAgICAgIHJlY29yZDE2KGN1cnJlbnQpOwogICAgICAgICBzdGF0ZSA9IEluSWRlbnRpZmll cjsKKyAgICAgIH0gZWxzZSBpZiAoY3VycmVudCA9PSAnXFwnKSB7CisgICAgICAgIHN0YXRlID0g SW5JZGVudGlmaWVyVW5pY29kZUVzY2FwZVN0YXJ0OwogICAgICAgfSBlbHNlIGlmIChjdXJyZW50 ID09ICcwJykgewogICAgICAgICByZWNvcmQ4KGN1cnJlbnQpOwogICAgICAgICBzdGF0ZSA9IElu TnVtMDsKQEAgLTIyNiw4ICsyNDAsNyBAQCBpbnQgTGV4ZXI6OmxleCgpCiAgICAgICAgIHN0YXRl ID0gSW5EZWNpbWFsOwogI2lmbmRlZiBLSlNfUFVSRV9FQ01BCiAgICAgICAgIC8vIDwhLS0gbWFy a3MgdGhlIGJlZ2lubmluZyBvZiBhIGxpbmUgY29tbWVudCAoZm9yIHd3dyB1c2FnZSkKLSAgICAg IH0gZWxzZSBpZiAoY3VycmVudCA9PSAnPCcgJiYgbmV4dDEgPT0gJyEnICYmCi0gICAgICAgICAg ICAgICAgIG5leHQyID09ICctJyAmJiBuZXh0MyA9PSAnLScpIHsKKyAgICAgIH0gZWxzZSBpZiAo Y3VycmVudCA9PSAnPCcgJiYgbmV4dDEgPT0gJyEnICYmIG5leHQyID09ICctJyAmJiBuZXh0MyA9 PSAnLScpIHsKICAgICAgICAgc2hpZnQoMyk7CiAgICAgICAgIHN0YXRlID0gSW5TaW5nbGVMaW5l Q29tbWVudDsKICAgICAgICAgLy8gc2FtZSBmb3IgLS0+CkBAIC0yNjAsOCArMjczLDcgQEAgaW50 IExleGVyOjpsZXgoKQogICAgIC8vIEVzY2FwZSBTZXF1ZW5jZXMgaW5zaWRlIG9mIHN0cmluZ3MK ICAgICBjYXNlIEluRXNjYXBlU2VxdWVuY2U6CiAgICAgICBpZiAoaXNPY3RhbERpZ2l0KGN1cnJl bnQpKSB7Ci0gICAgICAgIGlmIChjdXJyZW50ID49ICcwJyAmJiBjdXJyZW50IDw9ICczJyAmJgot ICAgICAgICAgICAgaXNPY3RhbERpZ2l0KG5leHQxKSAmJiBpc09jdGFsRGlnaXQobmV4dDIpKSB7 CisgICAgICAgIGlmIChjdXJyZW50ID49ICcwJyAmJiBjdXJyZW50IDw9ICczJyAmJiBpc09jdGFs RGlnaXQobmV4dDEpICYmIGlzT2N0YWxEaWdpdChuZXh0MikpIHsKICAgICAgICAgICByZWNvcmQx Nihjb252ZXJ0T2N0YWwoY3VycmVudCwgbmV4dDEsIG5leHQyKSk7CiAgICAgICAgICAgc2hpZnQo Mik7CiAgICAgICAgICAgc3RhdGUgPSBJblN0cmluZzsKQEAgLTMwMyw4ICszMTUsNyBAQCBpbnQg TGV4ZXI6OmxleCgpCiAgICAgICB9CiAgICAgICBicmVhazsKICAgICBjYXNlIEluVW5pY29kZUVz Y2FwZToKLSAgICAgIGlmIChpc0hleERpZ2l0KGN1cnJlbnQpICYmIGlzSGV4RGlnaXQobmV4dDEp ICYmCi0gICAgICAgICAgaXNIZXhEaWdpdChuZXh0MikgJiYgaXNIZXhEaWdpdChuZXh0MykpIHsK KyAgICAgIGlmIChpc0hleERpZ2l0KGN1cnJlbnQpICYmIGlzSGV4RGlnaXQobmV4dDEpICYmIGlz SGV4RGlnaXQobmV4dDIpICYmIGlzSGV4RGlnaXQobmV4dDMpKSB7CiAgICAgICAgIHJlY29yZDE2 KGNvbnZlcnRVbmljb2RlKGN1cnJlbnQsIG5leHQxLCBuZXh0MiwgbmV4dDMpKTsKICAgICAgICAg c2hpZnQoMyk7CiAgICAgICAgIHN0YXRlID0gSW5TdHJpbmc7CkBAIC0zNDAsMTEgKzM1MSwxMiBA QCBpbnQgTGV4ZXI6OmxleCgpCiAgICAgICB9CiAgICAgICBicmVhazsKICAgICBjYXNlIEluSWRl bnRpZmllcjoKLSAgICAgIGlmIChpc0lkZW50TGV0dGVyKGN1cnJlbnQpIHx8IGlzRGVjaW1hbERp Z2l0KGN1cnJlbnQpKSB7CisgICAgICBpZiAoaXNJZGVudFBhcnQoY3VycmVudCkpCiAgICAgICAg IHJlY29yZDE2KGN1cnJlbnQpOwotICAgICAgICBicmVhazsKLSAgICAgIH0KLSAgICAgIHNldERv bmUoSWRlbnRpZmllcik7CisgICAgICBlbHNlIGlmIChjdXJyZW50ID09ICdcXCcpCisgICAgICAg IHN0YXRlID0gSW5JZGVudGlmaWVyVW5pY29kZUVzY2FwZVN0YXJ0OworICAgICAgZWxzZQorICAg ICAgICBzZXREb25lKElkZW50aWZpZXIpOwogICAgICAgYnJlYWs7CiAgICAgY2FzZSBJbk51bTA6 CiAgICAgICBpZiAoY3VycmVudCA9PSAneCcgfHwgY3VycmVudCA9PSAnWCcpIHsKQEAgLTQxOSw2 ICs0MzEsMjEgQEAgaW50IExleGVyOjpsZXgoKQogICAgICAgfSBlbHNlCiAgICAgICAgIHNldERv bmUoTnVtYmVyKTsKICAgICAgIGJyZWFrOworICAgIGNhc2UgSW5JZGVudGlmaWVyVW5pY29kZUVz Y2FwZVN0YXJ0OgorICAgICAgaWYgKGN1cnJlbnQgPT0gJ3UnKQorICAgICAgICBzdGF0ZSA9IElu SWRlbnRpZmllclVuaWNvZGVFc2NhcGU7CisgICAgICBlbHNlCisgICAgICAgIHNldERvbmUoQmFk KTsKKyAgICAgIGJyZWFrOworICAgIGNhc2UgSW5JZGVudGlmaWVyVW5pY29kZUVzY2FwZToKKyAg ICAgIGlmIChpc0hleERpZ2l0KGN1cnJlbnQpICYmIGlzSGV4RGlnaXQobmV4dDEpICYmIGlzSGV4 RGlnaXQobmV4dDIpICYmIGlzSGV4RGlnaXQobmV4dDMpKSB7CisgICAgICAgIHJlY29yZDE2KGNv bnZlcnRVbmljb2RlKGN1cnJlbnQsIG5leHQxLCBuZXh0MiwgbmV4dDMpKTsKKyAgICAgICAgc2hp ZnQoMyk7CisgICAgICAgIHN0YXRlID0gSW5JZGVudGlmaWVyOworICAgICAgfSBlbHNlIHsKKyAg ICAgICAgc2V0RG9uZShCYWQpOworICAgICAgfQorICAgICAgYnJlYWs7CiAgICAgZGVmYXVsdDoK ICAgICAgIGFzc2VydCghIlVuaGFuZGxlZCBzdGF0ZSBpbiBzd2l0Y2ggc3RhdGVtZW50Iik7CiAg ICAgfQpAQCAtNDMzLDggKzQ2MCw3IEBAIGludCBMZXhlcjo6bGV4KCkKICAgfQogCiAgIC8vIG5v IGlkZW50aWZpZXJzIGFsbG93ZWQgZGlyZWN0bHkgYWZ0ZXIgbnVtZXJpYyBsaXRlcmFsLCBlLmcu ICIzaW4iIGlzIGJhZAotICBpZiAoKHN0YXRlID09IE51bWJlciB8fCBzdGF0ZSA9PSBPY3RhbCB8 fCBzdGF0ZSA9PSBIZXgpCi0gICAgICAmJiBpc0lkZW50TGV0dGVyKGN1cnJlbnQpKQorICBpZiAo KHN0YXRlID09IE51bWJlciB8fCBzdGF0ZSA9PSBPY3RhbCB8fCBzdGF0ZSA9PSBIZXgpICYmIGlz SWRlbnRTdGFydChjdXJyZW50KSkKICAgICBzdGF0ZSA9IEJhZDsKIAogICAvLyB0ZXJtaW5hdGUg c3RyaW5nCkBAIC01NTAsOCArNTc2LDcgQEAgaW50IExleGVyOjpsZXgoKQogCiBib29sIExleGVy Ojppc1doaXRlU3BhY2UoKSBjb25zdAogewotICByZXR1cm4gKGN1cnJlbnQgPT0gJyAnIHx8IGN1 cnJlbnQgPT0gJ1x0JyB8fAotICAgICAgICAgIGN1cnJlbnQgPT0gMHgwYiB8fCBjdXJyZW50ID09 IDB4MGMgfHwgY3VycmVudCA9PSAweGEwKTsKKyAgcmV0dXJuIChjdXJyZW50ID09ICdcdCcgfHwg Y3VycmVudCA9PSAweDBiIHx8IGN1cnJlbnQgPT0gMHgwYyB8fCB1X2NoYXJUeXBlKGN1cnJlbnQp ID09IFVfU1BBQ0VfU0VQQVJBVE9SKTsKIH0KIAogYm9vbCBMZXhlcjo6aXNMaW5lVGVybWluYXRv cigpCkBAIC01NjIsMzIgKzU4NywzMiBAQCBib29sIExleGVyOjppc0xpbmVUZXJtaW5hdG9yKCkK ICAgICAgIHNraXBMRiA9IHRydWU7CiAgIGVsc2UgaWYgKGxmKQogICAgICAgc2tpcENSID0gdHJ1 ZTsKLSAgcmV0dXJuIGNyIHx8IGxmOworICByZXR1cm4gY3IgfHwgbGYgfHwgY3VycmVudCA9PSAw eDIwMjggfHwgY3VycmVudCA9PSAweDIwMjk7CiB9CiAKLWJvb2wgTGV4ZXI6OmlzSWRlbnRMZXR0 ZXIodW5zaWduZWQgc2hvcnQgYykKK2Jvb2wgTGV4ZXI6OmlzSWRlbnRTdGFydCh1bnNpZ25lZCBz aG9ydCBjKQogewotICAvKiBUT0RPOiBhbGxvdyBvdGhlciBsZWdpdGltYXRlIHVuaWNvZGUgY2hh cnMgKi8KLSAgcmV0dXJuIChjID49ICdhJyAmJiBjIDw9ICd6JyB8fAotICAgICAgICAgIGMgPj0g J0EnICYmIGMgPD0gJ1onIHx8Ci0gICAgICAgICAgYyA9PSAnJCcgfHwgYyA9PSAnXycpOworICBy ZXR1cm4gKFVfR0VUX0dDX01BU0soYykgJiAoVV9HQ19MX01BU0sgfCBVX0dDX05MX01BU0spKSB8 fCBjID09ICckJyB8fCBjID09ICdfJzsKIH0KIAotYm9vbCBMZXhlcjo6aXNEZWNpbWFsRGlnaXQo dW5zaWduZWQgc2hvcnQgYykKK2Jvb2wgTGV4ZXI6OmlzSWRlbnRQYXJ0KHVuc2lnbmVkIHNob3J0 IGMpCiB7Ci0gIHJldHVybiAoYyA+PSAnMCcgJiYgYyA8PSAnOScpOworICByZXR1cm4gKFVfR0VU X0dDX01BU0soYykgJiAoVV9HQ19MX01BU0sgfCBVX0dDX05MX01BU0sgfCBVX0dDX01OX01BU0sg fCBVX0dDX01DX01BU0sgfCBVX0dDX05EX01BU0sgfCBVX0dDX1BDX01BU0spKSB8fCBjID09ICck JyB8fCBjID09ICdfJzsKK30KKworc3RhdGljIGJvb2wgaXNEZWNpbWFsRGlnaXQodW5zaWduZWQg c2hvcnQgYykKK3sKKyAgcmV0dXJuIGMgPj0gJzAnICYmIGMgPD0gJzknOwogfQogCiBib29sIExl eGVyOjppc0hleERpZ2l0KHVuc2lnbmVkIHNob3J0IGMpCiB7Ci0gIHJldHVybiAoYyA+PSAnMCcg JiYgYyA8PSAnOScgfHwKLSAgICAgICAgICBjID49ICdhJyAmJiBjIDw9ICdmJyB8fAotICAgICAg ICAgIGMgPj0gJ0EnICYmIGMgPD0gJ0YnKTsKKyAgcmV0dXJuIGMgPj0gJzAnICYmIGMgPD0gJzkn IHx8IGMgPj0gJ2EnICYmIGMgPD0gJ2YnIHx8IGMgPj0gJ0EnICYmIGMgPD0gJ0YnOwogfQogCi1i b29sIExleGVyOjppc09jdGFsRGlnaXQodW5zaWduZWQgc2hvcnQgYykgY29uc3QKK3N0YXRpYyBi b29sIGlzT2N0YWxEaWdpdCh1bnNpZ25lZCBzaG9ydCBjKQogewotICByZXR1cm4gKGMgPj0gJzAn ICYmIGMgPD0gJzcnKTsKKyAgcmV0dXJuIGMgPj0gJzAnICYmIGMgPD0gJzcnOwogfQogCiBpbnQg TGV4ZXI6Om1hdGNoUHVuY3R1YXRvcih1bnNpZ25lZCBzaG9ydCBjMSwgdW5zaWduZWQgc2hvcnQg YzIsCkBAIC03MDUsNyArNzMwLDcgQEAgaW50IExleGVyOjptYXRjaFB1bmN0dWF0b3IodW5zaWdu ZWQgc2hvcgogICB9CiB9CiAKLXVuc2lnbmVkIHNob3J0IExleGVyOjpzaW5nbGVFc2NhcGUodW5z aWduZWQgc2hvcnQgYykgY29uc3QKK3N0YXRpYyB1bnNpZ25lZCBzaG9ydCBzaW5nbGVFc2NhcGUo dW5zaWduZWQgc2hvcnQgYykKIHsKICAgc3dpdGNoKGMpIHsKICAgY2FzZSAnYic6CkBAIC03MjAs MTkgKzc0NSwxMiBAQCB1bnNpZ25lZCBzaG9ydCBMZXhlcjo6c2luZ2xlRXNjYXBlKHVuc2lnCiAg ICAgcmV0dXJuIDB4MEM7CiAgIGNhc2UgJ3InOgogICAgIHJldHVybiAweDBEOwotICBjYXNlICci JzoKLSAgICByZXR1cm4gMHgyMjsKLSAgY2FzZSAnXCcnOgotICAgIHJldHVybiAweDI3OwotICBj YXNlICdcXCc6Ci0gICAgcmV0dXJuIDB4NUM7CiAgIGRlZmF1bHQ6CiAgICAgcmV0dXJuIGM7CiAg IH0KIH0KIAotdW5zaWduZWQgc2hvcnQgTGV4ZXI6OmNvbnZlcnRPY3RhbCh1bnNpZ25lZCBzaG9y dCBjMSwgdW5zaWduZWQgc2hvcnQgYzIsCi0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIHVuc2lnbmVkIHNob3J0IGMzKSBjb25zdAorc3RhdGljIHVuc2lnbmVkIHNob3J0IGNv bnZlcnRPY3RhbCh1bnNpZ25lZCBzaG9ydCBjMSwgdW5zaWduZWQgc2hvcnQgYzIsIHVuc2lnbmVk IHNob3J0IGMzKQogewogICByZXR1cm4gKChjMSAtICcwJykgKiA2NCArIChjMiAtICcwJykgKiA4 ICsgYzMgLSAnMCcpOwogfQpAQCAtODIwLDcgKzgzOCw3IEBAIGJvb2wgTGV4ZXI6OnNjYW5SZWdF eHAoKQogICAgIHNoaWZ0KDEpOwogICB9CiAKLSAgd2hpbGUgKGlzSWRlbnRMZXR0ZXIoY3VycmVu dCkpIHsKKyAgd2hpbGUgKGlzSWRlbnRQYXJ0KGN1cnJlbnQpKSB7CiAgICAgcmVjb3JkMTYoY3Vy cmVudCk7CiAgICAgc2hpZnQoMSk7CiAgIH0KQEAgLTg3NSwzICs4OTMsNSBAQCBVU3RyaW5nICpM ZXhlcjo6bWFrZVVTdHJpbmcoVUNoYXIgKmJ1ZmZlCiAgIHN0cmluZ3NbbnVtU3RyaW5ncysrXSA9 IHN0cmluZzsKICAgcmV0dXJuIHN0cmluZzsKIH0KKworfQpJbmRleDoga2pzL2xleGVyLmgKPT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PQpSQ1MgZmlsZTogL2N2cy9yb290L0phdmFTY3JpcHRDb3JlL2tqcy9sZXhlci5oLHYK cmV0cmlldmluZyByZXZpc2lvbiAxLjEwCmRpZmYgLXAgLXUgLXIxLjEwIGtqcy9sZXhlci5oCi0t LSBranMvbGV4ZXIuaAkxNCBKdWwgMjAwNSAxODoyNzowMiAtMDAwMAkxLjEwCisrKyBranMvbGV4 ZXIuaAk4IFNlcCAyMDA1IDE1OjM0OjIxIC0wMDAwCkBAIC02Nyw2ICs2Nyw4IEBAIG5hbWVzcGFj ZSBLSlMgewogICAgICAgICAgICAgICAgICBJbkVzY2FwZVNlcXVlbmNlLAogICAgICAgICAgICAg ICAgICBJbkhleEVzY2FwZSwKICAgICAgICAgICAgICAgICAgSW5Vbmljb2RlRXNjYXBlLAorICAg ICAgICAgICAgICAgICBJbklkZW50aWZpZXJVbmljb2RlRXNjYXBlU3RhcnQsCisgICAgICAgICAg ICAgICAgIEluSWRlbnRpZmllclVuaWNvZGVFc2NhcGUsCiAgICAgICAgICAgICAgICAgIE90aGVy LAogICAgICAgICAgICAgICAgICBCYWQgfTsKIApAQCAtMTAwLDIwICsxMDIsMTcgQEAgbmFtZXNw YWNlIEtKUyB7CiAKICAgICBib29sIGlzV2hpdGVTcGFjZSgpIGNvbnN0OwogICAgIGJvb2wgaXNM aW5lVGVybWluYXRvcigpOwotICAgIGJvb2wgaXNPY3RhbERpZ2l0KHVuc2lnbmVkIHNob3J0IGMp IGNvbnN0OwogCiAgICAgaW50IG1hdGNoUHVuY3R1YXRvcih1bnNpZ25lZCBzaG9ydCBjMSwgdW5z aWduZWQgc2hvcnQgYzIsCiAgICAgICAgICAgICAgICAgICAgICAgICB1bnNpZ25lZCBzaG9ydCBj MywgdW5zaWduZWQgc2hvcnQgYzQpOwotICAgIHVuc2lnbmVkIHNob3J0IHNpbmdsZUVzY2FwZSh1 bnNpZ25lZCBzaG9ydCBjKSBjb25zdDsKLSAgICB1bnNpZ25lZCBzaG9ydCBjb252ZXJ0T2N0YWwo dW5zaWduZWQgc2hvcnQgYzEsIHVuc2lnbmVkIHNob3J0IGMyLAotICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICB1bnNpZ25lZCBzaG9ydCBjMykgY29uc3Q7CisKICAgcHVibGljOgogICAg IHN0YXRpYyB1bnNpZ25lZCBjaGFyIGNvbnZlcnRIZXgodW5zaWduZWQgc2hvcnQgYzEpOwogICAg IHN0YXRpYyB1bnNpZ25lZCBjaGFyIGNvbnZlcnRIZXgodW5zaWduZWQgc2hvcnQgYzEsIHVuc2ln bmVkIHNob3J0IGMyKTsKICAgICBzdGF0aWMgVUNoYXIgY29udmVydFVuaWNvZGUodW5zaWduZWQg c2hvcnQgYzEsIHVuc2lnbmVkIHNob3J0IGMyLAogICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICB1bnNpZ25lZCBzaG9ydCBjMywgdW5zaWduZWQgc2hvcnQgYzQpOwotICAgIHN0YXRpYyBi b29sIGlzSWRlbnRMZXR0ZXIodW5zaWduZWQgc2hvcnQgYyk7Ci0gICAgc3RhdGljIGJvb2wgaXNE ZWNpbWFsRGlnaXQodW5zaWduZWQgc2hvcnQgYyk7CisgICAgc3RhdGljIGJvb2wgaXNJZGVudFN0 YXJ0KHVuc2lnbmVkIHNob3J0IGMpOworICAgIHN0YXRpYyBib29sIGlzSWRlbnRQYXJ0KHVuc2ln bmVkIHNob3J0IGMpOwogICAgIHN0YXRpYyBib29sIGlzSGV4RGlnaXQodW5zaWduZWQgc2hvcnQg Yyk7CiAKICNpZmRlZiBLSlNfREVCVUdfTUVNCkluZGV4OiBranMvc2ltcGxlX251bWJlci5oCj09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT0KUkNTIGZpbGU6IC9jdnMvcm9vdC9KYXZhU2NyaXB0Q29yZS9ranMvc2ltcGxlX251 bWJlci5oLHYKcmV0cmlldmluZyByZXZpc2lvbiAxLjExCmRpZmYgLXAgLXUgLXIxLjExIGtqcy9z aW1wbGVfbnVtYmVyLmgKLS0tIGtqcy9zaW1wbGVfbnVtYmVyLmgJMTQgSnVsIDIwMDUgMTg6Mjc6 MDMgLTAwMDAJMS4xMQorKysga2pzL3NpbXBsZV9udW1iZXIuaAk4IFNlcCAyMDA1IDE1OjM0OjIx IC0wMDAwCkBAIC0yNiw3ICsyNiwxMiBAQAogI2luY2x1ZGUgPG1hdGguaD4KICNpbmNsdWRlIDxz dHJpbmcuaD4KIAorI2lmIFdJTjMyCisjaW5jbHVkZSA8ZmxvYXQuaD4KKyNkZWZpbmUgSVNfTkVH QVRJVkVfWkVSTyhudW0pIChfZnBjbGFzcyhudW0pID09IF9GUENMQVNTX05aKQorI2Vsc2UKICNk ZWZpbmUgSVNfTkVHQVRJVkVfWkVSTyhudW0pIChudW0gPT0gMC4wICYmICFtZW1jbXAoJm51bSwg JlNpbXBsZU51bWJlcjo6bmVnWmVybywgc2l6ZW9mKGRvdWJsZSkpKQorI2VuZGlmCiAKIG5hbWVz cGFjZSBLSlMgewogICAgIGNsYXNzIFZhbHVlSW1wOwpJbmRleDogdGVzdHMvbW96aWxsYS9lY21h L0FycmF5LzE1LjQuNS4xLTEuanMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQpSQ1MgZmlsZTogL2N2cy9yb290L0phdmFT Y3JpcHRDb3JlL3Rlc3RzL21vemlsbGEvZWNtYS9BcnJheS8xNS40LjUuMS0xLmpzLHYKcmV0cmll dmluZyByZXZpc2lvbiAxLjEKZGlmZiAtcCAtdSAtcjEuMSB0ZXN0cy9tb3ppbGxhL2VjbWEvQXJy YXkvMTUuNC41LjEtMS5qcwotLS0gdGVzdHMvbW96aWxsYS9lY21hL0FycmF5LzE1LjQuNS4xLTEu anMJMTMgT2N0IDIwMDMgMDI6MDU6MjggLTAwMDAJMS4xCisrKyB0ZXN0cy9tb3ppbGxhL2VjbWEv QXJyYXkvMTUuNC41LjEtMS5qcwk4IFNlcCAyMDA1IDE1OjM0OjIxIC0wMDAwCkBAIC0xMDQsNyAr MTA0LDkgQEAgZnVuY3Rpb24gZ2V0VGVzdENhc2VzKCkgewogCiAKICAgICBmb3IgKCB2YXIgaSA9 IDBYMDAyMCwgVEVTVF9TVFJJTkcgPSAidmFyIEEgPSBuZXcgQXJyYXkoICIgOyBpIDwgMHgwMGZm OyBpKysgKSB7Ci0gICAgICAgIFRFU1RfU1RSSU5HICs9ICJcJ1xcIisgU3RyaW5nLmZyb21DaGFy Q29kZSggaSApICsiXCciOworICAgICAgICBpZiAoaSA9PSAweEFEKSAvLyBNdXN0IHNraXAgb3Zl ciBzb2Z0IGh5cGhlbiwgYmVjYXVzZSBpdCdzIGEgQ2YgY2hhcmFjdGVyIGFuZCBjYW4ndCBiZSBw YXJzZWQuCisgICAgICAgICAgICBpKys7CisgICAgICAgIFRFU1RfU1RSSU5HICs9ICInXFwiKyBT dHJpbmcuZnJvbUNoYXJDb2RlKCBpICkgKyInIjsKICAgICAgICAgaWYgKCBpIDwgMHgwMEZGIC0g MSAgICkgewogICAgICAgICAgICAgVEVTVF9TVFJJTkcgKz0gIiwiOwogICAgICAgICB9IGVsc2Ug ewpAQCAtMTEyLDcgKzExNCw3IEBAIGZ1bmN0aW9uIGdldFRlc3RDYXNlcygpIHsKICAgICAgICAg fQogICAgIH0KIAotICAgIHZhciBMRU5HVEggPSAweDAwZmYgLSAweDAwMjA7CisgICAgdmFyIExF TkdUSCA9IDB4MDBmZiAtIDB4MDAyMCAtIDE7CiAKICAgICBhcnJheVtpdGVtKytdID0gbmV3IFRl c3RDYXNlKCAgIFNFQ1RJT04sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBU RVNUX1NUUklORyArIiBBWzE1MF0gPSAnaGVsbG8nOyBBWzE1MF0iLAo= 3817 2005-09-08 10:42:37 -0700 2005-09-08 10:42:37 -0700 Deseret test case javascript-deseret.html text/html 245 ap 77u/PEhFQUQ+CjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9o dG1sOyBjaGFyc2V0PXV0Zi04Ij4KPHRpdGxlPkRlc2VyZXQgY2hhcmFjdGVycyBpbiBKYXZhc2Ny aXB0PC90aXRsZT4KPC9IRUFEPgo8Qk9EWSBvbkxvYWQ9ImphdmFzY3JpcHQ6dmFyIPCQkIA9J/CQ kIAnO2FsZXJ0KPCQkIApOyI+CkFuIGFsZXJ0IHNob3VsZCBoYXZlIGFwcGVhcmVkIG9uIGxvYWQu CjwvQk9EWT4KPC9odG1sPgo= 3821 2005-09-08 21:43:25 -0700 2005-09-08 21:43:25 -0700 normalization test case javascript-decomposed.html text/html 298 ap PEhFQUQ+CjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1s OyBjaGFyc2V0PXV0Zi04Ij4KPHRpdGxlPk5vcm1hbGl6aW5nIEphdmFTY3JpcHQgdGV4dCB0byBj YW5vbmljYWwgZm9ybSBDPC90aXRsZT4KPC9IRUFEPgo8Qk9EWSBvbkxvYWQ9ImphdmFzY3JpcHQ6 dmFyINC40LzRj1/RhNCw0LjMhtC70LA9J1dvcmtzIGZpbmUnO2FsZXJ0KNC40LzRj1/RhNCw0LnQ u9CwKTsiPgo8cD5BbiBhbGVydCBzaG91bGQgaGF2ZSBhcHBlYXJlZCBvbiBsb2FkLjwvcD4KPC9C T0RZPgo8L2h0bWw+Cg==