24420 2009-03-06 01:30:03 -0800 Consider doing encoding detection before decoding starts 2022-09-16 15:34:51 -0700 1 1 1 Unclassified WebKit Page Loading 528+ (Nightly build) Mac All NEW https://bugs.webkit.org/show_bug.cgi?id=245305 P2 Normal --- 1 jshin webkit-unassigned ap oldest_to_newest 112555 0 jshin 2009-03-06 01:30:03 -0800 Spun off bug 16482. Character encoding auto-detection is done after a part of an input file is decoded. We have to find a way to do the encoding detection before any decoding starts. 113405 1 jshin 2009-03-12 10:20:24 -0700 bug 16482 comment #30 How about doing a similar "pre-scanning"(?) with auto-detection? We can ask autodetector to return "an indicator of undeterministic/unknown) until it reaches a confidence level higher than a certain value (that thresold should be internal to each implementation of autodetector). Then, we can set a flag (similar to m_checkedForHeadCharset). Would it be a good plan for bug 24420? bug 16482 comment #31 Yes, it may we ll be. Are you well familiar with how Gecko encoding detector works? How much data does it normally need to choose an encoding with confidence (and how much data would you expect our detector to need)? 113412 2 jshin 2009-03-12 10:38:25 -0700 (In reply to comment #1) > bug 16482 comment #31 > > Yes, it may we ll be. Are you well familiar with how Gecko encoding detector > works? How much data does it normally need to choose an encoding with > confidence (and how much data would you expect our detector to need)? I have an overall understanding of Gecko's algorithms (byte unigram and byte bigram), but don't know those criteria you asked about. However, I know two (or three) people who developed them and can ask as a short cut instead of reading the source code. As for ICU's encoding detector, it uses byte-trigrams and my not-so-scientific experiment indicates that it can be pretty reliable after 1kB or so. Needless to say, if the first 1kB is (almost) entirely made up of ASCII bytes, it'd not work as well. ICU gives back a confidence level. The way it's calculated is not so elaborate, though. So, we need to employee some heuristics when using it. 113416 3 ap 2009-03-12 10:51:07 -0700 Looks like ICU also has an upper limit: <http://sourceforge.net/mailarchive/message.php?msg_name=49382ACE.2080804%40icu-project.org>.