Unicode and Beyond. Data Translation and Comprehension in the Quest for Total Global Data Quality. Are You Ready for Global Data?

Transcription

1 Unicode and Beyond Data Translation and Comprehension in the Quest for Total Global Data Quality Are You Ready for Global Data? Explore what Unicode is, how it benefits business, and what it means in the context of global data processing. This document also covers the current Trillium Software Unicode strategy and how it is used by the Trillium Software System to provide total global data quality. 25 Linnell Circle, Billerica, MA (ph), (fx)

2 Unicode Defined Among the most significant recent global software technology trends are Unicode and the tools that support it. Unicode is a method of indexing all of the world s major character sets (also called scripts or encodings) for computer use. A universal, language-, platform- and applicationindependent character index, it allows machines to read, understand and communicate data in most of the world s languages. Most importantly, Unicode allows business functions to be language independent able to use all international data, regardless of whether it is represented in Japanese Kanji, Korean, Chinese, Greek, Hebrew or another character set. Unicode-enable programs such as the Trillium Software System allow business to expand across borders without undermining mission-critical systems or sacrificing information quality. Unicode currently contains about 94,140 different characters that cover all of the principal written languages of the Americas, Europe, the Middle East, Africa, Asia and the Pacific Rim. The character sets that comprise the newest Unicode version (3.2) are listed at right. To index characters, Unicode uses 16 bit encoding (i.e., UTF-16) or double-byte character encoding (16 bits equals two bytes). These terms are by contrast to smaller character sets, such as ASCII, which use only 8 bit or single-byte encoding. The larger byte-value character sets in Unicode accommodate Chinese, Japanese, and Korean scripts, which can contain thousands of symbols and far exceed the 265-character capabilities of single-byte encoding. Unicode contains both single- and double-byte character sets. It is divided into sections that represent a particular language, most of which contain 256 characters or fewer. The Chinese, Korean, and Japanese sections (referred to collectively as CJK) exceed this count. CHARACTER SETS SUPPORTED BY THE TRILLIUM SOFTWARE SYSTEM: Arabic Baltic Celtic Chinese, Simplified Chinese, Traditional Croatian Cyrillic Devanagari Greek Gujarati Gurmukhi Hebrew Icelandic Japanese Korean Latin Malayalam Nordic Romanian Slavic Symbol Thai Turkish Ukrainian Vietnamese typographic symbols technical symbols Unicode enablement allows the Trillium Software System to seamlessly process global content, such as this Japanese data. 05/07/02 CONFIDENTIAL AND PROPRIETARY PAGE 2 OF 6

3 Business and Technical Benefits of Unicode Unicode hasn t made obsolete the hundreds of different character encoding systems that predate it. Many companies still use multiple singlebyte encodings or legacy character sets in conjunction with each other. This is advantageous primarily when the companies use data in only one or a few languages. For companies with data spanning many languages and character sets, however, Unicode offers several advantages over patchwork encoding systems: 1. Greater growth support: As business networks become more global and complex, Unicode enables systems to accept international data without significant new development investments. 2. Lower IT demands: Unicode averts the common challenge of conflicting encoding systems. 3. Higher data fidelity: As a single encoding, Unicode eliminates the risk of corruption that occurs whenever data is passed between different encodings or platforms. The strategic benefits of Unicode-enabled systems are still more impressive. Across the enterprise, Unicode into client-server or multi-tiered applications and Web sites offers significant cost savings over the use of legacy character sets. 1. Unicode multiplies the value of software investments, allowing a single product to be deployed across multiple platforms, languages and countries without re-engineering. 2. Unicode enables real-time data from global subsidiaries to support operations, since worldwide data can be centrally stored and processed. 3. Unicode empowers companies to create and enact stronger global visions, using worldwide data across systems to empower decisions about global strategy and foster better understanding across the enterprise. 4. Unicode reduces knowledge latency, allowing users to access and manipulate global information from anywhere in their preferred language. These benefits and advantages have spurred Unicode s widespread adoption, although they come at a price. Unicode support demands almost complete recoding for most applications. This substantial capital burden is the chief reason Unicode is not more universally employed. 05/07/02 CONFIDENTIAL AND PROPRIETARY PAGE 3 OF 6

4 Unicode Capabilities of the Trillium Software System Trillium Software provides the leading global data quality solution. Our international business capabilities extend even beyond the industry s most complete Unicode support to encompass an understanding of data content based on its context. The Trillium Software System is the only data quality solution that provides both translation and comprehension of global data. The Trillium Software System supports all Unicode character sets, including all seven UTF character transformation formats. These capabilities are mature and proven in real-world business environments. In addition, because many characters have multiple meanings, the software uses business rules and context-sensitive processing to ensure the correct interpretation of data. Of course, many systems still use legacy data encodings, and we continue to support those as well. The standard modules of the Trillium Software System support both single-byte and double-byte character sets, using complex encoding transformation routines to ensure that the global data quality process remains complete and efficient. Trillium Software recognizes that our customers businesses are becoming increasingly globalized. To help them continuously meet the demands of international business the Trillium Software System provides all of the benefits of Unicode data cited in this document and compounds those benefits by improving the quality of international data. Data transformation Although Unicode has standardized character encoding for more than 10 years, most of the world s data is still subject to single-byte encoding or legacy character encoding systems. In fact, the vast majority of data uses only the first 256 characters of Unicode (the extended Latin character set), which is also single-byte encoding. The Trillium Software System is optimized for performance and resource use in single-byte and multi-byte environments. Transforming Unicode data to take advantage of this optimized environment, the software: 1) Identifies with which section of the Unicode table the data corresponds. 2) Processes the data with double-byte specific modules or converts data from Unicode into an appropriate single-byte encoding (codepage). Our platform-independent conversion library allows us to seamlessly transform Unicode data into a particular codepage and back again, if necessary. 3) Processes the data through the Trillium Software System using country- and language-specific business rules built with the same codepage or Unicode encoding as targeted data. 4) Optionally converts the improved data back into Unicode using the conversion library. TRANSFORMATION IN PRACTICE Consider Greek data. Greek characters occupy the Unicode range 0x370 to 0x37F. The codepage for Greek data is 737, which allows the Trillium Software System to transform the data for cleansing. As part of a massive international table development effort, Trillium Software has also built business rules for Greek data based on codepage 737 encoding. When Greek data enters the Trillium Software System, it transforms the Unicode characters according to codepage 737. This lets the software correlate the transformed data with the Greek-specific business rules. After the transformation, the Trillium Software System processes the now-single-byte data through Trillium modules. After the business rules are applied and the data is cleansed, the conversion library routines transform the data back into Unicode for deployment to a Unicodeenabled database. 05/07/02 CONFIDENTIAL AND PROPRIETARY PAGE 4 OF 6

5 Trillium Software Fig. 2: A Sample Codepage mapping up to 265 characters CHARACTER SETS SUPPORTED BY TRILLIUM SOFTWARE BUSINESS RULES 02/2001 Latin Japanese Chinese Greek Korean Cyrillic Hebrew Within the double-byte CJK character sets, the Trillium Software System identifies data elements by word breaks (space characters). In Japanese data, which might not include word breaks, Trillium Software has developed enhanced transformation modules that work entirely in Unicode. We have replicated all of the data quality functionality necessary to ensure high quality within Japanese data. The Japanese version of the Trillium Software System is currently live in more than 30 major Japanese companies. Data normalization A built-in advantage of our conversion library is its ability to normalize data. In addition to mapping the data, the Trillium Software System can transform any character to another [similar to the Find and Replace functions in popular software], giving identical forms to words with identical meanings. This process of normalization often improves data cleansing results. For example, the converter might change a character-accent combination into a single character, standardizing character sets across records and enhancing its match accuracy. Increasing global data support Trillium Software System is already completely Unicode enabled. This technical backbone allows us to throw the weight of our development efforts into building the country- and language-specific business rules that provide true data understanding. Rules development is an ongoing developmental drive at Trillium Software. In particular, we are currently expanding our group of Unicode-enabled transformation modules to include Chinese and Korean linguistics. This will enable the Trillium Software System to process business names, personal names, geographic data, complex inexact matching and business data cleansing in support of total information quality enterprise-wide. 05/07/02 CONFIDENTIAL AND PROPRIETARY PAGE 5 OF 6

6 Usage Notice This document is 2002 by Trillium Software, a Division of Harte-Hanks. It is illegal to reproduce, distribute or broadcast this document in any context without express written permission from Trillium Software. Use for any purposes other than those expressly granted by Trillium Software is expressly prohibited by law, and may result in severe civil and criminal penalties. Violators will be prosecuted to the maximum extent possible. This document is subject to change at any time by Trillium Software. Modification by any party other than Trillium Software is prohibited. Trillium Software hereby disclaims all warranties and conditions with regard to this information, including warranties and conditions of merchantability, whether express, implied or statutory, fitness for a particular purpose, title and noninfringement. In no event shall Trillium Software. and/or its respective suppliers be liable for any special, indirect or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out of or in connection with the use or performance of information available from this white paper. 05/07/02 CONFIDENTIAL AND PROPRIETARY PAGE 6 OF 6