Dochula Pass, Bhutan

If you use my Unicode character pickers, you may have noticed some changes recently. I’ve moved several pickers on to version 14. Most of the noticeable changes are in the location and styling of elements on the UI – the features remain pretty much unchanged.

Pages have acquired a header at the top (which is typically hidden), that provides links to related pages, and integrates the style into that of the rest of the site. What you don’t see is a large effort to tidy the code base and style sheets.

So far, I have changed the following: Arabic block, Armenian, Balinese, Bengali, Khmer, IPA, Lao, Mongolian, Myanmar, and Tibetan.

I will convert more as and when I get time.

However, in parallel, I have already made a start on version 15, which is a significant rewrite. Gone are the graphics, to be replaced by characters and webfonts. This makes a huge improvement to the loading time of the page. I’m also hoping to introduce more automated transcription methods, and simpler shape matching approaches.

Some of the pickers I already upgraded to version 14 have mechanisms for transcription and shape-based identification that took a huge effort to create, and will take a substantial effort to upgrade to version 15. So they may stay as they are for a while. However, easier to handle and new pickers will move to the new format.

Actually, I already made a start with Gurmukhi v15, which yanks that picker out of the stone-age and into the future. There’s also a new picker for the Uighur language that uses v15 technology. I’ll write separate blogs about those.

 

[By the way, if you are viewing the pickers on a mobile device such as an iPad, don't forget to turn Autofocus off (click on 'more controls' to find the switch). This will stop the onscreen keyboard popping up, annoyingly, each time you try to tap on a character.]

Picture of the page in action.

>> Use UniView

This version updates the app per the changes during beta phase of the specification, so that it now reflects the finalised Unicode 7.0.0.

The initial in-app help information displayed for new users was significantly updated, and the help tab now links directly to the help page.

A more significant improvement was the addition of links to character descriptions (on the right) where such details exist. This finally reintegrates the information that was previously pulled in from a database. Links are only provided where additional data actually exists. To see an example, go here and click on See character notes at the bottom right.

Rather than pull the data into the page, the link opens a new window containing the appropriate information. This has advantages for comparing data, but it was also the best solution I could find without using PHP (which is no longer available on the server I use). It also makes it easier to edit the character notes, so the amount of such detail should grow faster. In fact, some additional pages of notes were added along with this upgrade.

A pop-up window containing resource information used to appear when you used the query to show a block. This no longer happens.

Changes in version 7beta

I forgot to announce this version on my blog, so for good measure, here are the (pretty big) changes it introduced.

This version adds the 2,834 new characters encoded in the Unicode 7.0.0 beta, including characters for 23 new scripts. It also simplified the user interface, and eliminated most of the bugs introduced in the quick port to JavaScript that was the previous version.

Some features that were available in version 6.1.0a are still not available, but they are minor.

Significant changes to the UI include the removal of the ‘popout’ box, and the merging of the search input box with that of the other features listed under Find.

In addition, the buttons that used to appear when you select a Unicode block have changed. Now the block name appears near the top right of the page with a I icon icon. Clicking on the icon takes you to a page listing resources for that block, rather than listing the resources in the lower right part of UniView’s interface.

UniView no longer uses a database to display additional notes about characters. Instead, the information is being added to HTML files.

Following up on a suggestion by Nathan Hill of SOAS, I added a la-swe glyph to the default view of the picker alongside the medial consonants. If you click on it, it produces U+1039 MYANMAR SIGN VIRAMA + U+101C MYANMAR LETTER LA.

I also rearranged the font pull-down list a little, adding information about what fonts are available on your Mac OS X or Windows7 system, and added a placeholder, like I did recently for the Khmer picker.

You can find the Myanmar picker at http://rishida.net/scripts/pickers/myanmar/

Following up on a very good suggestion by Roger Sperberg, I added two webfonts to the Khmer picker and arranged the font selection list so that you can see which fonts are available on your Mac OS X or Windows7 system.

The webfonts make it possible to use the picker on an iPad or other device that doesn’t have a Khmer font installed. I added two webfonts because one worked on my iPad and the other didn’t, and it was vice versa on my Snow Leopard Macbook.

I also added an HTML5 placeholder for the output box. (I’m wishing you could style that differently from the standard content – and wishing that markup designers would think about this sort of thing and stop using attributes for natural language text…).

You can find the Khmer picker at http://rishida.net/scripts/pickers/khmer/

Picture of the page in action.

>> Use UniView

The main addition in this version is a couple of buttons that appear when you ask UniView to display a block.

Clicking on Show annotated list generates a list of all characters in the block, with annotations.

Clicking on Show script links displays a list of links to key sources of information about the script of the block, links to relevant articles and apps on the rishida.net site, and related fonts and input methods. This provides a very quick way of finding this information. One particularly useful link (‘Historical documentation’, which links to a Scriptsource.org page) allows you to find the proposals for all additions to Unicode related to the relevant script. These proposals are a mine of useful information about the individual characters in a block, and SIL staff should get a medal for trawling through all the relevant data to provide this.

In addition, there were some changes to the user interface, including the following:

  • The order of information in the lower right panel (detailed character information) was slightly changed, and two alterative representations of the character were added: an HTML escape, and a URI escape.
  • The search box at the top left was constrained to appear closer to the other controls when the window is stretched wide.

Various bugs were also fixed.

>> Use it

This HTML page allows you to expand information in the lines of the UnicodeData.txt file, edit them and generate a new version. It also checks the data for validity in a number of areas.

It can be helpful if you have the misfortune to pore over the source code of the UnicodeData.txt file and find your eyes blurring as you count fields. And it is particularly useful for people submitting proposals for new scripts or characters to the Unicode Consortium, to help them generate correct lists of unicode properties for inclusion in the proposal. (You can even build the whole thing in the UI, error free, by starting with a number of blank lines, such as 1111;NAME;;;;;;;;;;;;;.)

The image below shows the page in action. I dropped in a couple of lines from the Ahom script proposal, and vandalised them slightly. The first panel shows that the app has spotted an error. I used the column to the right to edit out the error in the second panel, and regenerated the lines in the box below.

Picture of the page in action.

Having made edits you can copy paste the output back into the top box to send it through the sausage machine again, and check that there are no remaining errors.

You can add a whole script block at a time to the top box, or a single line – as you like.

Well, it’s a bit esoteric, but hopefully it will be useful to someone somewhere.

Picture of the page in action.

>> Use it

This picker contains characters from the Unicode Balinese block needed for writing the Balinese language. Characters needed for Sasak are also available in the Advanced section. Balinese musical notation characters are not included.

About the tool: Pickers allow you to quickly create phrases in a script by clicking on Unicode characters arranged in a way that aids their identification. Pickers are likely to be most useful if you don’t know a script well enough to use the native keyboard. The arrangement of characters also makes it much more usable than a regular character map utility.

About this picker: Characters are grouped to aid input. The consonant block includes characters needed for Kawi and Sanskrit as well as the native Balinese characters, all arranged according to the Brahmi pronunciation grid.

The picker has only a default view and a font grid view. It’s difficult to put in the time for the shape-based, keyboard-based, and various transcription-based views in some other pickers. In a new departure, however, I have included a list of Latin characters on the default view to assist in writing transcriptions alongside Balinese text.

There is, however, a significant issue with this picker, due to the lack of support for Balinese as a script in computers. The only Unicode-based Balinese font I know of is Aksara Bali, but that font seems to only work as expected in Firefox on Mac OS X. Furthermore, the Aksara Bali font doesn’t handle ra repa as described in the Unicode Standard. The sequence <consonant , adeg-adeg, ra repa> produces a visible adeg-adeg, rather than the post-fixed form of ra repa. The sequence <consonant , vowel sign ra repa> produces the post-fixed form of ra repa, rather than the subjoined form. You can produce the post-fixed form with this font by using <consonant , vowel sign ra repa> and the subjoined form by using <consonant , adeg-adeg, ra, pepet>, but these sequences will produce content that cannot be matched against sequences using the Unicode approach, and content that may fail with other Unicode-compliant fonts in the future.

Hopefully some new, fully Unicode-compliant fonts will come along soon. This is one of the most beautiful scripts I have come across.

(Btw, I’m working on a set of notes for Balinese characters, linked from UniView, with some feature innovations to get around the font issue. Look out for that later. And I’m thinking I should develop a Javanese picker to go with this one. Just need a bit of time…)

For the curious, here’s the first article of the Universal Declaration of Human Rights, as typed in the Balinese picker. Translation by Tri Ediwan (reproduced from Omniglot).

Picture of the page in action.

>> Use UniView

The major change in this update is the update of the data to support Unicode version 6.1.0, which should be released today. (See the list of links to new Unicode blocks below.)

There are also a number of feature and bug related changes.

What UniView does: Look up and see characters (using graphics or fonts) and property information, view whole character blocks or custom ranges, select characters to paste into your document, paste in and discover unknown characters, search for characters, do hex/dec/ncr conversions, highlight character types, etc. etc. Supports Unicode 6.1 and written with Web Standards to work on a variety of browsers. No need to install anything.

List of changes:

  • One significant change enables you to display information in a separate window, rather than overwriting the information currently displayed. This can be done by typing/pasting/dragging a set of characters or character code values into the new Popout area and selecting the  icon alongside the Characters or Copy & paste input fields (depending on what you put in the popout window).

  • Two new icons were added to the Copy & paste area:

    Analyse Clicking on this will display the characters in the area in the lower right part of the page with all relevant characters converted to uppercase, lowercase and titlecase. Characters that had no case conversion information are also listed.

    Analyse Clicking on this produces the same kind of output as clicking on the icon just above, but shows the mappings for those characters that have been changed, eg. e→E.

  • Where character information displayed in the lower right panel has a case or decomposition mapping, UniView now displays the characters involved, rather than just giving the hex value(s), eg. Uppercase mapping: 0043 C. You will need a font on your system to see the characters displayed in this way, but whether or not you have a font, this provides a quick and easy way to copy the case-changed character (rather than having to copy the hex value and convert it first).

  • There is also a new line, slightly further down, when UniView is in graphic mode. This line starts with ‘As text:’, and shows the character using whatever default font you have on your system. Of course, if you don’t have a font that includes that character you won’t see it. This has been added to make it easier to copy and paste a character into text.

  • There is also a new line, slightly further down, when UniView is in graphic mode. This line starts with ‘As text:’, and shows the character using whatever default font you have on your system. Of course, if you don’t have a font that includes that character you won’t see it. This has been added to make it easier to copy and paste a character into text.

  • Fixed some small bugs, such as problems with search when U+29DC INCOMPLETE INFINITY is returned.

Enjoy.

Here are direct links to the new blocks added to Unicode 6.1:

One of the more useful features of UniView is its ability to list the characters in a string with names and codepoints. This is particularly useful when you can’t tell what a string of characters contains because you don’t have a font, or because the script is too complex, etc.

'ishida' in Persian in  nastaliq font style

For example, I was recently sent an email where my name was written in Persian as ایشی‌دا. The image shows how it looks in a nastaliq font.

To see the component characters, drop the string into UniView’s Copy & Paste field and click on the downwards pointing arrow icon. Here is the result:

list of characters

Note how you can now see that there’s an invisible control character in the string. Note also that you see a graphic image for each character, which is a big help if the string you are investigating is just a sequence of boxes on your system.

Not only can you discover characters in this way, but you can create lists of characters which can be pasted into another document, and customise the format of those lists.

Pasting the list elsewhere

If you select this list and paste it into a document, you’ll see something like this:

  0627  ARABIC LETTER ALEF
  06CC  ARABIC LETTER FARSI YEH
  0634  ARABIC LETTER SHEEN
  06CC  ARABIC LETTER FARSI YEH
  200C  ZERO WIDTH NON-JOINER
  062F  ARABIC LETTER DAL
  0627  ARABIC LETTER ALEF

You can make the characters appear by deselecting Use graphics on the Look up tab. (Of course, you need an arabic font to see the list as intended.)

ا  ‎0627  ARABIC LETTER ALEF
ی  ‎06CC  ARABIC LETTER FARSI YEH
ش  ‎0634  ARABIC LETTER SHEEN
ی  ‎06CC  ARABIC LETTER FARSI YEH
‌  ‎200C  ZERO WIDTH NON-JOINER
د  ‎062F  ARABIC LETTER DAL
ا  ‎0627  ARABIC LETTER ALEF

Customising the list format

What may be less obvious is that you can also customise the format of this list using the settings under the Options tab. For example, using the List format settings, I can produce a list that moves the character column between the number and the name, like this:

  0627  ا  ARABIC LETTER ALEF
  ‎06CC  ی  ARABIC LETTER FARSI YEH
  ‎0634  ش  ARABIC LETTER SHEEN
  ‎06CC  ی  ARABIC LETTER FARSI YEH
  ‎200C  ‌  ZERO WIDTH NON-JOINER
  ‎062F  د  ARABIC LETTER DAL
  ‎0627  ا  ARABIC LETTER ALEF

Or I can remove one or more columns from the list, such as:

  ا  ARABIC LETTER ALEF
  ی  ARABIC LETTER FARSI YEH
  ش  ARABIC LETTER SHEEN
  ی  ARABIC LETTER FARSI YEH
  ‌  ZERO WIDTH NON-JOINER
  د  ARABIC LETTER DAL
  ا  ARABIC LETTER ALEF

With the option Show U+ in lists I can also add or remove the U+ before the codepoint value. For example, this lets me produce the following list:

  ‎U+0627  ARABIC LETTER ALEF
  ‎U+06CC  ARABIC LETTER FARSI YEH
  ‎U+0634  ARABIC LETTER SHEEN
  ‎U+06CC  ARABIC LETTER FARSI YEH
  ‎U+200C  ZERO WIDTH NON-JOINER
  ‎U+062F  ARABIC LETTER DAL
  ‎U+0627  ARABIC LETTER ALEF

Other lists in UniView

We’ve shown how you can make a list of characters in the Cut & Paste box, but don’t forget that you can create lists for a Unicode block, custom range of characters, search list results, or list of codepoint values, etc. And not only that, but you can filter lists in various ways.

Here is just one quick example of how you can obtain a list of numbers for the Devanagari script.

  1. On the Look up tab, select Devanagari from the Unicode block pull down list.
  2. Select Show range as list and deselect (optional) Use graphics.
  3. Under the Filter tab, select Number from the Show properties pull down list.
  4. Click on Make list from highlights

You end up with the following list, that you can paste into your document.

०  ‎0966  DEVANAGARI DIGIT ZERO
१  ‎0967  DEVANAGARI DIGIT ONE
२  ‎0968  DEVANAGARI DIGIT TWO
३  ‎0969  DEVANAGARI DIGIT THREE
४  ‎096A  DEVANAGARI DIGIT FOUR
५  ‎096B  DEVANAGARI DIGIT FIVE
६  ‎096C  DEVANAGARI DIGIT SIX
७  ‎096D  DEVANAGARI DIGIT SEVEN
८  ‎096E  DEVANAGARI DIGIT EIGHT
९  ‎096F  DEVANAGARI DIGIT NINE

(Of course, you can also customise the layout of this list as described in the previous section.)

Try it out.

Reversing the process: from list to string

To complete the circle, you can also cut & paste any of the lists in the blog text above into UniView, to explore each character’s properties or recreate the string.

Select one of the lists above and paste it into the Characters input field on the Look up tab. Hit the downwards pointing arrow icon alongside, and UniView will recreate the list for you. Click on each character to view detailed information about it.

If you want to recreate the string from the list, simply click on the upwards pointing arrow icon below the Copy & paste box, and the list of characters will be reconstituted in the box as a string.

Voila!

Picture of the page in action.

The ‘i18n checker’ is a free service by the W3C that provides information about internationalization-related aspects of your HTML page, and advice on how to improve your use of markup, where needed, to support the multilingual Web.

This latest release uses a new user interface and redesigned source code. It also adds a number of new tests, a file upload facility, and support for HTML5.

This is still a ‘pre-final’ release and development continues. There are already plans to add further tests and features, to translate the user interface, to add support for XHTML5 and polyglot documents, to integrate with the W3C Unicorn checker, and to add various other features. At this stage we are particularly interested in receiving user feedback.

Try the checker and let us know if you find any bugs or have any suggestions.

Picture of the page in action.

>> Use UniView

About the tool: Look up and see characters (using graphics or fonts) and property information, view whole character blocks or custom ranges, select characters to paste into your document, paste in and discover unknown characters, search for characters, do hex/dec/ncr conversions, highlight character types, etc. etc. Supports Unicode 6.0 and written with Web Standards to work on a variety of browsers. No need to install anything.

Latest changes: The majority of changes in this update relate to the user interface. They include the following:

  • Many controls have been grouped under three tabs: Look up, Filter, and Options. Various previously dispersed controls were gathered together under the Filter and Options tabs. Many of the controls have been slightly renamed.
  • The Search control has been moved to the top right of the window, where it is always visible.
  • The old Text Area is now a Copy & Paste control that has a 2-dimensional input box. In browser such as Safari, Chrome and Firefox 4, this box can be stretched by the user to whatever size is preferred.
  • The icon that provides a toggle switch between revealing detailed information for a character in a list or table, or copying that character to the Copy & Paste box has been redesigned. It stands alone and indicates the location of the current outcome using arrows.
    It looks like this: with the two arrows or this with the two arrows.
  • Title text has been provided for all controls, describing briefly what that control does. You can see this information by hovering over the control with the mouse.

Many of these changes were introduced to make it a little easier for newcomers to get to grips with UniView.

There were also some feature changes:

  • The ‘Codepoints’ control was converted to accept text as well as code points and renamed ‘Characters’. By default the control expect hex code point values, but this can be switched using the radio buttons. For text, you would usually use the ‘Copy & Paste’ control, but if you want to check out some characters without disturbing the contents of that control, you can now do so by setting the ‘Character’ radio button on the ‘Characters’ control.
  • The control to look up characters in the Unihan database the icon that looks like a Japanese character was fixed, but also extended to handle multiple characters at a time, opening a separate window for each character. (UniView warns you if you try to open more than 5 windows.)
  • The control to send characters to the Unicode Conversion tool the icon with overlapping boxes was fixed and now puts the character content of the field in the green box of the Converter Tool. If you need to convert hex or decimal code point values, do that in the converter.
  • The Show Age feature now works with lists, not just tables.

It has always been possible to pass a string to the converter in the URI, but that was never documented.

Now it is, and you can pass a string using the q parameter. For example, http://rishida.net/tools/conversion/?q=Crêpes. You can also pass a string with escapes in it, but you will need to be especially careful to percent escape characters such as &, + and # which affect the URI syntax. For example, http://rishida.net/tools/conversion/?q=CrU%2B00EApes.

>> Use it

Inspired by some comments on John Well’s blog, I decided to add a keyboard layout to the IPA picker today. It follows the layout of Mark Huckvale’s Unicode Phonetic Keyboard (UCL) v1.01.

I can’t say I understand why many of the characters are allocated to the keys they are, but I figured that if John Wells uses this keyboard it would be probably worth using its layout.

Picture of the page in action.

>> Use it

This picker contains characters from the Unicode Mongolian block needed for writing the Mongolian language. It doesn’t include Sibe, Todo or Manchu characters. Mongolian is a complex script, and I am still familiarising myself with it. This is an initial trial version of a Mongolian picker, and as people use it and raise feedback I may need to make changes.

About the tool: Pickers allow you to quickly create phrases in a script by clicking on Unicode characters arranged in a way that aids their identification. Pickers are likely to be most useful if you don’t know a script well enough to use the native keyboard. The arrangement of characters also makes it much more usable than a regular character map utility.

About this picker: The output area for this picker is set up for vertical text. However, only Internet Explorer currently supports vertical text display, and only IE8 supports Mongolian’s left-to-right column progression. In addition, it seems that IE doesn’t support ltr columns in textarea elements. The bottom line is that, although the output area is the right shape and position for vertical text, mostly the output will be horizontal. You will see vertical text in IE, but the column positions will look wrong. Nevertheless, in any of these cases, when you cut and paste text into another document, the characters will still be correctly ordered.

Consonants are to the left, and in the order listed in the Wikipedia article about Mongolian text. To their right are vowels, then punctuation, spaces and control characters, and number digits. The variation selectors are positioned just below the consonants.

As you mouse over the letters, the various combining forms appear in a column to the far left. This is to help identify characters, for those less familiar with the alphabet.

Analyser: http://rishida.net/tools/analysestring/

Converter: http://rishida.net/tools/conversion/

The string analyser tool provides information about the characters in a string. One difference in this version is a new section “Data input as graphics”, where you see a horizontal sequence of graphics for each of the characters in the string you are analysing. This can be useful to get a screen snap of the characters. Of course, there is no combining or ligaturing behaviour involved – just a graphic per character.

You can reverse the character order for right-to-left scripts.

Another difference is that you can explode example text in the notes. Take this example: if you click on the Arabic word for Koran (red word near the bottom of the notes), you’ll see a pop-up window in the bottom right corner of the window that lists the characters in that word.

The other change is that the former “Related links” section in the sidebar is now called “Do more”, and the links carry the string you are analysing to the Converter or UniView apps.

Oh, and the page now remembers the options you set between refreshes, which makes life much easier.

The converter tool converts between characters and various escaped character formats. It was changed so that the “View names” button sends the characters to the string analyser tool. This means that you’ll now see graphics for the characters, and that, once on the string analyser page, you can change the amount of information displayed for each character (including showing font-based characters, if you need to).

I also fixed a bug related to the UTF-8 and UTF-16 input. Including spaces after the code values no longer fires off a bug.

PS: The string analyser tool has graphics for all new Unicode 6.0 characters, however I haven’t updated the data for those characters yet. I was planning to do so with the next release of UniView, which should be in October, when the final Unicode database is available.

Next Page »