| I first saw the term "UTF8" when I was learning Cocoa
programming for Mac OSX. It appeared to me that Cocoa was using term UTF8
for char *; I thought that UTF8 was just a new name for the ASCII code. We
used to call 8-bit text as ASCII text. Win32 API later started calling it
ANSI. I was not keeping track of what happened to ASCII. I thought Apple
just wanted to give it a different name for it. In Cocoa, NSString class
represents a text string, and it internally stores a string in Unicode. It
has a function that converts NSString to UTF8 string. I wondered what was
going to happen if I converted an NSString that includes multi-byte
characters into UTF8 string and then converted it back to NSString.
The following test code can be compiled from Terminal by typing: g++ conv.m
-framework Cocoa
List: conv.m
#import <Cocoa/Cocoa.h>
int main(void)
{
int i;
NSAutoreleasePool *pool=[[NSAutoreleasePool alloc] init];
unsigned char utf16[]={0x5c,0x71,0x5d,0xdd,0x03,0xc0,0x00,'A',0x03,0x94,0,0};
NSString *rawString=[[NSString alloc]
initWithBytes:utf16
length:10
encoding:NSUTF16BigEndianStringEncoding];
printf("Original Unicode:\n");
for(i=0; i<[rawString length]; i++)
{
printf("%04x ",[rawString characterAtIndex:i]);
}
printf("\n");
const char *utf8=[rawString UTF8String];
// You will see two kanji characters, greek letter pi, alphabet A, and greek
// letter Delta, if the terminal supports UTF8.
printf("UTF8: %s\n",utf8);
for(i=0; 0!=utf8[i]; i++)
{
int c=utf8[i];
c&=0xff;
printf("%02x ",c);
}
printf("\n");
printf("Recovered Unicode:\n");
NSString *verifyString=[[NSString alloc] initWithUTF8String:utf8];
for(i=0; i<[verifyString length]; i++)
{
printf("%04x ",[verifyString characterAtIndex:i]);
}
printf("\n");
[rawString release];
[verifyString release];
[pool release];
return 0;
}
|
To my surprise, NSString translated Kanji characters (which are clearly
multi-byte characters) into strange code, and it was able to convert it back
to NSString without loss. Wow! That was great. I didn't have to do something
special for dealing with non-English characters. But, I was still thinking
it was Apple's original extension.
By the way, I am using a sort of brute-force method to localize YSFLIGHT.
I am storing all the UI messages in PNG bitmap files. I used to make each
PNG bitmap by Paint Brush, but it was becoming too painful. I decided to
make a program that reads a text file with PNG file names and messages and
outputs UI PNG files. I was working on Japanese localization of YSFLIGHT
Scenery Editor, and I didn't want to make all the PNG bitmap files by hand.
I didn't have to make this program cross platform. Once I make UI bitmaps
on Windows, I could use them in Mac OSX and Linux. I didn't want to spend
too much time on this program, so initially I really did a quick-and-dirty
programming with no effort to make it cross platform. But, later I thought
that since there are such strong YSFLIGHT user communities, if I make this
program available, they may generate localization files for many other
languages. So, I made it less dirty and re-wrote some part.
In process of cleaning my program, I reviewed my code used in Blue
Impulse 3D Graphics SDK, which I developed through my master's research
project. I had been thinking that if I were able to write a class that
generated an off-screen bitmap from a Unicode text, I could use it
everywhere.
Here's what I recalled from my old code. To create an off-screen bitmap
(called Device Independent Bitmap or DIB) in Windows, I needed to use a
function called CreateCompatibleDC, but I needed to pass a valid
Device-Context handle to CreateCompatibleDC, and to get a valid
Device-Context handle, I had to create a window. As far as I remembered,
CreateCompatibleDC used to crash if I gave a NULL pointer as a source
Device-Context handle. The problem was, as soon as I needed a window, I
either needed to get the main Window handle from the application, or I
needed to create a dummy window. If I use the main Window handle from the
application, the application becomes Windows dependent. I need to assume
that the program has a main window, which is not good in many ways. If I
make a dummy window, I can keep the window hidden (SW_HIDE state), and keep
the presence of my dummy window secret from the user. But, it didn't seem to
me a nice clean solution. But, nowadays necessity of the off-screen bitmap
is much higher than before. Win32 API might have been changed so that
CreateCompatibleDC can take a NULL pointer. I went to MSDN, and searched for
CreateCompatibleDC.
I made a good guess! CreateCompatibleDC can now take a NULL pointer and
creates a memory device context. I don't know when Microsoft made this
change. At least I remembered the program crashed when I was programming in
Windows 3.1. I couldn't give NULL pointer as a source Device-Context handle
to CreateCompatibleDC. Maybe I was wrong. My program might have been
crashing for the different purpose, and for some reason it didn't crash when
I gave a non-NULL pointer to CreateCompatibleDC. I don't have Windows
3.1 PC any more and cannot verify whether I was really not able to give NULL
pointer to CreateCompatibleDC.
That time Game SDK was called WinG. I was fully expecting that WinG was
going to support 3D graphics and was utterly disappointed because it didn't.
Nonetheless, support for game programming lead to the dominance of Windows
95 and 98. It was mid 1990s and Unix people and Mac people were thinking
gaming was ignoble. They were essentially on denial. They thought that
computing was for royal purposes such as scientific calculation. Watching
videos, playing music, and 3D graphics were all dirty technologies for them.
Mac was a little different from Unix though. In my impression, Apple liked
some multi-media stuff, but they were thinking desktop publishing was
everything. It was hell difficult to write a game program in Mac OS then.
That's how they lost Microsoft. Things are so much different now as you
know. Microsoft is slowly losing the dominance, and Apple has the momentum.
And, Linux is .... slightly better. At least I can install Ubuntu by myself.
Anyways, it's not a main topic of this page.
So, I realized that I could write a program that creates an off-screen
bitmap and draws a Unicode characters and returns the array of bitmaps. So,
I wrote a simple test program as follows. You can compile from Visual Studio
command prompt as:
cl font.cpp kernel32.lib user32.lib gdi32.lib
List: font.cpp
// cl font.cpp kernel32.lib user32.lib gdi32.lib
// or
// cl font.cpp kernel32.lib user32.lib gdi32.lib /DUNICODE <- This way this code can be checked against default-unicode environment.
#include <windows.h>
#include <stdio.h>
#include <string.h>
static int CALLBACK EnumFontFamExProc(
const LOGFONTA *fontInfo,
const TEXTMETRICA *textMetric,
DWORD fontType,
LPARAM lParam)
{
printf("%s\n",fontInfo->lfFaceName);
return 1;
}
void EnumerateFont(HDC hDC)
{
LOGFONTA fontInfo;
fontInfo.lfCharSet=DEFAULT_CHARSET;
fontInfo.lfFaceName[0]=0;
fontInfo.lfPitchAndFamily=0;
EnumFontFamiliesExA(hDC,&fontInfo,EnumFontFamExProc,(LPARAM)0,0);
}
int main(void)
{
HDC hDC=CreateCompatibleDC(NULL);
EnumerateFont(hDC);
LOGFONTW logFont;
ZeroMemory(&logFont,sizeof(logFont));
logFont.lfHeight=-12; // 12 pixels (Positive 12 for 12 points)
logFont.lfWeight=FW_DONTCARE;
logFont.lfCharSet=DEFAULT_CHARSET;
logFont.lfOutPrecision=OUT_TT_PRECIS;
logFont.lfClipPrecision=CLIP_DEFAULT_PRECIS;
logFont.lfQuality=DEFAULT_QUALITY;
logFont.lfPitchAndFamily=DEFAULT_PITCH;
wcscpy(logFont.lfFaceName,L"Arial"); // Up to LF_FACESIZE chars.
HFONT hFont=CreateFontIndirectW(&logFont);
HGDIOBJ hPrevFont=NULL;
if(NULL!=hFont)
{
printf("Font Created.\n");
hPrevFont=SelectObject(hDC,(HGDIOBJ)hFont);
}
RECT rect;
rect.left=0;
rect.top=0;
rect.right=100;
rect.bottom=100;
SetTextAlign(hDC,TA_LEFT|TA_TOP|TA_NOUPDATECP); // Prerequisite for DrawTextEx
wchar_t str[]={0x5c71,0x5ddd,0x03c0,'A',0x0394,0};
DrawTextExW(hDC,str,4,&rect,DT_TOP|DT_LEFT|DT_CALCRECT,NULL);
const int wid=rect.right;
const int hei=rect.bottom;
const int bitPerPixel=24;
printf("Width=%d Height=%d\n",wid,hei);
BITMAPINFOHEADER hdr=
{
sizeof(BITMAPINFOHEADER),
0,0,1,8,BI_RGB,0,0,0,0,0
};
hdr.biWidth=wid;
hdr.biHeight=-hei;
hdr.biBitCount=bitPerPixel;
hdr.biCompression=BI_RGB;
void *bit;
HBITMAP hBmp=CreateDIBSection(hDC,(BITMAPINFO *)&hdr,DIB_RGB_COLORS,&bit,NULL,0);
HGDIOBJ hPrevBmp;
hPrevBmp=SelectObject(hDC,hBmp);
int bytePerLine=((wid+3)/4)*4*bitPerPixel/8;
unsigned char *dat=(unsigned char *)bit;
int i;
for(i=0; i<bytePerLine*hei; i++)
{
dat[i]=0;
}
SetTextColor(hDC,RGB(255,255,255));
SetBkColor(hDC,RGB(0,0,0));
SetBkMode(hDC,OPAQUE);
SetDCPenColor(hDC,RGB(255,0,0));
rect.left=0;
rect.top=0;
rect.right=wid;
rect.bottom=hei;
DrawTextExW(hDC,str,4,&rect,DT_TOP|DT_LEFT,NULL);
int x,y;
unsigned char *lineTop=dat;
for(y=0; y<hei; y++)
{
for(x=0; x<wid; x++)
{
if(0!=lineTop[x*3])
{
printf("1");
}
else
{
printf("0");
}
}
lineTop+=bytePerLine;
printf("\n");
}
if(NULL!=hPrevFont)
{
printf("Reset font.\n");
SelectObject(hDC,hPrevFont);
}
if(NULL!=hPrevBmp)
{
printf("Reset bmp.\n");
SelectObject(hDC,hPrevBmp);
}
DeleteObject(hBmp);
DeleteDC(hDC);
return 0;
}
|
Then I needed to do the same on Mac OSX and Linux to support majority of
the popular operating systems. It was easy on Mac OSX. It took a while to
see the resolution (DPI) of NSBitmapImageRep, but I later realized that it
was by default 1 pixel == 1 point. I think it is a common struggle for
programmers that we need to specify the font size by points not pixels no
matter where you are writing your font. It makes sense to specify font size
by pixels when I am writing font on the monitor. But, since default
NSBitmapImageRep's resolution is 72 DPI, I didn't have to worry about point
to pixel unit conversion. Another issue was the usage of NSAttributedString.
For this, sample codes in "Cocoa & Objective-C: Up and Running" (Scott
Stevenson, O'Reilly) around page 360 helped me a lot. You can compile the
following sample code by:
g++ font.m –framework Cocoa
List: font.m
#import <Cocoa/Cocoa.h>
int main(void)
{
NSAutoreleasePool *pool=[[NSAutoreleasePool alloc] init];
NSFontManager *fontManager=[NSFontManager sharedFontManager];
if(NULL==fontManager)
{
printf("No font manager is present.\n");
}
else
{
NSArray *availableFonts=[fontManager availableFonts];
int i;
for(i=0; i<[availableFonts count]; i++)
{
NSString *str=[availableFonts objectAtIndex:i];
printf("[%d] %s\n",i,[str UTF8String]);
}
}
float fontSize=20.0f;
NSFont *font=[NSFont fontWithName:@"Times-Roman" size:fontSize];
NSMutableParagraphStyle *parag=[[NSMutableParagraphStyle alloc] init];
[parag setMaximumLineHeight:fontSize];
printf("line spacing %f\n",[[NSParagraphStyle defaultParagraphStyle] lineSpacing]);
printf("paragraph spacing %f\n",[[NSParagraphStyle defaultParagraphStyle] paragraphSpacing]);
printf("paragraph spacing before %f\n",[[NSParagraphStyle defaultParagraphStyle] paragraphSpacingBefore]);
printf("minimum line height %f\n",[[NSParagraphStyle defaultParagraphStyle] minimumLineHeight]);
printf("maximum line height %f\n",[[NSParagraphStyle defaultParagraphStyle] maximumLineHeight]);
printf("line height multiplet %f\n",[[NSParagraphStyle defaultParagraphStyle] lineHeightMultiple]);
NSRect rect;
NSMutableDictionary *attributes=[NSMutableDictionary dictionary];
[attributes setObject:font forKey:NSFontAttributeName];
[attributes setObject:[NSNumber numberWithFloat:0.0] forKey:NSKernAttributeName];
[attributes setObject:parag forKey:NSParagraphStyleAttributeName];
[attributes setObject:[NSColor whiteColor] forKey:NSForegroundColorAttributeName];
[attributes setObject:[NSColor blackColor] forKey:NSBackgroundColorAttributeName];
unsigned char utf16[]={0x5c,0x71,0x5d,0xdd,0x03,0xc0,0x00,'A',0x03,0x94,0,0};
NSString *rawString=[[NSString alloc] initWithBytes:utf16 length:10 encoding:NSUTF16BigEndianStringEncoding];
NSAttributedString *string=[[NSAttributedString alloc] initWithString:rawString attributes:attributes];
NSSize stringSize=[string size];
printf("Size %f %f\n",stringSize.width,stringSize.height);
NSBitmapImageRep *bmpImg=[[NSBitmapImageRep alloc]
initWithBitmapDataPlanes:NULL
pixelsWide:(int)stringSize.width
pixelsHigh:(int)stringSize.height
bitsPerSample:8 // 8 bit per component
samplesPerPixel:3 // 3 components
hasAlpha:NO
isPlanar:NO
colorSpaceName:NSDeviceRGBColorSpace
bytesPerRow:0
bitsPerPixel:32];
// The following commented-out block is optional since the resolution of
// a bitmap seems tobe 72 DPI by default, which makes 1 pixel per 1 point.
/* int pixelsWide=[bmpImg pixelsWide],pixelsHigh=[bmpImg pixelsHigh];
printf("BMP Pix =%d %d\n",pixelsWide,pixelsHigh);
NSSize sizeInPoint={pixelsWide,pixelsHigh};
[bmpImg setSize:sizeInPoint]; // This makes 1 point per pixel
sizeInPoint=[bmpImg size];
printf("BMP Size=%f %f\n",sizeInPoint.width,sizeInPoint.height); */
NSGraphicsContext *gc=[NSGraphicsContext
graphicsContextWithBitmapImageRep:bmpImg];
NSGraphicsContext *prevGc=[NSGraphicsContext currentContext];
[NSGraphicsContext setCurrentContext:gc];
[gc setShouldAntialias:NO];
[[NSColor darkGrayColor] set];
rect.origin.x=0;
rect.origin.y=0;
rect.size.width=stringSize.width;
rect.size.height=stringSize.height;
NSRectFill(rect);
[[NSColor whiteColor] set];
rect.origin.x=0;
rect.origin.y=0;
rect.size.width=stringSize.width;
rect.size.height=stringSize.height;
[string drawWithRect:rect options:NSStringDrawingUsesLineFragmentOrigin];
unsigned char *dat=[bmpImg bitmapData];
unsigned int bytesPerRow=[bmpImg bytesPerRow];
printf("bytesPerRow %d\n",bytesPerRow);
int y;
for(y=0; y<(int)stringSize.height; y++)
{
int x;
for(x=0; x<(int)stringSize.width; x++)
{
if(0==dat[y*bytesPerRow+x*4])
{
printf("0");
}
else
{
printf("1");
}
}
printf("\n");
}
[NSGraphicsContext setCurrentContext:prevGc];
[parag release];
[string release];
[rawString release];
[bmpImg release];
[pool release];
return 0;
}
|
It appeared that I was getting closer to terminate my brute-force (PNG
based) localization soon. But, the problem was Unicode. I thought there were
no standardized Unicode text-file format.
I remembered that my favorite text editor gave me a choice of text
encoding like EUC, Shift-JIS (both are for Japanese characters), and
Unicode. I got curious what kind of binary I would get if I choose Unicode
as text encoding. I opened up the text editor, and chose "Save As", and
opened the "Encoding" drop list. Then I saw "Unicode (UTF-8)". What? UTF-8
is supposed to be 8-bit encoding. How can it store multi-byte characters?
.... Suddenly the strange code from [NSString UTF8String] flashed back, and
I understood what they were. That was not Apple's original extension. UTF8
was a standard encoding for a Unicode text. I felt I was too far away from
the localization business too long. Then I searched on Wikipedia and
found how I could convert wchar to/from UTF8. (http://en.wikipedia.org/wiki/UTF-8).
It's easy. I could write two-way converter in a few minutes.
List: Sample UTF8 <-> WChar conversion functions.
#include <stdio.h>
// Reference: http://en.wikipedia.org/wiki/UTF-8
template <class WCHAR_TYPE>
size_t UnicodeWChar2UTF8(size_t lOut,char out[],WCHAR_TYPE in[])
{
unsigned int iPtr=0,oPtr=0;
unsigned char *uOut=(unsigned char *)out;
while(oPtr<lOut-1 && 0!=in[iPtr])
{
if(in[iPtr]<=0x7f)
{
uOut[oPtr]=(unsigned char)in[iPtr];
oPtr++;
}
else if(in[iPtr]<=0x7ff)
{
if(lOut-2<=oPtr)
{
// Buffer overrun. Cannot put zero at the end.
break;
}
unsigned int low6=in[iPtr]&0x3f;
unsigned int high5=(in[iPtr]>>6)&0x1f;
uOut[oPtr++]=(unsigned char)(0xc0|high5);
uOut[oPtr++]=(unsigned char)(0x80|low6);
}
else if(in[iPtr]<=0xffff)
{
if(lOut-3<=oPtr)
{
// Buffer overrun. Cannot put zero at the end.
break;
}
unsigned int low6=in[iPtr]&0x3f;
unsigned int mid6=(in[iPtr]>>6)&0x3f;
unsigned int high4=(in[iPtr]>>12)&0x0f;
uOut[oPtr++]=(unsigned char)(0xe0|high4);
uOut[oPtr++]=(unsigned char)(0x80|mid6);
uOut[oPtr++]=(unsigned char)(0x80|low6);
}
else // The following block will never be used unless wchar_t is greater than 16-bit.
{
unsigned long inChar=in[iPtr];
unsigned int bit00to05=inChar&0x3f;
unsigned int bit06to11=(inChar>>6)&0x3f;
unsigned int bit12to17=(inChar>>12)&0x3f;
unsigned int bit18to23=(inChar>>18)&0x3f;
unsigned int bit24to29=(inChar>>24)&0x3f;
unsigned int bit30=(inChar>>30)&0x01;
if(in[iPtr]<=0x1fffff)
{
if(lOut-4<=oPtr)
{
// Buffer overrun. Cannot put zero at the end.
break;
}
uOut[oPtr++]=(unsigned char)(0xf0|(bit18to23&0x07));
uOut[oPtr++]=(unsigned char)(0x80|bit12to17);
uOut[oPtr++]=(unsigned char)(0x80|bit06to11);
uOut[oPtr++]=(unsigned char)(0x80|bit00to05);
}
else if(in[iPtr]<=0x3ffffff)
{
if(lOut-5<=oPtr)
{
// Buffer overrun. Cannot put zero at the end.
break;
}
uOut[oPtr++]=(unsigned char)(0xf8|(bit24to29&0x03));
uOut[oPtr++]=(unsigned char)(0x80|bit18to23);
uOut[oPtr++]=(unsigned char)(0x80|bit12to17);
uOut[oPtr++]=(unsigned char)(0x80|bit06to11);
uOut[oPtr++]=(unsigned char)(0x80|bit00to05);
}
else if(in[iPtr]<=0x7fffffff)
{
if(lOut-6<=oPtr)
{
// Buffer overrun. Cannot put zero at the end.
break;
}
uOut[oPtr++]=(unsigned char)(0xfc|bit30);
uOut[oPtr++]=(unsigned char)(0x80|bit24to29);
uOut[oPtr++]=(unsigned char)(0x80|bit18to23);
uOut[oPtr++]=(unsigned char)(0x80|bit12to17);
uOut[oPtr++]=(unsigned char)(0x80|bit06to11);
uOut[oPtr++]=(unsigned char)(0x80|bit00to05);
}
}
iPtr++;
}
if(oPtr<lOut)
{
out[oPtr]=0;
oPtr++;
}
return oPtr;
}
template <class WCHAR_TYPE>
size_t UnicodeUTF8toWChar(size_t lOut,WCHAR_TYPE out[],const char in[])
{
unsigned int iPtr=0,oPtr=0;
unsigned char *uIn=(unsigned char *)in;
while(oPtr<lOut-1 && 0!=uIn[iPtr])
{
if(0==(uIn[iPtr]&0x80))
{
out[oPtr]=uIn[iPtr];
iPtr++;
}
else if(0xc0==(uIn[iPtr]&0xe0))
{
if(0==uIn[iPtr+1])
{
// Pre-mature termination.
break;
}
unsigned int high5=uIn[iPtr]&0x1f;
unsigned int low6=uIn[iPtr+1]&0x3f;
out[oPtr]=(WCHAR_TYPE)((high5<<6)|low6);
iPtr+=2;
}
else if(0xe0==(uIn[iPtr]&0xf0))
{
if(0==uIn[iPtr+1] || 0==uIn[iPtr+2])
{
// Pre-mature termination.
break;
}
unsigned int high4=uIn[iPtr]&0x0f;
unsigned int mid6=uIn[iPtr+1]&0x3f;
unsigned int low6=uIn[iPtr+2]&0x3f;
out[oPtr]=(WCHAR_TYPE)((high4<<12)|(mid6<<6)|low6);
iPtr+=3;
}
else if(0xf0==(uIn[iPtr]&0xf8))
{
if(0==uIn[iPtr+1] || 0==uIn[iPtr+2] || 0==uIn[iPtr+3])
{
// Pre-mature termination.
break;
}
unsigned int bit18to20=uIn[iPtr]&0x07;
unsigned int bit12to17=uIn[iPtr+1]&0x3f;
unsigned int bit06to11=uIn[iPtr+2]&0x3f;
unsigned int bit00to05=uIn[iPtr+3]&0x3f;
out[oPtr]=(WCHAR_TYPE)((bit18to20<<18)|(bit12to17<<12)|(bit06to11<<6)|bit00to05);
iPtr+=4;
}
else if(0xf8==(uIn[iPtr]&0xfc))
{
if(0==uIn[iPtr+1] || 0==uIn[iPtr+2] || 0==uIn[iPtr+3] || 0==uIn[iPtr+4])
{
// Pre-mature termination.
break;
}
unsigned int bit24to25=uIn[iPtr]&0x03;
unsigned int bit18to23=uIn[iPtr+1]&0x3f;
unsigned int bit12to17=uIn[iPtr+2]&0x3f;
unsigned int bit06to11=uIn[iPtr+3]&0x3f;
unsigned int bit00to05=uIn[iPtr+4]&0x3f;
out[oPtr]=(WCHAR_TYPE)((bit24to25<<24)|(bit18to23<<18)|(bit12to17<<12)|(bit06to11<<6)|bit00to05);
iPtr+=5;
}
else if(0xfc==(uIn[iPtr]&0xfe))
{
if(0==uIn[iPtr+1] || 0==uIn[iPtr+2] || 0==uIn[iPtr+3] || 0==uIn[iPtr+4])
{
// Pre-mature termination.
break;
}
unsigned int bit30=uIn[iPtr]&0x01;
unsigned int bit24to29=uIn[iPtr+1]&0x3f;
unsigned int bit18to23=uIn[iPtr+2]&0x3f;
unsigned int bit12to17=uIn[iPtr+3]&0x3f;
unsigned int bit06to11=uIn[iPtr+4]&0x3f;
unsigned int bit00to05=uIn[iPtr+5]&0x3f;
out[oPtr]=(WCHAR_TYPE)((bit30<<30)|(bit24to29<<24)|(bit18to23<<18)|(bit12to17<<12)|(bit06to11<<6)|bit00to05);
iPtr+=6;
}
else
{
// Invalid utf8 string
break;
}
oPtr++;
}
if(oPtr<lOut)
{
out[oPtr]=0;
oPtr++;
}
return oPtr;
}
// Should be encoded to E5 B1 B1 E5 B7 9D CF 80
int main(void)
{
wchar_t in[]={0x5c71,0x5ddd,0x03c0,0};
char utf8[256];
int i;
for(i=0; 0!=in[i]; i++)
{
printf("%04x ",in[i]);
}
printf("\n");
size_t bytes=UnicodeWChar2UTF8 <wchar_t> (256,utf8,in);
printf("Encoded to %d bytes (including the last zero terminator).\n",bytes);
for(i=0; 0!=utf8[i]; i++)
{
printf("%02x ",((unsigned char *)utf8)[i]);
}
printf("\n");
wchar_t backConv[256];
size_t wchars=UnicodeUTF8toWChar <wchar_t> (256,backConv,utf8);
printf("Decoded to %d wchars (including the last zero terminator).\n",wchars);
for(i=0; 0!=backConv[i]; i++)
{
printf("%04x ",backConv[i]);
}
printf("\n");
return 0;
}
|
Unicode problem solved.
I knew how to make an off-screen bitmap from a Unicode string in Windows
and Mac OSX. iOS is almost same as Mac OSX. So, if I found how to do the
same in Linux, I could support most of the major operating systems.
But, doing it in X-Window was the most difficult among three systems.
X-Window is obsolete. I want to avoid if I can. But, it comes with Linux by
default. If I want to support Linux, I cannot avoid it.
What surprised me was organization of the off-screen bitmap. If I made a
24-bit bitmap, and if I pick three consecutive bytes, I expect that three
bytes will give red, green, and blue component. That's the case in Windows
and Mac OSX. But, when I made a bitmap (called pixmap in X) and retrieved a
bitmap, first of the 24 bits were packed in a sequence, followed by the
second, third, ..., 24th bits.
I first wrote a program that draws "Text" in black color on white
background. The first byte of the bitmap was $E0. I wondered why the hell I
was getting a byte other than $00 and $FF? I couldn't solve the mystery for
an hour or so. Then I gave up and took a shower. When I was taking a shower,
I noticed the first 5 bits of $E0 were zero. Maybe the top bar of letter T
was expanded to five pixels, and that's why I was getting $E0. So, I scanned
bit by bit and printed 1 and 0 on the console, and I saw the pattern that I
expected. It reminded me of the structure of the Video-RAM of my very first
8-bit computer. It's ok. In X-Window, you can experience how people used to
do things decades ago. You can learn history. That's the joy of X-Window. At
least I figured how to do it in X-Window. You can compile the
following program by: g++ font.cpp -lX11
List: font.cpp
#include <stdio.h>
#include <string.h>
#include <X11/Xlib.h>
#include <X11/Xutil.h>
#include <X11/Xlocale.h>
/* Memo for future reference
Meaning of font string.
http://en.wikipedia.org/wiki/X_logical_font_description
X-Window drawing functions
http://www-yano.is.tokushima-u.ac.jp/~mituhara/software_e/draw.html
XwcDrawString sample code
http://homepage3.nifty.com/rio_i/lab/xlib/011mbcs.htm
This page suggests:
- Environment variable LANG should be set.
- A function call setlocale(LC_CTYPE,""); is required.
May also be possible to use XLoadFont, XSetFont
Font font=XLoadFont(ysXDsp,"6x10");
*/
int main(void)
{
Display *dsp=XOpenDisplay(NULL);
if(NULL!=dsp)
{
printf("Display Opened. (Depth=%d)\n",DefaultDepth(dsp,0));
Window rootWin=DefaultRootWindow(dsp);
int nFont;
char **fontList=XListFonts(dsp,"*iso10646*",1024,&nFont);
int i;
for(i=0; i<nFont; i++)
{
printf("[%d] %s\n",i,fontList[i]);
}
const char *baseFontName=
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-1,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-2,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-4,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-5,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-6,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-7,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-8,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-9,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-15,"
"-*-*-*-*-*-*-16-*-*-*-*-*-iso10646-1";
const char *localeStr=setlocale(LC_CTYPE,"");
if(NULL==localeStr || True!=XSupportsLocale())
{
setlocale(LC_CTYPE,"en_US.UTF-8"); // Desperate attempt!
}
int missCharSetCount;
char *defStr,**missCharSet;
XFontSet fontSet=XCreateFontSet(
dsp,
baseFontName,
&missCharSet,
&missCharSetCount,
&defStr);
if(NULL!=fontSet)
{
printf("Acquired font set (Missing=%d)\n",missCharSetCount);
wchar_t str[]={0x5c71,0x5ddd,0x03c0,'A',0x0394,0};
XRectangle inkRect,logRect;
int ext=XwcTextExtents(fontSet,str,5,&inkRect,&logRect);
printf("Extent %d\n",ext);
printf("Ink Rect %d %d %d %d\n",inkRect.x,inkRect.y,inkRect.width,inkRect.height);
printf("Log Rect %d %d %d %d\n",logRect.x,logRect.y,logRect.width,logRect.height);
int bmpWid=ext,bmpHei=logRect.height;
int depth=DefaultDepth(dsp,0);
Pixmap pixmap=XCreatePixmap(dsp,rootWin,bmpWid,bmpHei,depth);
GC gc=XCreateGC(dsp,pixmap,0,0);
XSetForeground(dsp,gc,BlackPixel(dsp,0));
XFillRectangle(dsp,pixmap,gc,0,0,bmpWid,bmpHei);
XSetForeground(dsp,gc,WhitePixel(dsp,0));
XwcDrawString(dsp,pixmap,fontSet,gc,0,bmpHei-1,str,5);
XImage *image=XGetImage(dsp,pixmap,0,0,bmpWid,bmpHei,~0,XYPixmap);
if(NULL!=image)
{
printf("Acquired Image %dx%d\n",image->width,image->height);
printf("Bitmap Unit=%d\n",image->bitmap_unit);
printf("Bytes per line=%d\n",image->bytes_per_line);
printf("Depth=%d\n",image->depth);
int x,y;
for(y=0; y<image->height; y++)
{
unsigned char mask=1;
unsigned char *byteTop=(unsigned char *)
image->data+y*image->bytes_per_line;
for(x=0; x<image->width; x++)
{
if((*byteTop)&mask)
{
printf("1");
}
else
{
printf("0");
}
if(0!=(mask&0x80))
{
byteTop++;
mask=1;
}
else
{
mask<<=1;
}
}
printf("\n");
}
/* Attempt 1 Failed. Apparently, each of 24 layers is
packed contiguously. I was forgetting X-Window is from the stone age.
int byte_per_pixel=image->bitmap_unit/8;
for(y=0; y<image->height; y++)
{
for(x=0; x<20; x++)
{
int offset=y*(64*byte_per_pixel)+x*byte_per_pixel;
unsigned char *ptr=(unsigned char *)image->data+offset;
printf("%02x%02x%02x ",ptr[0],ptr[1],ptr[2]);
}
printf("\n");
} */
}
XDestroyImage(image);
XFreeGC(dsp,gc);
XFreePixmap(dsp,pixmap);
XFreeFontSet(dsp,fontSet);
}
XCloseDisplay(dsp);
}
}
|
Now I know how to do it in Windows, Mac OSX, and Linux. Next thing is to
make a class that has a set of interfaces that allows the application to
access these features. Then I will be able to get rid of brute-force PNG-based
localization. 04/26/2011 |