Sample codes for generating bitmap images from a UNICODE string for Windows, Mac OSX, and Linux (Plus UTF8 <-> WChar conversion functions)
I first saw the term "UTF8" when I was learning Cocoa programming for Mac OSX. It appeared to me that Cocoa was using term UTF8 for char *; I thought that UTF8 was just a new name for the ASCII code. We used to call 8-bit text as ASCII text. Win32 API later started calling it ANSI. I was not keeping track of what happened to ASCII. I thought Apple just wanted to give it a different name for it.

In Cocoa, NSString class represents a text string, and it internally stores a string in Unicode. It has a function that converts NSString to UTF8 string. I wondered what was going to happen if I converted an NSString that includes multi-byte characters into UTF8 string and then converted it back to NSString.  The following test code can be compiled from Terminal by typing: g++ conv.m -framework Cocoa

List: conv.m

#import <Cocoa/Cocoa.h>

int main(void)
{
    int i;
    NSAutoreleasePool *pool=[[NSAutoreleasePool alloc] init];

    unsigned char utf16[]={0x5c,0x71,0x5d,0xdd,0x03,0xc0,0x00,'A',0x03,0x94,0,0};
    NSString *rawString=[[NSString alloc] 
          initWithBytes:utf16 
          length:10 
          encoding:NSUTF16BigEndianStringEncoding];
    printf("Original Unicode:\n");
    for(i=0; i<[rawString length]; i++)
    {
        printf("%04x ",[rawString characterAtIndex:i]);
    }
    printf("\n");



    const char *utf8=[rawString UTF8String];

    // You will see two kanji characters, greek letter pi, alphabet A, and greek 
    // letter Delta, if the terminal supports UTF8.
    printf("UTF8: %s\n",utf8);
    for(i=0; 0!=utf8[i]; i++)
    {
        int c=utf8[i];
        c&=0xff;
        printf("%02x ",c);
    }
    printf("\n");



    printf("Recovered Unicode:\n");
    NSString *verifyString=[[NSString alloc] initWithUTF8String:utf8];
    for(i=0; i<[verifyString length]; i++)
    {
        printf("%04x ",[verifyString characterAtIndex:i]);
    }
    printf("\n");

    [rawString release];
    [verifyString release];

    [pool release];
    return 0;
}

To my surprise, NSString translated Kanji characters (which are clearly multi-byte characters) into strange code, and it was able to convert it back to NSString without loss. Wow! That was great. I didn't have to do something special for dealing with non-English characters. But, I was still thinking it was Apple's original extension.

By the way, I am using a sort of brute-force method to localize YSFLIGHT. I am storing all the UI messages in PNG bitmap files. I used to make each PNG bitmap by Paint Brush, but it was becoming too painful. I decided to make a program that reads a text file with PNG file names and messages and outputs UI PNG files. I was working on Japanese localization of YSFLIGHT Scenery Editor, and I didn't want to make all the PNG bitmap files by hand.

I didn't have to make this program cross platform. Once I make UI bitmaps on Windows, I could use them in Mac OSX and Linux. I didn't want to spend too much time on this program, so initially I really did a quick-and-dirty programming with no effort to make it cross platform. But, later I thought that since there are such strong YSFLIGHT user communities, if I make this program available, they may generate localization files for many other languages. So, I made it less dirty and re-wrote some part.

In process of cleaning my program, I reviewed my code used in Blue Impulse 3D Graphics SDK, which I developed through my master's research project. I had been thinking that if I were able to write a class that generated an off-screen bitmap from a Unicode text, I could use it everywhere.

Here's what I recalled from my old code. To create an off-screen bitmap (called Device Independent Bitmap or DIB) in Windows, I needed to use a function called CreateCompatibleDC, but I needed to pass a valid Device-Context handle to CreateCompatibleDC, and to get a valid Device-Context handle, I had to create a window. As far as I remembered, CreateCompatibleDC used to crash if I gave a NULL pointer as a source Device-Context handle. The problem was, as soon as I needed a window, I either needed to get the main Window handle from the application, or I needed to create a dummy window. If I use the main Window handle from the application, the application becomes Windows dependent. I need to assume that the program has a main window, which is not good in many ways. If I make a dummy window, I can keep the window hidden (SW_HIDE state), and keep the presence of my dummy window secret from the user. But, it didn't seem to me a nice clean solution. But, nowadays necessity of the off-screen bitmap is much higher than before. Win32 API might have been changed so that CreateCompatibleDC can take a NULL pointer. I went to MSDN, and searched for CreateCompatibleDC.

I made a good guess! CreateCompatibleDC can now take a NULL pointer and creates a memory device context. I don't know when Microsoft made this change. At least I remembered the program crashed when I was programming in Windows 3.1. I couldn't give NULL pointer as a source Device-Context handle to CreateCompatibleDC. Maybe I was wrong.  My program might have been crashing for the different purpose, and for some reason it didn't crash when I gave a non-NULL pointer to CreateCompatibleDC.  I don't have Windows 3.1 PC any more and cannot verify whether I was really not able to give NULL pointer to CreateCompatibleDC.

That time Game SDK was called WinG. I was fully expecting that WinG was going to support 3D graphics and was utterly disappointed because it didn't. Nonetheless, support for game programming lead to the dominance of Windows 95 and 98. It was mid 1990s and Unix people and Mac people were thinking gaming was ignoble. They were essentially on denial. They thought that computing was for royal purposes such as scientific calculation. Watching videos, playing music, and 3D graphics were all dirty technologies for them. Mac was a little different from Unix though. In my impression, Apple liked some multi-media stuff, but they were thinking desktop publishing was everything. It was hell difficult to write a game program in Mac OS then. That's how they lost Microsoft. Things are so much different now as you know. Microsoft is slowly losing the dominance, and Apple has the momentum. And, Linux is .... slightly better. At least I can install Ubuntu by myself. Anyways, it's not a main topic of this page.

So, I realized that I could write a program that creates an off-screen bitmap and draws a Unicode characters and returns the array of bitmaps. So, I wrote a simple test program as follows. You can compile from Visual Studio command prompt as:

cl font.cpp kernel32.lib user32.lib gdi32.lib

List: font.cpp

// cl font.cpp kernel32.lib user32.lib gdi32.lib
//   or
// cl font.cpp kernel32.lib user32.lib gdi32.lib /DUNICODE <- This way this code can be checked against default-unicode environment.

#include <windows.h>
#include <stdio.h>
#include <string.h>

static int CALLBACK EnumFontFamExProc(
    const LOGFONTA *fontInfo,
    const TEXTMETRICA *textMetric,
    DWORD fontType,
    LPARAM lParam)
{
    printf("%s\n",fontInfo->lfFaceName);
    return 1;
}

void EnumerateFont(HDC hDC)
{
    LOGFONTA fontInfo;
    fontInfo.lfCharSet=DEFAULT_CHARSET;
    fontInfo.lfFaceName[0]=0;
    fontInfo.lfPitchAndFamily=0;

    EnumFontFamiliesExA(hDC,&fontInfo,EnumFontFamExProc,(LPARAM)0,0);
}

int main(void)
{
    HDC hDC=CreateCompatibleDC(NULL);
    EnumerateFont(hDC);


    LOGFONTW logFont;
    ZeroMemory(&logFont,sizeof(logFont));

    logFont.lfHeight=-12; // 12 pixels (Positive 12 for 12 points)
    logFont.lfWeight=FW_DONTCARE;
    logFont.lfCharSet=DEFAULT_CHARSET;
    logFont.lfOutPrecision=OUT_TT_PRECIS;
    logFont.lfClipPrecision=CLIP_DEFAULT_PRECIS;
    logFont.lfQuality=DEFAULT_QUALITY;
    logFont.lfPitchAndFamily=DEFAULT_PITCH;
    wcscpy(logFont.lfFaceName,L"Arial"); // Up to LF_FACESIZE chars.
    HFONT hFont=CreateFontIndirectW(&logFont);

    HGDIOBJ hPrevFont=NULL;
    if(NULL!=hFont)
    {
        printf("Font Created.\n");
        hPrevFont=SelectObject(hDC,(HGDIOBJ)hFont);
    }


    RECT rect;
    rect.left=0;
    rect.top=0;
    rect.right=100;
    rect.bottom=100;


    SetTextAlign(hDC,TA_LEFT|TA_TOP|TA_NOUPDATECP);  // Prerequisite for DrawTextEx


    wchar_t str[]={0x5c71,0x5ddd,0x03c0,'A',0x0394,0};
    DrawTextExW(hDC,str,4,&rect,DT_TOP|DT_LEFT|DT_CALCRECT,NULL);
    const int wid=rect.right;
    const int hei=rect.bottom;
    const int bitPerPixel=24;

    printf("Width=%d Height=%d\n",wid,hei);

    BITMAPINFOHEADER hdr=
    {
        sizeof(BITMAPINFOHEADER),
        0,0,1,8,BI_RGB,0,0,0,0,0
    };
    hdr.biWidth=wid;
    hdr.biHeight=-hei;
    hdr.biBitCount=bitPerPixel;
    hdr.biCompression=BI_RGB;



    void *bit;
    HBITMAP hBmp=CreateDIBSection(hDC,(BITMAPINFO *)&hdr,DIB_RGB_COLORS,&bit,NULL,0);
    HGDIOBJ hPrevBmp;
    hPrevBmp=SelectObject(hDC,hBmp);

    int bytePerLine=((wid+3)/4)*4*bitPerPixel/8;
    unsigned char *dat=(unsigned char *)bit;



    int i;
    for(i=0; i<bytePerLine*hei; i++)
    {
        dat[i]=0;
    }

    SetTextColor(hDC,RGB(255,255,255));
    SetBkColor(hDC,RGB(0,0,0));
    SetBkMode(hDC,OPAQUE);

    SetDCPenColor(hDC,RGB(255,0,0));


    rect.left=0;
    rect.top=0;
    rect.right=wid;
    rect.bottom=hei;
    DrawTextExW(hDC,str,4,&rect,DT_TOP|DT_LEFT,NULL);

    int x,y;
    unsigned char *lineTop=dat;
    for(y=0; y<hei; y++)
    {
        for(x=0; x<wid; x++)
        {
            if(0!=lineTop[x*3])
            {
                printf("1");
            }
            else
            {
                printf("0");
            }
        }
        lineTop+=bytePerLine;
        printf("\n");
    }


    if(NULL!=hPrevFont)
    {
        printf("Reset font.\n");
        SelectObject(hDC,hPrevFont);
    }
    if(NULL!=hPrevBmp)
    {
        printf("Reset bmp.\n");
        SelectObject(hDC,hPrevBmp);
    }

    DeleteObject(hBmp);
    DeleteDC(hDC);
    return 0;
}

Then I needed to do the same on Mac OSX and Linux to support majority of the popular operating systems. It was easy on Mac OSX. It took a while to see the resolution (DPI) of NSBitmapImageRep, but I later realized that it was by default 1 pixel == 1 point. I think it is a common struggle for programmers that we need to specify the font size by points not pixels no matter where you are writing your font. It makes sense to specify font size by pixels when I am writing font on the monitor. But, since default NSBitmapImageRep's resolution is 72 DPI, I didn't have to worry about point to pixel unit conversion. Another issue was the usage of NSAttributedString. For this, sample codes in "Cocoa & Objective-C: Up and Running" (Scott Stevenson, O'Reilly) around page 360 helped me a lot. You can compile the following sample code by:

g++ font.m –framework Cocoa

List: font.m

#import <Cocoa/Cocoa.h>

int main(void)
{
    NSAutoreleasePool *pool=[[NSAutoreleasePool alloc] init];

    NSFontManager *fontManager=[NSFontManager sharedFontManager];
    if(NULL==fontManager)
    {
        printf("No font manager is present.\n");
    }
    else
    {
        NSArray *availableFonts=[fontManager availableFonts];
        int i;
        for(i=0; i<[availableFonts count]; i++)
        {
            NSString *str=[availableFonts objectAtIndex:i];
            printf("[%d] %s\n",i,[str UTF8String]);
        }
    }

    float fontSize=20.0f;

    NSFont *font=[NSFont fontWithName:@"Times-Roman" size:fontSize];

    NSMutableParagraphStyle *parag=[[NSMutableParagraphStyle alloc] init];
    [parag setMaximumLineHeight:fontSize];

    printf("line spacing %f\n",[[NSParagraphStyle defaultParagraphStyle] lineSpacing]);
    printf("paragraph spacing %f\n",[[NSParagraphStyle defaultParagraphStyle] paragraphSpacing]);
    printf("paragraph spacing before %f\n",[[NSParagraphStyle defaultParagraphStyle] paragraphSpacingBefore]);
    printf("minimum line height %f\n",[[NSParagraphStyle defaultParagraphStyle] minimumLineHeight]);
    printf("maximum line height %f\n",[[NSParagraphStyle defaultParagraphStyle] maximumLineHeight]);
    printf("line height multiplet %f\n",[[NSParagraphStyle defaultParagraphStyle] lineHeightMultiple]);

    NSRect rect;

    NSMutableDictionary *attributes=[NSMutableDictionary dictionary];
    [attributes setObject:font forKey:NSFontAttributeName];
    [attributes setObject:[NSNumber numberWithFloat:0.0] forKey:NSKernAttributeName];
    [attributes setObject:parag forKey:NSParagraphStyleAttributeName];
    [attributes setObject:[NSColor whiteColor] forKey:NSForegroundColorAttributeName];
    [attributes setObject:[NSColor blackColor] forKey:NSBackgroundColorAttributeName];

    unsigned char utf16[]={0x5c,0x71,0x5d,0xdd,0x03,0xc0,0x00,'A',0x03,0x94,0,0};
    NSString *rawString=[[NSString alloc] initWithBytes:utf16 length:10 encoding:NSUTF16BigEndianStringEncoding];

    NSAttributedString *string=[[NSAttributedString alloc] initWithString:rawString attributes:attributes];

    NSSize stringSize=[string size];
    printf("Size %f %f\n",stringSize.width,stringSize.height);




    NSBitmapImageRep *bmpImg=[[NSBitmapImageRep alloc] 
         initWithBitmapDataPlanes:NULL
         pixelsWide:(int)stringSize.width
         pixelsHigh:(int)stringSize.height
         bitsPerSample:8 // 8 bit per component
         samplesPerPixel:3 // 3 components
         hasAlpha:NO
         isPlanar:NO
         colorSpaceName:NSDeviceRGBColorSpace
         bytesPerRow:0
         bitsPerPixel:32];
    // The following commented-out block is optional since the resolution of 
    // a bitmap seems tobe 72 DPI by default, which makes 1 pixel per 1 point.
    /* int pixelsWide=[bmpImg pixelsWide],pixelsHigh=[bmpImg pixelsHigh];
    printf("BMP Pix =%d %d\n",pixelsWide,pixelsHigh);
    NSSize sizeInPoint={pixelsWide,pixelsHigh};
    [bmpImg setSize:sizeInPoint]; // This makes 1 point per pixel
    sizeInPoint=[bmpImg size];
    printf("BMP Size=%f %f\n",sizeInPoint.width,sizeInPoint.height); */



    NSGraphicsContext *gc=[NSGraphicsContext 
              graphicsContextWithBitmapImageRep:bmpImg];

    NSGraphicsContext *prevGc=[NSGraphicsContext currentContext];

    [NSGraphicsContext setCurrentContext:gc];


    [gc setShouldAntialias:NO];

    [[NSColor darkGrayColor] set];
    rect.origin.x=0;
    rect.origin.y=0;
    rect.size.width=stringSize.width;
    rect.size.height=stringSize.height;
    NSRectFill(rect);

    [[NSColor whiteColor] set];
    rect.origin.x=0;
    rect.origin.y=0;
    rect.size.width=stringSize.width;
    rect.size.height=stringSize.height;
    [string drawWithRect:rect options:NSStringDrawingUsesLineFragmentOrigin];

    unsigned char *dat=[bmpImg bitmapData];
    unsigned int bytesPerRow=[bmpImg bytesPerRow];
    printf("bytesPerRow %d\n",bytesPerRow);
    int y;
    for(y=0; y<(int)stringSize.height; y++)
    {
        int x;
        for(x=0; x<(int)stringSize.width; x++)
        {
            if(0==dat[y*bytesPerRow+x*4])
            {
                printf("0");
            }
            else
            {
                printf("1");
            }
        }
        printf("\n");
    }

    [NSGraphicsContext setCurrentContext:prevGc];

    [parag release];
    [string release];
    [rawString release];
    [bmpImg release];

    [pool release];

    return 0;
}

It appeared that I was getting closer to terminate my brute-force (PNG based) localization soon. But, the problem was Unicode. I thought there were no standardized Unicode text-file format.

I remembered that my favorite text editor gave me a choice of text encoding like EUC, Shift-JIS (both are for Japanese characters), and Unicode. I got curious what kind of binary I would get if I choose Unicode as text encoding. I opened up the text editor, and chose "Save As", and opened the "Encoding" drop list. Then I saw "Unicode (UTF-8)". What? UTF-8 is supposed to be 8-bit encoding. How can it store multi-byte characters? .... Suddenly the strange code from [NSString UTF8String] flashed back, and I understood what they were. That was not Apple's original extension. UTF8 was a standard encoding for a Unicode text. I felt I was too far away from the localization business too long.  Then I searched on Wikipedia and found how I could convert wchar to/from UTF8. (http://en.wikipedia.org/wiki/UTF-8). It's easy. I could write two-way converter in a few minutes.

List: Sample UTF8 <-> WChar conversion functions.

#include <stdio.h>


// Reference: http://en.wikipedia.org/wiki/UTF-8
template <class WCHAR_TYPE>
size_t UnicodeWChar2UTF8(size_t lOut,char out[],WCHAR_TYPE in[])
{
    unsigned int iPtr=0,oPtr=0;
    unsigned char *uOut=(unsigned char *)out;

    while(oPtr<lOut-1 && 0!=in[iPtr])
    {
        if(in[iPtr]<=0x7f)
        {
            uOut[oPtr]=(unsigned char)in[iPtr];
            oPtr++;
        }
        else if(in[iPtr]<=0x7ff)
        {
            if(lOut-2<=oPtr)
            {
                // Buffer overrun.  Cannot put zero at the end.
                break;
            }
            unsigned int low6=in[iPtr]&0x3f;
            unsigned int high5=(in[iPtr]>>6)&0x1f;

            uOut[oPtr++]=(unsigned char)(0xc0|high5);
            uOut[oPtr++]=(unsigned char)(0x80|low6);
        }
        else if(in[iPtr]<=0xffff)
        {
            if(lOut-3<=oPtr)
            {
                // Buffer overrun.  Cannot put zero at the end.
                break;
            }

            unsigned int low6=in[iPtr]&0x3f;
            unsigned int mid6=(in[iPtr]>>6)&0x3f;
            unsigned int high4=(in[iPtr]>>12)&0x0f;

            uOut[oPtr++]=(unsigned char)(0xe0|high4);
            uOut[oPtr++]=(unsigned char)(0x80|mid6);
            uOut[oPtr++]=(unsigned char)(0x80|low6);
        }
        else // The following block will never be used unless wchar_t is greater than 16-bit.
        {
            unsigned long inChar=in[iPtr];
            unsigned int bit00to05=inChar&0x3f;
            unsigned int bit06to11=(inChar>>6)&0x3f;
            unsigned int bit12to17=(inChar>>12)&0x3f;
            unsigned int bit18to23=(inChar>>18)&0x3f;
            unsigned int bit24to29=(inChar>>24)&0x3f;
            unsigned int bit30=(inChar>>30)&0x01;

            if(in[iPtr]<=0x1fffff)
            {
                if(lOut-4<=oPtr)
                {
                    // Buffer overrun.  Cannot put zero at the end.
                    break;
                }
                uOut[oPtr++]=(unsigned char)(0xf0|(bit18to23&0x07));
                uOut[oPtr++]=(unsigned char)(0x80|bit12to17);
                uOut[oPtr++]=(unsigned char)(0x80|bit06to11);
                uOut[oPtr++]=(unsigned char)(0x80|bit00to05);
            }
            else if(in[iPtr]<=0x3ffffff)
            {
                if(lOut-5<=oPtr)
                {
                    // Buffer overrun.  Cannot put zero at the end.
                    break;
                }
                uOut[oPtr++]=(unsigned char)(0xf8|(bit24to29&0x03));
                uOut[oPtr++]=(unsigned char)(0x80|bit18to23);
                uOut[oPtr++]=(unsigned char)(0x80|bit12to17);
                uOut[oPtr++]=(unsigned char)(0x80|bit06to11);
                uOut[oPtr++]=(unsigned char)(0x80|bit00to05);
            }
            else if(in[iPtr]<=0x7fffffff)
            {
                if(lOut-6<=oPtr)
                {
                    // Buffer overrun.  Cannot put zero at the end.
                    break;
                }
                uOut[oPtr++]=(unsigned char)(0xfc|bit30);
                uOut[oPtr++]=(unsigned char)(0x80|bit24to29);
                uOut[oPtr++]=(unsigned char)(0x80|bit18to23);
                uOut[oPtr++]=(unsigned char)(0x80|bit12to17);
                uOut[oPtr++]=(unsigned char)(0x80|bit06to11);
                uOut[oPtr++]=(unsigned char)(0x80|bit00to05);
            }
        }
        iPtr++;
    }

    if(oPtr<lOut)
    {
        out[oPtr]=0;
        oPtr++;
    }

    return oPtr;
}

template <class WCHAR_TYPE>
size_t UnicodeUTF8toWChar(size_t lOut,WCHAR_TYPE out[],const char in[])
{
    unsigned int iPtr=0,oPtr=0;
    unsigned char *uIn=(unsigned char *)in;

    while(oPtr<lOut-1 && 0!=uIn[iPtr])
    {
        if(0==(uIn[iPtr]&0x80))
        {
            out[oPtr]=uIn[iPtr];
            iPtr++;
        }
        else if(0xc0==(uIn[iPtr]&0xe0))
        {
            if(0==uIn[iPtr+1])
            {
                // Pre-mature termination.
                break;
            }
            unsigned int high5=uIn[iPtr]&0x1f;
            unsigned int low6=uIn[iPtr+1]&0x3f;
            out[oPtr]=(WCHAR_TYPE)((high5<<6)|low6);
            iPtr+=2;
        }
        else if(0xe0==(uIn[iPtr]&0xf0))
        {
            if(0==uIn[iPtr+1] || 0==uIn[iPtr+2])
            {
                // Pre-mature termination.
                break;
            }
            unsigned int high4=uIn[iPtr]&0x0f;
            unsigned int mid6=uIn[iPtr+1]&0x3f;
            unsigned int low6=uIn[iPtr+2]&0x3f;
            out[oPtr]=(WCHAR_TYPE)((high4<<12)|(mid6<<6)|low6);
            iPtr+=3;
        }
        else if(0xf0==(uIn[iPtr]&0xf8))
        {
            if(0==uIn[iPtr+1] || 0==uIn[iPtr+2] || 0==uIn[iPtr+3])
            {
                // Pre-mature termination.
                break;
            }
            unsigned int bit18to20=uIn[iPtr]&0x07;
            unsigned int bit12to17=uIn[iPtr+1]&0x3f;
            unsigned int bit06to11=uIn[iPtr+2]&0x3f;
            unsigned int bit00to05=uIn[iPtr+3]&0x3f;
            out[oPtr]=(WCHAR_TYPE)((bit18to20<<18)|(bit12to17<<12)|(bit06to11<<6)|bit00to05);
            iPtr+=4;
        }
        else if(0xf8==(uIn[iPtr]&0xfc))
        {
            if(0==uIn[iPtr+1] || 0==uIn[iPtr+2] || 0==uIn[iPtr+3] || 0==uIn[iPtr+4])
            {
                // Pre-mature termination.
                break;
            }
            unsigned int bit24to25=uIn[iPtr]&0x03;
            unsigned int bit18to23=uIn[iPtr+1]&0x3f;
            unsigned int bit12to17=uIn[iPtr+2]&0x3f;
            unsigned int bit06to11=uIn[iPtr+3]&0x3f;
            unsigned int bit00to05=uIn[iPtr+4]&0x3f;
            out[oPtr]=(WCHAR_TYPE)((bit24to25<<24)|(bit18to23<<18)|(bit12to17<<12)|(bit06to11<<6)|bit00to05);
            iPtr+=5;
        }
        else if(0xfc==(uIn[iPtr]&0xfe))
        {
            if(0==uIn[iPtr+1] || 0==uIn[iPtr+2] || 0==uIn[iPtr+3] || 0==uIn[iPtr+4])
            {
                // Pre-mature termination.
                break;
            }
            unsigned int bit30=uIn[iPtr]&0x01;
            unsigned int bit24to29=uIn[iPtr+1]&0x3f;
            unsigned int bit18to23=uIn[iPtr+2]&0x3f;
            unsigned int bit12to17=uIn[iPtr+3]&0x3f;
            unsigned int bit06to11=uIn[iPtr+4]&0x3f;
            unsigned int bit00to05=uIn[iPtr+5]&0x3f;
            out[oPtr]=(WCHAR_TYPE)((bit30<<30)|(bit24to29<<24)|(bit18to23<<18)|(bit12to17<<12)|(bit06to11<<6)|bit00to05);
            iPtr+=6;
        }
        else
        {
            // Invalid utf8 string
            break;
        }
        oPtr++;
    }

    if(oPtr<lOut)
    {
        out[oPtr]=0;
        oPtr++;
    }

    return oPtr;
}


// Should be encoded to  E5 B1 B1  E5  B7 9D  CF 80
int main(void)
{
    wchar_t in[]={0x5c71,0x5ddd,0x03c0,0};
    char utf8[256];

    int i;
    for(i=0; 0!=in[i]; i++)
    {
        printf("%04x ",in[i]);
    }
    printf("\n");

    size_t bytes=UnicodeWChar2UTF8 <wchar_t> (256,utf8,in);
    printf("Encoded to %d bytes (including the last zero terminator).\n",bytes);

    for(i=0; 0!=utf8[i]; i++)
    {
        printf("%02x ",((unsigned char *)utf8)[i]);
    }
    printf("\n");


    wchar_t backConv[256];
    size_t wchars=UnicodeUTF8toWChar <wchar_t> (256,backConv,utf8);
    printf("Decoded to %d wchars (including the last zero terminator).\n",wchars);

    for(i=0; 0!=backConv[i]; i++)
    {
        printf("%04x ",backConv[i]);
    }
    printf("\n");

    return 0;
}

Unicode problem solved.

I knew how to make an off-screen bitmap from a Unicode string in Windows and Mac OSX. iOS is almost same as Mac OSX. So, if I found how to do the same in Linux, I could support most of the major operating systems.

But, doing it in X-Window was the most difficult among three systems. X-Window is obsolete. I want to avoid if I can. But, it comes with Linux by default. If I want to support Linux, I cannot avoid it.

What surprised me was organization of the off-screen bitmap. If I made a 24-bit bitmap, and if I pick three consecutive bytes, I expect that three bytes will give red, green, and blue component. That's the case in Windows and Mac OSX. But, when I made a bitmap (called pixmap in X) and retrieved a bitmap, first of the 24 bits were packed in a sequence, followed by the second, third, ..., 24th bits.

I first wrote a program that draws "Text" in black color on white background. The first byte of the bitmap was $E0. I wondered why the hell I was getting a byte other than $00 and $FF? I couldn't solve the mystery for an hour or so. Then I gave up and took a shower. When I was taking a shower, I noticed the first 5 bits of $E0 were zero. Maybe the top bar of letter T was expanded to five pixels, and that's why I was getting $E0. So, I scanned bit by bit and printed 1 and 0 on the console, and I saw the pattern that I expected. It reminded me of the structure of the Video-RAM of my very first 8-bit computer. It's ok. In X-Window, you can experience how people used to do things decades ago. You can learn history. That's the joy of X-Window. At least I figured how to do it in X-Window.  You can compile the following program by: g++ font.cpp -lX11

List: font.cpp

#include <stdio.h>
#include <string.h>
#include <X11/Xlib.h>
#include <X11/Xutil.h>
#include <X11/Xlocale.h>

/* Memo for future reference
Meaning of font string.
  http://en.wikipedia.org/wiki/X_logical_font_description

X-Window drawing functions
  http://www-yano.is.tokushima-u.ac.jp/~mituhara/software_e/draw.html

XwcDrawString sample code
  http://homepage3.nifty.com/rio_i/lab/xlib/011mbcs.htm
    This page suggests:
    - Environment variable LANG should be set.
    - A function call setlocale(LC_CTYPE,""); is required.

May also be possible to use XLoadFont, XSetFont
  Font font=XLoadFont(ysXDsp,"6x10");
*/

int main(void)
{
    Display *dsp=XOpenDisplay(NULL);
    if(NULL!=dsp)
    {
        printf("Display Opened. (Depth=%d)\n",DefaultDepth(dsp,0));

        Window rootWin=DefaultRootWindow(dsp);

        int nFont;
        char **fontList=XListFonts(dsp,"*iso10646*",1024,&nFont);
        int i;
        for(i=0; i<nFont; i++)
        {
            printf("[%d] %s\n",i,fontList[i]);
        }

        const char *baseFontName=
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-1,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-2,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-4,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-5,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-6,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-7,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-8,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-9,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso8859-15,"
            "-*-*-*-*-*-*-16-*-*-*-*-*-iso10646-1";

        const char *localeStr=setlocale(LC_CTYPE,"");
        if(NULL==localeStr || True!=XSupportsLocale())
        {
            setlocale(LC_CTYPE,"en_US.UTF-8");  // Desperate attempt!
        }

        int missCharSetCount;
        char *defStr,**missCharSet;
        XFontSet fontSet=XCreateFontSet(
        dsp,
        baseFontName,
        &missCharSet,
        &missCharSetCount,
        &defStr);
        if(NULL!=fontSet)
        {
            printf("Acquired font set (Missing=%d)\n",missCharSetCount);

            wchar_t str[]={0x5c71,0x5ddd,0x03c0,'A',0x0394,0};
            XRectangle inkRect,logRect;
            int ext=XwcTextExtents(fontSet,str,5,&inkRect,&logRect);

            printf("Extent %d\n",ext);
            printf("Ink Rect %d %d %d %d\n",inkRect.x,inkRect.y,inkRect.width,inkRect.height);
            printf("Log Rect %d %d %d %d\n",logRect.x,logRect.y,logRect.width,logRect.height);


            int bmpWid=ext,bmpHei=logRect.height;
            int depth=DefaultDepth(dsp,0);

            Pixmap pixmap=XCreatePixmap(dsp,rootWin,bmpWid,bmpHei,depth);
            GC gc=XCreateGC(dsp,pixmap,0,0);


            XSetForeground(dsp,gc,BlackPixel(dsp,0));
            XFillRectangle(dsp,pixmap,gc,0,0,bmpWid,bmpHei);

            XSetForeground(dsp,gc,WhitePixel(dsp,0));
            XwcDrawString(dsp,pixmap,fontSet,gc,0,bmpHei-1,str,5);

            XImage *image=XGetImage(dsp,pixmap,0,0,bmpWid,bmpHei,~0,XYPixmap);
            if(NULL!=image)
            {
                printf("Acquired Image %dx%d\n",image->width,image->height);
                printf("Bitmap Unit=%d\n",image->bitmap_unit);
                printf("Bytes per line=%d\n",image->bytes_per_line);
                printf("Depth=%d\n",image->depth);

                int x,y;
                for(y=0; y<image->height; y++)
                {
                    unsigned char mask=1;
                    unsigned char *byteTop=(unsigned char *)
                    image->data+y*image->bytes_per_line;
                    for(x=0; x<image->width; x++)
                    {
                        if((*byteTop)&mask)
                        {
                            printf("1");
                        }
                        else
                        {
                            printf("0");
                        }
                        if(0!=(mask&0x80))
                        {
                            byteTop++;
                            mask=1;
                        }
                        else
                        {
                            mask<<=1;
                        }
                    }
                    printf("\n");
                }

                /* Attempt 1 Failed. Apparently, each of 24 layers is
                packed contiguously.  I was forgetting X-Window is from the stone age.
                int byte_per_pixel=image->bitmap_unit/8;
                for(y=0; y<image->height; y++)
                {
                    for(x=0; x<20; x++)
                    {
                        int offset=y*(64*byte_per_pixel)+x*byte_per_pixel;
                        unsigned char *ptr=(unsigned char *)image->data+offset;
                        printf("%02x%02x%02x ",ptr[0],ptr[1],ptr[2]);
                    }
                    printf("\n");
                } */
            }

            XDestroyImage(image);
            XFreeGC(dsp,gc);
            XFreePixmap(dsp,pixmap);
            XFreeFontSet(dsp,fontSet);
        }
        XCloseDisplay(dsp);
    }
}

Now I know how to do it in Windows, Mac OSX, and Linux. Next thing is to make a class that has a set of interfaces that allows the application to access these features. Then I will be able to get rid of brute-force PNG-based localization.

04/26/2011