1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
|
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"><head><title>Facets</title><meta name="generator" content="DocBook XSL-NS Stylesheets V1.76.1"/><meta name="keywords" content=" ISO C++ , library "/><link rel="home" href="../spine.html" title="The GNU C++ Library"/><link rel="up" href="localization.html" title="Chapter 8. Localization"/><link rel="prev" href="localization.html" title="Chapter 8. Localization"/><link rel="next" href="containers.html" title="Chapter 9. Containers"/></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Facets</th></tr><tr><td align="left"><a accesskey="p" href="localization.html">Prev</a> </td><th width="60%" align="center">Chapter 8.
Localization
</th><td align="right"> <a accesskey="n" href="containers.html">Next</a></td></tr></table><hr/></div><div class="section" title="Facets"><div class="titlepage"><div><div><h2 class="title"><a id="std.localization.facet"/>Facets</h2></div></div></div><div class="section" title="ctype"><div class="titlepage"><div><div><h3 class="title"><a id="std.localization.facet.ctype"/>ctype</h3></div></div></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.impl"/>Implementation</h4></div></div></div><div class="section" title="Specializations"><div class="titlepage"><div><div><h5 class="title"><a id="id476560"/>Specializations</h5></div></div></div><p>
For the required specialization codecvt<wchar_t, char, mbstate_t> ,
conversions are made between the internal character set (always UCS4
on GNU/Linux) and whatever the currently selected locale for the
LC_CTYPE category implements.
</p><p>
The two required specializations are implemented as follows:
</p><p>
<code class="code">
ctype<char>
</code>
</p><p>
This is simple specialization. Implementing this was a piece of cake.
</p><p>
<code class="code">
ctype<wchar_t>
</code>
</p><p>
This specialization, by specifying all the template parameters, pretty
much ties the hands of implementors. As such, the implementation is
straightforward, involving mcsrtombs for the conversions between char
to wchar_t and wcsrtombs for conversions between wchar_t and char.
</p><p>
Neither of these two required specializations deals with Unicode
characters.
</p></div></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.future"/>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
How to deal with the global locale issue?
</p></li><li class="listitem"><p>
How to deal with different types than char, wchar_t? </p></li><li class="listitem"><p>
Overlap between codecvt/ctype: narrow/widen
</p></li><li class="listitem"><p>
Mask typedef in codecvt_base, argument types in codecvt. what
is know about this type?
</p></li><li class="listitem"><p>
Why mask* argument in codecvt?
</p></li><li class="listitem"><p>
Can this be made (more) generic? is there a simple way to
straighten out the configure-time mess that is a by-product of
this class?
</p></li><li class="listitem"><p>
Get the ctype<wchar_t>::mask stuff under control. Need to
make some kind of static table, and not do lookup every time
somebody hits the do_is... functions. Too bad we can't just
redefine mask for ctype<wchar_t>
</p></li><li class="listitem"><p>
Rename abstract base class. See if just smash-overriding is a
better approach. Clarify, add sanity to naming.
</p></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.biblio"/>Bibliography</h4></div></div></div><div class="biblioentry"><a id="id476684"/><p><span class="citetitle"><em class="citetitle">
The GNU C Library
</em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling and 7 Locales and Internationalization. </span></p></div><div class="biblioentry"><a id="id476724"/><p><span class="citetitle"><em class="citetitle">
Correspondence
</em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="id476750"/><p><span class="citetitle"><em class="citetitle">
ISO/IEC 14882:1998 Programming languages - C++
</em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="id476769"/><p><span class="citetitle"><em class="citetitle">
ISO/IEC 9899:1999 Programming languages - C
</em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry"><a id="id476788"/><p><span class="biblioid">
. </span><span class="citetitle"><em class="citetitle">
The Open Group Base Specifications, Issue 6 (IEEE Std. 1003.1-2004)
</em>. </span><span class="copyright">Copyright © 1999
The Open Group/The Institute of Electrical and Electronics Engineers, Inc.. </span></p></div><div class="biblioentry"><a id="id476817"/><p><span class="citetitle"><em class="citetitle">
The C++ Programming Language, Special Edition
</em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
Addison Wesley
. </span></span></p></div><div class="biblioentry"><a id="id476856"/><p><span class="citetitle"><em class="citetitle">
Standard C++ IOStreams and Locales
</em>. </span><span class="subtitle">
Advanced Programmer's Guide and Reference
. </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
Addison Wesley Longman
. </span></span></p></div></div></div><div class="section" title="codecvt"><div class="titlepage"><div><div><h3 class="title"><a id="std.localization.facet.codecvt"/>codecvt</h3></div></div></div><p>
The standard class codecvt attempts to address conversions between
different character encoding schemes. In particular, the standard
attempts to detail conversions between the implementation-defined wide
characters (hereafter referred to as wchar_t) and the standard type
char that is so beloved in classic <span class="quote">“<span class="quote">C</span>”</span> (which can now be
referred to as narrow characters.) This document attempts to describe
how the GNU libstdc++ implementation deals with the conversion between
wide and narrow characters, and also presents a framework for dealing
with the huge number of other encodings that iconv can convert,
including Unicode and UTF8. Design issues and requirements are
addressed, and examples of correct usage for both the required
specializations for wide and narrow characters and the
implementation-provided extended functionality are given.
</p><div class="section" title="Requirements"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.req"/>Requirements</h4></div></div></div><p>
Around page 425 of the C++ Standard, this charming heading comes into view:
</p><div class="blockquote"><blockquote class="blockquote"><p>
22.2.1.5 - Template class codecvt
</p></blockquote></div><p>
The text around the codecvt definition gives some clues:
</p><div class="blockquote"><blockquote class="blockquote"><p>
<span class="emphasis"><em>
-1- The class codecvt<internT,externT,stateT> is for use when
converting from one codeset to another, such as from wide characters
to multibyte characters, between wide character encodings such as
Unicode and EUC.
</em></span>
</p></blockquote></div><p>
Hmm. So, in some unspecified way, Unicode encodings and
translations between other character sets should be handled by this
class.
</p><div class="blockquote"><blockquote class="blockquote"><p>
<span class="emphasis"><em>
-2- The stateT argument selects the pair of codesets being mapped between.
</em></span>
</p></blockquote></div><p>
Ah ha! Another clue...
</p><div class="blockquote"><blockquote class="blockquote"><p>
<span class="emphasis"><em>
-3- The instantiations required in the Table ??
(lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and
codecvt<char,char,mbstate_t>, convert the implementation-defined
native character set. codecvt<char,char,mbstate_t> implements a
degenerate conversion; it does not convert at
all. codecvt<wchar_t,char,mbstate_t> converts between the native
character sets for tiny and wide characters. Instantiations on
mbstate_t perform conversion between encodings known to the library
implementor. Other encodings can be converted by specializing on a
user-defined stateT type. The stateT object can contain any state that
is useful to communicate to or from the specialized do_convert member.
</em></span>
</p></blockquote></div><p>
At this point, a couple points become clear:
</p><p>
One: The standard clearly implies that attempts to add non-required
(yet useful and widely used) conversions need to do so through the
third template parameter, stateT.</p><p>
Two: The required conversions, by specifying mbstate_t as the third
template parameter, imply an implementation strategy that is mostly
(or wholly) based on the underlying C library, and the functions
mcsrtombs and wcsrtombs in particular.</p></div><div class="section" title="Design"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.design"/>Design</h4></div></div></div><div class="section" title="wchar_t Size"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.wchar_t_size"/><span class="type">wchar_t</span> Size</h5></div></div></div><p>
The simple implementation detail of wchar_t's size seems to
repeatedly confound people. Many systems use a two byte,
unsigned integral type to represent wide characters, and use an
internal encoding of Unicode or UCS2. (See AIX, Microsoft NT,
Java, others.) Other systems, use a four byte, unsigned integral
type to represent wide characters, and use an internal encoding
of UCS4. (GNU/Linux systems using glibc, in particular.) The C
programming language (and thus C++) does not specify a specific
size for the type wchar_t.
</p><p>
Thus, portable C++ code cannot assume a byte size (or endianness) either.
</p></div><div class="section" title="Support for Unicode"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.unicode"/>Support for Unicode</h5></div></div></div><p>
Probably the most frequently asked question about code conversion
is: "So dudes, what's the deal with Unicode strings?"
The dude part is optional, but apparently the usefulness of
Unicode strings is pretty widely appreciated. Sadly, this specific
encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10,
etc etc etc) are not mentioned in the C++ standard.
</p><p>
A couple of comments:
</p><p>
The thought that all one needs to convert between two arbitrary
codesets is two types and some kind of state argument is
unfortunate. In particular, encodings may be stateless. The naming
of the third parameter as stateT is unfortunate, as what is really
needed is some kind of generalized type that accounts for the
issues that abstract encodings will need. The minimum information
that is required includes:
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
Identifiers for each of the codesets involved in the
conversion. For example, using the iconv family of functions
from the Single Unix Specification (what used to be called
X/Open) hosted on the GNU/Linux operating system allows
bi-directional mapping between far more than the following
tantalizing possibilities:
</p><p>
(An edited list taken from <code class="code">`iconv --list`</code> on a
Red Hat 6.2/Intel system:
</p><div class="blockquote"><blockquote class="blockquote"><pre class="programlisting">
8859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7,
ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD,
GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3,
ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8,
ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14,
ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4,
ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4,
UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8,
UTF-16, UTF8, UTF16).
</pre></blockquote></div><p>
For iconv-based implementations, string literals for each of the
encodings (i.e. "UCS-2" and "UTF-8") are necessary,
although for other,
non-iconv implementations a table of enumerated values or some other
mechanism may be required.
</p></li><li class="listitem"><p>
Maximum length of the identifying string literal.
</p></li><li class="listitem"><p>
Some encodings require explicit endian-ness. As such, some kind
of endian marker or other byte-order marker will be necessary. See
"Footnotes for C/C++ developers" in Haible for more information on
UCS-2/Unicode endian issues. (Summary: big endian seems most likely,
however implementations, most notably Microsoft, vary.)
</p></li><li class="listitem"><p>
Types representing the conversion state, for conversions involving
the machinery in the "C" library, or the conversion descriptor, for
conversions using iconv (such as the type iconv_t.) Note that the
conversion descriptor encodes more information than a simple encoding
state type.
</p></li><li class="listitem"><p>
Conversion descriptors for both directions of encoding. (i.e., both
UCS-2 to UTF-8 and UTF-8 to UCS-2.)
</p></li><li class="listitem"><p>
Something to indicate if the conversion requested if valid.
</p></li><li class="listitem"><p>
Something to represent if the conversion descriptors are valid.
</p></li><li class="listitem"><p>
Some way to enforce strict type checking on the internal and
external types. As part of this, the size of the internal and
external types will need to be known.
</p></li></ul></div></div><div class="section" title="Other Issues"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.issues"/>Other Issues</h5></div></div></div><p>
In addition, multi-threaded and multi-locale environments also impact
the design and requirements for code conversions. In particular, they
affect the required specialization codecvt<wchar_t, char, mbstate_t>
when implemented using standard "C" functions.
</p><p>
Three problems arise, one big, one of medium importance, and one small.
</p><p>
First, the small: mcsrtombs and wcsrtombs may not be multithread-safe
on all systems required by the GNU tools. For GNU/Linux and glibc,
this is not an issue.
</p><p>
Of medium concern, in the grand scope of things, is that the functions
used to implement this specialization work on null-terminated
strings. Buffers, especially file buffers, may not be null-terminated,
thus giving conversions that end prematurely or are otherwise
incorrect. Yikes!
</p><p>
The last, and fundamental problem, is the assumption of a global
locale for all the "C" functions referenced above. For something like
C++ iostreams (where codecvt is explicitly used) the notion of
multiple locales is fundamental. In practice, most users may not run
into this limitation. However, as a quality of implementation issue,
the GNU C++ library would like to offer a solution that allows
multiple locales and or simultaneous usage with computationally
correct results. In short, libstdc++ is trying to offer, as an
option, a high-quality implementation, damn the additional complexity!
</p><p>
For the required specialization codecvt<wchar_t, char, mbstate_t> ,
conversions are made between the internal character set (always UCS4
on GNU/Linux) and whatever the currently selected locale for the
LC_CTYPE category implements.
</p></div></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.impl"/>Implementation</h4></div></div></div><p>
The two required specializations are implemented as follows:
</p><p>
<code class="code">
codecvt<char, char, mbstate_t>
</code>
</p><p>
This is a degenerate (i.e., does nothing) specialization. Implementing
this was a piece of cake.
</p><p>
<code class="code">
codecvt<char, wchar_t, mbstate_t>
</code>
</p><p>
This specialization, by specifying all the template parameters, pretty
much ties the hands of implementors. As such, the implementation is
straightforward, involving mcsrtombs for the conversions between char
to wchar_t and wcsrtombs for conversions between wchar_t and char.
</p><p>
Neither of these two required specializations deals with Unicode
characters. As such, libstdc++ implements a partial specialization
of the codecvt class with and iconv wrapper class, encoding_state as the
third template parameter.
</p><p>
This implementation should be standards conformant. First of all, the
standard explicitly points out that instantiations on the third
template parameter, stateT, are the proper way to implement
non-required conversions. Second of all, the standard says (in Chapter
17) that partial specializations of required classes are a-ok. Third
of all, the requirements for the stateT type elsewhere in the standard
(see 21.1.2 traits typedefs) only indicate that this type be copy
constructible.
</p><p>
As such, the type encoding_state is defined as a non-templatized, POD
type to be used as the third type of a codecvt instantiation. This
type is just a wrapper class for iconv, and provides an easy interface
to iconv functionality.
</p><p>
There are two constructors for encoding_state:
</p><p>
<code class="code">
encoding_state() : __in_desc(0), __out_desc(0)
</code>
</p><p>
This default constructor sets the internal encoding to some default
(currently UCS4) and the external encoding to whatever is returned by
nl_langinfo(CODESET).
</p><p>
<code class="code">
encoding_state(const char* __int, const char* __ext)
</code>
</p><p>
This constructor takes as parameters string literals that indicate the
desired internal and external encoding. There are no defaults for
either argument.
</p><p>
One of the issues with iconv is that the string literals identifying
conversions are not standardized. Because of this, the thought of
mandating and or enforcing some set of pre-determined valid
identifiers seems iffy: thus, a more practical (and non-migraine
inducing) strategy was implemented: end-users can specify any string
(subject to a pre-determined length qualifier, currently 32 bytes) for
encodings. It is up to the user to make sure that these strings are
valid on the target system.
</p><p>
<code class="code">
void
_M_init()
</code>
</p><p>
Strangely enough, this member function attempts to open conversion
descriptors for a given encoding_state object. If the conversion
descriptors are not valid, the conversion descriptors returned will
not be valid and the resulting calls to the codecvt conversion
functions will return error.
</p><p>
<code class="code">
bool
_M_good()
</code>
</p><p>
Provides a way to see if the given encoding_state object has been
properly initialized. If the string literals describing the desired
internal and external encoding are not valid, initialization will
fail, and this will return false. If the internal and external
encodings are valid, but iconv_open could not allocate conversion
descriptors, this will also return false. Otherwise, the object is
ready to convert and will return true.
</p><p>
<code class="code">
encoding_state(const encoding_state&)
</code>
</p><p>
As iconv allocates memory and sets up conversion descriptors, the copy
constructor can only copy the member data pertaining to the internal
and external code conversions, and not the conversion descriptors
themselves.
</p><p>
Definitions for all the required codecvt member functions are provided
for this specialization, and usage of codecvt<internal character type,
external character type, encoding_state> is consistent with other
codecvt usage.
</p></div><div class="section" title="Use"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.use"/>Use</h4></div></div></div><p>A conversions involving string literal.</p><pre class="programlisting">
typedef codecvt_base::result result;
typedef unsigned short unicode_t;
typedef unicode_t int_type;
typedef char ext_type;
typedef encoding_state state_type;
typedef codecvt<int_type, ext_type, state_type> unicode_codecvt;
const ext_type* e_lit = "black pearl jasmine tea";
int size = strlen(e_lit);
int_type i_lit_base[24] =
{ 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184,
27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696,
25856, 24832, 2560
};
const int_type* i_lit = i_lit_base;
const ext_type* efrom_next;
const int_type* ifrom_next;
ext_type* e_arr = new ext_type[size + 1];
ext_type* eto_next;
int_type* i_arr = new int_type[size + 1];
int_type* ito_next;
// construct a locale object with the specialized facet.
locale loc(locale::classic(), new unicode_codecvt);
// sanity check the constructed locale has the specialized facet.
VERIFY( has_facet<unicode_codecvt>(loc) );
const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc);
// convert between const char* and unicode strings
unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1");
initialize_state(state01);
result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next,
i_arr, i_arr + size, ito_next);
VERIFY( r1 == codecvt_base::ok );
VERIFY( !int_traits::compare(i_arr, i_lit, size) );
VERIFY( efrom_next == e_lit + size );
VERIFY( ito_next == i_arr + size );
</pre></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.future"/>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
a. things that are sketchy, or remain unimplemented:
do_encoding, max_length and length member functions
are only weakly implemented. I have no idea how to do
this correctly, and in a generic manner. Nathan?
</p></li><li class="listitem"><p>
b. conversions involving std::string
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
how should operators != and == work for string of
different/same encoding?
</p></li><li class="listitem"><p>
what is equal? A byte by byte comparison or an
encoding then byte comparison?
</p></li><li class="listitem"><p>
conversions between narrow, wide, and unicode strings
</p></li></ul></div></li><li class="listitem"><p>
c. conversions involving std::filebuf and std::ostream
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
how to initialize the state object in a
standards-conformant manner?
</p></li><li class="listitem"><p>
how to synchronize the "C" and "C++"
conversion information?
</p></li><li class="listitem"><p>
wchar_t/char internal buffers and conversions between
internal/external buffers?
</p></li></ul></div></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.biblio"/>Bibliography</h4></div></div></div><div class="biblioentry"><a id="id477506"/><p><span class="citetitle"><em class="citetitle">
The GNU C Library
</em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">
Chapters 6 Character Set Handling and 7 Locales and Internationalization
. </span></p></div><div class="biblioentry"><a id="id477546"/><p><span class="citetitle"><em class="citetitle">
Correspondence
</em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="id477571"/><p><span class="citetitle"><em class="citetitle">
ISO/IEC 14882:1998 Programming languages - C++
</em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="id477590"/><p><span class="citetitle"><em class="citetitle">
ISO/IEC 9899:1999 Programming languages - C
</em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry"><a id="id477609"/><p><span class="biblioid">
. </span><span class="citetitle"><em class="citetitle">
System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
</em>. </span><span class="copyright">Copyright © 2008
The Open Group/The Institute of Electrical and Electronics
Engineers, Inc.
. </span></p></div><div class="biblioentry"><a id="id477639"/><p><span class="citetitle"><em class="citetitle">
The C++ Programming Language, Special Edition
</em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
Addison Wesley
. </span></span></p></div><div class="biblioentry"><a id="id477677"/><p><span class="citetitle"><em class="citetitle">
Standard C++ IOStreams and Locales
</em>. </span><span class="subtitle">
Advanced Programmer's Guide and Reference
. </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
Addison Wesley Longman
. </span></span></p></div><div class="biblioentry"><a id="id477724"/><p><span class="biblioid">
. </span><span class="citetitle"><em class="citetitle">
A brief description of Normative Addendum 1
</em>. </span><span class="author"><span class="firstname">Clive</span> <span class="surname">Feather</span>. </span><span class="pagenums">Extended Character Sets. </span></p></div><div class="biblioentry"><a id="id477754"/><p><span class="biblioid">
. </span><span class="citetitle"><em class="citetitle">
The Unicode HOWTO
</em>. </span><span class="author"><span class="firstname">Bruno</span> <span class="surname">Haible</span>. </span></p></div><div class="biblioentry"><a id="id477779"/><p><span class="biblioid">
. </span><span class="citetitle"><em class="citetitle">
UTF-8 and Unicode FAQ for Unix/Linux
</em>. </span><span class="author"><span class="firstname">Markus</span> <span class="surname">Khun</span>. </span></p></div></div></div><div class="section" title="messages"><div class="titlepage"><div><div><h3 class="title"><a id="manual.localization.facet.messages"/>messages</h3></div></div></div><p>
The std::messages facet implements message retrieval functionality
equivalent to Java's java.text.MessageFormat .using either GNU gettext
or IEEE 1003.1-200 functions.
</p><div class="section" title="Requirements"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.req"/>Requirements</h4></div></div></div><p>
The std::messages facet is probably the most vaguely defined facet in
the standard library. It's assumed that this facility was built into
the standard library in order to convert string literals from one
locale to the other. For instance, converting the "C" locale's
<code class="code">const char* c = "please"</code> to a German-localized <code class="code">"bitte"</code>
during program execution.
</p><div class="blockquote"><blockquote class="blockquote"><p>
22.2.7.1 - Template class messages [lib.locale.messages]
</p></blockquote></div><p>
This class has three public member functions, which directly
correspond to three protected virtual member functions.
</p><p>
The public member functions are:
</p><p>
<code class="code">catalog open(const string&, const locale&) const</code>
</p><p>
<code class="code">string_type get(catalog, int, int, const string_type&) const</code>
</p><p>
<code class="code">void close(catalog) const</code>
</p><p>
While the virtual functions are:
</p><p>
<code class="code">catalog do_open(const string&, const locale&) const</code>
</p><div class="blockquote"><blockquote class="blockquote"><p>
<span class="emphasis"><em>
-1- Returns: A value that may be passed to get() to retrieve a
message, from the message catalog identified by the string name
according to an implementation-defined mapping. The result can be used
until it is passed to close(). Returns a value less than 0 if no such
catalog can be opened.
</em></span>
</p></blockquote></div><p>
<code class="code">string_type do_get(catalog, int, int, const string_type&) const</code>
</p><div class="blockquote"><blockquote class="blockquote"><p>
<span class="emphasis"><em>
-3- Requires: A catalog cat obtained from open() and not yet closed.
-4- Returns: A message identified by arguments set, msgid, and dfault,
according to an implementation-defined mapping. If no such message can
be found, returns dfault.
</em></span>
</p></blockquote></div><p>
<code class="code">void do_close(catalog) const</code>
</p><div class="blockquote"><blockquote class="blockquote"><p>
<span class="emphasis"><em>
-5- Requires: A catalog cat obtained from open() and not yet closed.
-6- Effects: Releases unspecified resources associated with cat.
-7- Notes: The limit on such resources, if any, is implementation-defined.
</em></span>
</p></blockquote></div></div><div class="section" title="Design"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.design"/>Design</h4></div></div></div><p>
A couple of notes on the standard.
</p><p>
First, why is <code class="code">messages_base::catalog</code> specified as a typedef
to int? This makes sense for implementations that use
<code class="code">catopen</code>, but not for others. Fortunately, it's not heavily
used and so only a minor irritant.
</p><p>
Second, by making the member functions <code class="code">const</code>, it is
impossible to save state in them. Thus, storing away information used
in the 'open' member function for use in 'get' is impossible. This is
unfortunate.
</p><p>
The 'open' member function in particular seems to be oddly
designed. The signature seems quite peculiar. Why specify a <code class="code">const
string& </code> argument, for instance, instead of just <code class="code">const
char*</code>? Or, why specify a <code class="code">const locale&</code> argument that is
to be used in the 'get' member function? How, exactly, is this locale
argument useful? What was the intent? It might make sense if a locale
argument was associated with a given default message string in the
'open' member function, for instance. Quite murky and unclear, on
reflection.
</p><p>
Lastly, it seems odd that messages, which explicitly require code
conversion, don't use the codecvt facet. Because the messages facet
has only one template parameter, it is assumed that ctype, and not
codecvt, is to be used to convert between character sets.
</p><p>
It is implicitly assumed that the locale for the default message
string in 'get' is in the "C" locale. Thus, all source code is assumed
to be written in English, so translations are always from "en_US" to
other, explicitly named locales.
</p></div><div class="section" title="Implementation"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.impl"/>Implementation</h4></div></div></div><div class="section" title="Models"><div class="titlepage"><div><div><h5 class="title"><a id="messages.impl.models"/>Models</h5></div></div></div><p>
This is a relatively simple class, on the face of it. The standard
specifies very little in concrete terms, so generic
implementations that are conforming yet do very little are the
norm. Adding functionality that would be useful to programmers and
comparable to Java's java.text.MessageFormat takes a bit of work,
and is highly dependent on the capabilities of the underlying
operating system.
</p><p>
Three different mechanisms have been provided, selectable via
configure flags:
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
generic
</p><p>
This model does very little, and is what is used by default.
</p></li><li class="listitem"><p>
gnu
</p><p>
The gnu model is complete and fully tested. It's based on the
GNU gettext package, which is part of glibc. It uses the
functions <code class="code">textdomain, bindtextdomain, gettext</code> to
implement full functionality. Creating message catalogs is a
relatively straight-forward process and is lightly documented
below, and fully documented in gettext's distributed
documentation.
</p></li><li class="listitem"><p>
ieee_1003.1-200x
</p><p>
This is a complete, though untested, implementation based on
the IEEE standard. The functions <code class="code">catopen, catgets,
catclose</code> are used to retrieve locale-specific messages
given the appropriate message catalogs that have been
constructed for their use. Note, the script <code class="code">
po2msg.sed</code> that is part of the gettext distribution can
convert gettext catalogs into catalogs that
<code class="code">catopen</code> can use.
</p></li></ul></div><p>
A new, standards-conformant non-virtual member function signature was
added for 'open' so that a directory could be specified with a given
message catalog. This simplifies calling conventions for the gnu
model.
</p></div><div class="section" title="The GNU Model"><div class="titlepage"><div><div><h5 class="title"><a id="messages.impl.gnu"/>The GNU Model</h5></div></div></div><p>
The messages facet, because it is retrieving and converting
between characters sets, depends on the ctype and perhaps the
codecvt facet in a given locale. In addition, underlying "C"
library locale support is necessary for more than just the
<code class="code">LC_MESSAGES</code> mask: <code class="code">LC_CTYPE</code> is also
necessary. To avoid any unpleasantness, all bits of the "C" mask
(i.e. <code class="code">LC_ALL</code>) are set before retrieving messages.
</p><p>
Making the message catalogs can be initially tricky, but become
quite simple with practice. For complete info, see the gettext
documentation. Here's an idea of what is required:
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
Make a source file with the required string literals that need
to be translated. See <code class="code">intl/string_literals.cc</code> for
an example.
</p></li><li class="listitem"><p>
Make initial catalog (see "4 Making the PO Template File" from
the gettext docs).</p><p>
<code class="code"> xgettext --c++ --debug string_literals.cc -o libstdc++.pot </code>
</p></li><li class="listitem"><p>Make language and country-specific locale catalogs.</p><p>
<code class="code">cp libstdc++.pot fr_FR.po</code>
</p><p>
<code class="code">cp libstdc++.pot de_DE.po</code>
</p></li><li class="listitem"><p>
Edit localized catalogs in emacs so that strings are
translated.
</p><p>
<code class="code">emacs fr_FR.po</code>
</p></li><li class="listitem"><p>Make the binary mo files.</p><p>
<code class="code">msgfmt fr_FR.po -o fr_FR.mo</code>
</p><p>
<code class="code">msgfmt de_DE.po -o de_DE.mo</code>
</p></li><li class="listitem"><p>Copy the binary files into the correct directory structure.</p><p>
<code class="code">cp fr_FR.mo (dir)/fr_FR/LC_MESSAGES/libstdc++.mo</code>
</p><p>
<code class="code">cp de_DE.mo (dir)/de_DE/LC_MESSAGES/libstdc++.mo</code>
</p></li><li class="listitem"><p>Use the new message catalogs.</p><p>
<code class="code">locale loc_de("de_DE");</code>
</p><p>
<code class="code">
use_facet<messages<char> >(loc_de).open("libstdc++", locale(), dir);
</code>
</p></li></ul></div></div></div><div class="section" title="Use"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.use"/>Use</h4></div></div></div><p>
A simple example using the GNU model of message conversion.
</p><pre class="programlisting">
#include <iostream>
#include <locale>
using namespace std;
void test01()
{
typedef messages<char>::catalog catalog;
const char* dir =
"/mnt/egcs/build/i686-pc-linux-gnu/libstdc++/po/share/locale";
const locale loc_de("de_DE");
const messages<char>& mssg_de = use_facet<messages<char> >(loc_de);
catalog cat_de = mssg_de.open("libstdc++", loc_de, dir);
string s01 = mssg_de.get(cat_de, 0, 0, "please");
string s02 = mssg_de.get(cat_de, 0, 0, "thank you");
cout << "please in german:" << s01 << '\n';
cout << "thank you in german:" << s02 << '\n';
mssg_de.close(cat_de);
}
</pre></div><div class="section" title="Future"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.future"/>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
Things that are sketchy, or remain unimplemented:
</p><div class="itemizedlist"><ul class="itemizedlist"><li class="listitem"><p>
_M_convert_from_char, _M_convert_to_char are in flux,
depending on how the library ends up doing character set
conversions. It might not be possible to do a real character
set based conversion, due to the fact that the template
parameter for messages is not enough to instantiate the
codecvt facet (1 supplied, need at least 2 but would prefer
3).
</p></li><li class="listitem"><p>
There are issues with gettext needing the global locale set
to extract a message. This dependence on the global locale
makes the current "gnu" model non MT-safe. Future versions
of glibc, i.e. glibc 2.3.x will fix this, and the C++ library
bits are already in place.
</p></li></ul></div></li><li class="listitem"><p>
Development versions of the GNU "C" library, glibc 2.3 will allow
a more efficient, MT implementation of std::messages, and will
allow the removal of the _M_name_messages data member. If this is
done, it will change the library ABI. The C++ parts to support
glibc 2.3 have already been coded, but are not in use: once this
version of the "C" library is released, the marked parts of the
messages implementation can be switched over to the new "C"
library functionality.
</p></li><li class="listitem"><p>
At some point in the near future, std::numpunct will probably use
std::messages facilities to implement truename/falsename
correctly. This is currently not done, but entries in
libstdc++.pot have already been made for "true" and "false" string
literals, so all that remains is the std::numpunct coding and the
configure/make hassles to make the installed library search its
own catalog. Currently the libstdc++.mo catalog is only searched
for the testsuite cases involving messages members.
</p></li><li class="listitem"><p> The following member functions:</p><p>
<code class="code">
catalog
open(const basic_string<char>& __s, const locale& __loc) const
</code>
</p><p>
<code class="code">
catalog
open(const basic_string<char>&, const locale&, const char*) const;
</code>
</p><p>
Don't actually return a "value less than 0 if no such catalog
can be opened" as required by the standard in the "gnu"
model. As of this writing, it is unknown how to query to see
if a specified message catalog exists using the gettext
package.
</p></li></ul></div></div><div class="bibliography" title="Bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.biblio"/>Bibliography</h4></div></div></div><div class="biblioentry"><a id="id478453"/><p><span class="citetitle"><em class="citetitle">
The GNU C Library
</em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling, and 7 Locales and Internationalization
. </span></p></div><div class="biblioentry"><a id="id478493"/><p><span class="citetitle"><em class="citetitle">
Correspondence
</em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright © 2002 . </span></p></div><div class="biblioentry"><a id="id478519"/><p><span class="citetitle"><em class="citetitle">
ISO/IEC 14882:1998 Programming languages - C++
</em>. </span><span class="copyright">Copyright © 1998 ISO. </span></p></div><div class="biblioentry"><a id="id478538"/><p><span class="citetitle"><em class="citetitle">
ISO/IEC 9899:1999 Programming languages - C
</em>. </span><span class="copyright">Copyright © 1999 ISO. </span></p></div><div class="biblioentry"><a id="id478557"/><p><span class="biblioid">
. </span><span class="citetitle"><em class="citetitle">
System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008)
</em>. </span><span class="copyright">Copyright © 2008
The Open Group/The Institute of Electrical and Electronics
Engineers, Inc.
. </span></p></div><div class="biblioentry"><a id="id478586"/><p><span class="citetitle"><em class="citetitle">
The C++ Programming Language, Special Edition
</em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername">
Addison Wesley
. </span></span></p></div><div class="biblioentry"><a id="id478624"/><p><span class="citetitle"><em class="citetitle">
Standard C++ IOStreams and Locales
</em>. </span><span class="subtitle">
Advanced Programmer's Guide and Reference
. </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright © 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername">
Addison Wesley Longman
. </span></span></p></div><div class="biblioentry"><a id="id478672"/><p><span class="biblioid">
. </span><span class="citetitle"><em class="citetitle">
API Specifications, Java Platform
</em>. </span><span class="pagenums">java.util.Properties, java.text.MessageFormat,
java.util.Locale, java.util.ResourceBundle
. </span></p></div><div class="biblioentry"><a id="id478694"/><p><span class="biblioid">
. </span><span class="citetitle"><em class="citetitle">
GNU gettext tools, version 0.10.38, Native Language Support
Library and Tools.
</em>. </span></p></div></div></div></div><div class="navfooter"><hr/><table width="100%" summary="Navigation footer"><tr><td align="left"><a accesskey="p" href="localization.html">Prev</a> </td><td align="center"><a accesskey="u" href="localization.html">Up</a></td><td align="right"> <a accesskey="n" href="containers.html">Next</a></td></tr><tr><td align="left" valign="top">Chapter 8.
Localization
</td><td align="center"><a accesskey="h" href="../spine.html">Home</a></td><td align="right" valign="top"> Chapter 9.
Containers
</td></tr></table></div></body></html>
|
