Alternate Method Encoding. Unicode. Getbytes in Native C++
Need Implement Encoding. Unicode. Getbytes in Native C++. .Net Implementation: Console. Writeline("Codepage Number: " + Encoding. Unicode. Codepage...
Need implement Encoding.Unicode.GetBytes in native C++.
.NET implementation:
Console.WriteLine("codePage number: " + Encoding.Unicode.CodePage.ToString());
Console.Write("string: ");
foreach (var ch in Encoding.Unicode.GetBytes("string"))
Console.Write(ch.ToString("X") + "-");
Console.WriteLine();
Console.Write("строка: ");
foreach (var ch in Encoding.Unicode.GetBytes("строка"))
Console.Write(ch.ToString("X") + "-");
Console.ReadLine();
.NET implementation output:
codePage number: 1200
string: 73-0-74-0-72-0-69-0-6E-0-67-0
строка: 41-4-42-4-40-4-3E-4-3A-4-30-4
How implement this method (without use boost, QT, etc..) to C++?
I found this method from Windows:
#include <exception>
#include <iostream>
#include <ostream>
#include <string>
#include <Windows.h>
std::wstring ConvertToUTF16(const std::string & source, const UINT codePage)
{
// Fail if an invalid input character is encountered
static const DWORD conversionFlags = MB_ERR_INVALID_CHARS;
// Require size for destination string
int utf16Length = ::MultiByteToWideChar(
codePage, // code page for the conversion
conversionFlags, // flags
source.c_str(), // source string
source.length(), // length (in chars) of source string
NULL, // unused - no conversion done in this step
0 // request size of destination buffer, in wchar_t's
);
if (utf16Length == 0)
{
const DWORD error = ::GetLastError();
throw std::exception(
"MultiByteToWideChar() failed: Can't get length of destination UTF-16 string.",
error);
}
// Allocate room for destination string
std::wstring utf16Text;
utf16Text.resize(utf16Length);
// Convert to Unicode
if (!::MultiByteToWideChar(
codePage, // code page for conversion
0, // validation was done in previous call
source.c_str(), // source string
source.length(), // length (in chars) of source string
&utf16Text[0], // destination buffer
utf16Text.length() // size of destination buffer, in wchar_t's
))
{
const DWORD error = ::GetLastError();
throw std::exception(
"MultiByteToWideChar() failed: Can't convert to UTF-16 string.",
error);
}
return utf16Text;
}
void main()
{
try
{
// ASCII text
std::string inText("string");
// Unicode
static const UINT codePage = 1200;
// Convert to Unicode
const std::wstring utf16Text = ConvertToUTF16(inText, codePage);
// Show result
for (size_t i = 0; i < utf16Text.size(); i++)
printf("%X-", utf16Text[i]);
}
catch (const std::exception& e)
{
std::cerr << "*** ERROR:\n";
std::cerr << e.what();
std::cerr << std::endl;
}
getchar();
}
but MultiByteToWideChar no return string size for 1200 code page (Unicode).
1 Answer
The codepage parameter of MultiByteToWideChar() specifies the encoding of the input char data so it can be converted FROM that encoding TO UTF-16. You never use codepage 1200 in Win32 programming.
Strings in .NET are encoded in UTF-16. Encoding.Unicode.GetBytes() returns a UTF-16LE encoded byte array. So the character data is returned as-is as bytes.
For UTF-16 on Windows, use wchar_t or char16_t based strings (like std::wstring or std::u16string). If you need a UTF-16 encoded byte array, allocate 2 * length bytes (such as with a std::vector) and copy the raw string characters as-is:
std::vector<BYTE> GetUnicodeBytes(const std::wstring &str)
{
std::vector<BYTE> result;
if (!str.empty())
{
result.resize(sizeof(wchar_t) * str.length());
CopyMemory(&result[0], str.c_str(), result.size());
}
return result;
}
std::wcout << L"string: ";
for (auto ch: GetUnicodeBytes(L"string"))
std::wcout << std::hex << (int)ch << L"-";
std::wcout << std::endl;
std::wcout << L"строка: ";
for (auto ch: GetUnicodeBytes(L"строка"))
std::wcout << std::hex << (int)ch << L"-";
std::wcout << std::endl;
Alternatively:
std::vector<BYTE> GetUnicodeBytes(const std::u16string &str)
{
std::vector<BYTE> result;
if (!str.empty())
{
result.resize(sizeof(char16_t) * str.length());
CopyMemory(&result[0], str.c_str(), result.size());
}
return result;
}
std::wcout << L"string: ";
for (auto ch: GetUnicodeBytes(u"string"))
std::wcout << std::hex << (int)ch << L"-";
std::wcout << std::endl;
std::wcout << L"строка: ";
for (auto ch: GetUnicodeBytes(u"строка"))
std::wcout << std::hex << (int)ch << L"-";
std::wcout << std::endl;
Marcus Vance is a cybersecurity auditor and technology writer dedicated to educating the public about online safety, data privacy regulations, enterprise security, and emerging cyber threats.
