Skip to content

Test Unicode string. #1511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 31, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions build-common/teamcity-hibernate.cfg.xml
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,7 @@
<property name="adonet.wrap_result_sets">false</property>

<property name="odbc.explicit_datetime_scale"></property>
<property name="oracle.use_n_prefixed_types_for_unicode"></property>
Copy link
Member Author

@fredericDelaporte fredericDelaporte Jan 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TeamCity Nant build logic just does nothing of properties to adjust when they are lacking in this file, thus causing Oracle to not be properly set up for actually using Unicode with DbType.String type. On our TeamCity agent, its default charset is an ascii one, and so we must use the "N" prefixed char types for storing Unicode strings.

<property name="query.default_cast_length"></property>
</session-factory>
</hibernate-configuration>
20 changes: 20 additions & 0 deletions default.build
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,26 @@
<namespace prefix="hbm" uri="urn:nhibernate-configuration-2.2" />
</namespaces>
</xmlpoke>

<!-- Make sure the property exists - it's only set for some scenarios. -->
<property name="nhibernate.oracle.use_n_prefixed_types_for_unicode" value="" unless="${property::exists('nhibernate.oracle.use_n_prefixed_types_for_unicode')}"/>
<xmlpoke file="${app.config}"
xpath="//*/hbm:property[@name='oracle.use_n_prefixed_types_for_unicode']"
value="${nhibernate.oracle.use_n_prefixed_types_for_unicode}">
<namespaces>
<namespace prefix="hbm" uri="urn:nhibernate-configuration-2.2" />
</namespaces>
</xmlpoke>

<!-- Make sure the property exists - it's only set for some scenarios. -->
<property name="nhibernate.query.default_cast_length" value="" unless="${property::exists('nhibernate.query.default_cast_length')}"/>
<xmlpoke file="${app.config}"
xpath="//*/hbm:property[@name='query.default_cast_length']"
value="${nhibernate.query.default_cast_length}">
<namespaces>
<namespace prefix="hbm" uri="urn:nhibernate-configuration-2.2" />
</namespaces>
</xmlpoke>
</target>

<target name="put-connection-settings-into-app-config">
Expand Down
11 changes: 7 additions & 4 deletions lib/teamcity/firebird/firebird_installation.txt
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ UserManager = Srp
WireCrypt = Enabled
11. Restart Firebird service.

For manual testing, take care of not creating it with inadequate acl on the file. This may happen
if you use ISQL with a connection string causing it to create it in embedded mode, without actually
using the server. Prefixing your path with "localhost:" should avoid that.
For manual testing, take care of not creating the NHibernate database with inadequate acl on the file.
This may happen if you use ISQL with a connection string causing it to create it in embedded mode,
without actually using the server. Prefixing your path with "localhost:" should avoid that.
Due to test constraints, make sure you create the database with a 16384 page size.

For tests performances, and since it is just an expandable test database, better disable forced writes.
Since those tests drop/create schema constantly, they are quite heavy on writes and this single setting
Expand All @@ -35,4 +36,6 @@ b. From Firebird installation folder, run:
gfix -w async nhibernate -user SYSDBA
(Change "nhibernate" to your own alias or path as appropriate for your setup)
c. Restart Firebird service.
Note that the TestDatabaseSetup will drop and recreate the database when run, with forced writes disabled.
Note that the TestDatabaseSetup will drop and recreate the database when run, with forced writes disabled
and with a 16384 page size. So you may consider running the TestDatabaseSetup test instead for creating the
NHibernate database.
5 changes: 3 additions & 2 deletions lib/teamcity/oracle/oracle_installation.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ Installation steps for Oracle for NH TeamCity:

2. Run the installer ...
3. Choose any HTTP port for the listener if asked (it may automatically choose the default 8080 if available);
4. Choose the ASCII character set. You will have to ensure nhibernate.oracle.use_n_prefixed_types_for_unicode
parameter is set to true; Choosing instead the Unicode character set will cause some NHibernate tests to fail.
4. Choose the ASCII character set. You will have to ensure oracle.use_n_prefixed_types_for_unicode
parameter is set to true and that query.default_cast_length is set to 2000. Choosing instead the Unicode
character set will cause some NHibernate tests to fail.
5. Enter 'password' as the password for the SYS and SYSTEM accounts;
6. The setup should install Oracle on the machine.

Expand Down
1 change: 1 addition & 0 deletions src/NHibernate.Config.Templates/FireBird.cfg.xml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ for your own use before compile tests in VisualStudio.
Database=nhibernate;
User ID=SYSDBA;Password=masterkey;
MaxPoolSize=200;
charset=utf8;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default charset for FireBird is "None", theoretically meaning that it is the application business to encode/decode it, Firebird being supposed to stores and gives data back "as is", maybe in some binary sens. In our case, that goes wrong. I have not investigated why, I have just switched that to utf8 instead.

This requires to recreate the database, which the TeamCity build does for Firebird. For local tests, adjust your configuration files accordingly and run the TestDatabaseSetup test for fixing this.

(Of course this file here is just the default template, the actual fix for TeamCity is the similar change done in teamcity.build file.)

</property>
<property name="show_sql">false</property>
<property name="dialect">NHibernate.Dialect.FirebirdDialect</property>
Expand Down
7 changes: 7 additions & 0 deletions src/NHibernate.Config.Templates/Oracle-Managed.cfg.xml
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,12 @@ for your own use before compile tests in VisualStudio.
<property name="show_sql">false</property>
<property name="dialect">NHibernate.Dialect.Oracle10gDialect</property>
<property name="query.substitutions">true 1, false 0, yes 'Y', no 'N'</property>
<!-- If your database setup use an ASCII charset, switch following property to true. -->
<property name="oracle.use_n_prefixed_types_for_unicode">false</property>
<!-- Depending on your database setup, the default cast length of 4000 may be too big.
By example, if previous setting is true, NHibernate may try to use nvarchar2(4000),
which will be rejected if its underlying charset is UTF16 and the database
MAX_STRING_SIZE is not extended. In such case, reduce it to 2000. -->
<property name="query.default_cast_length"></property>
</session-factory>
</hibernate-configuration>
7 changes: 7 additions & 0 deletions src/NHibernate.Config.Templates/Oracle.cfg.xml
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,12 @@ for your own use before compile tests in VisualStudio.
<property name="show_sql">false</property>
<property name="dialect">NHibernate.Dialect.OracleDialect</property>
<property name="query.substitutions">true 1, false 0, yes 'Y', no 'N'</property>
<!-- If your database setup use an ASCII charset, switch following property to true. -->
<property name="oracle.use_n_prefixed_types_for_unicode">false</property>
<!-- Depending on your database setup, the default cast length of 4000 may be too big.
By example, if previous setting is true, NHibernate may try to use nvarchar2(4000),
which will be rejected if its underlying charset is UTF16 and the database
MAX_STRING_SIZE is not extended. In such case, reduce it to 2000. -->
<property name="query.default_cast_length"></property>
</session-factory>
</hibernate-configuration>
39 changes: 30 additions & 9 deletions src/NHibernate.Test/Async/TypesTest/StringTypeFixture.cs
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,13 @@
//------------------------------------------------------------------------------


using System;
using System.Linq;
using NUnit.Framework;
using NHibernate.Linq;

namespace NHibernate.Test.TypesTest
{
using System.Threading.Tasks;
/// <summary>
/// Summary description for StringTypeFixture.
/// </summary>
[TestFixture]
public class StringTypeFixtureAsync : TypeFixtureBase
{
Expand All @@ -25,6 +23,14 @@ protected override string TypeName
get { return "String"; }
}

protected override void OnTearDown()
{
using (var s = OpenSession())
{
s.CreateQuery("delete from StringClass").ExecuteUpdate();
}
}

[Test]
public async Task InsertNullValueAsync()
{
Expand All @@ -38,12 +44,27 @@ public async Task InsertNullValueAsync()

using (ISession s = OpenSession())
{
StringClass b = (StringClass) await (s.CreateCriteria(
typeof(StringClass)).UniqueResultAsync());
Assert.IsNull(b.StringValue);
await (s.DeleteAsync(b));
StringClass b = (StringClass) await (s.CreateCriteria(typeof(StringClass)).UniqueResultAsync());
Assert.That(b.StringValue, Is.Null);
}
}

[Test]
public async Task InsertUnicodeValueAsync()
{
const string unicode = "길동 최고 新闻 地图 ます プル éèêëîïôöõàâäåãçùûü бджзй αβ ខគឃ ضذخ";
using (var s = OpenSession())
{
var b = new StringClass { StringValue = unicode };
await (s.SaveAsync(b));
await (s.FlushAsync());
}

using (var s = OpenSession())
{
var b = await (s.Query<StringClass>().SingleAsync());
Assert.That(b.StringValue, Is.EqualTo(unicode));
}
}
}
}
}
38 changes: 29 additions & 9 deletions src/NHibernate.Test/TypesTest/StringTypeFixture.cs
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
using System;
using System.Linq;
using NUnit.Framework;

namespace NHibernate.Test.TypesTest
{
/// <summary>
/// Summary description for StringTypeFixture.
/// </summary>
[TestFixture]
public class StringTypeFixture : TypeFixtureBase
{
Expand All @@ -14,6 +11,14 @@ protected override string TypeName
get { return "String"; }
}

protected override void OnTearDown()
{
using (var s = OpenSession())
{
s.CreateQuery("delete from StringClass").ExecuteUpdate();
}
}

[Test]
public void InsertNullValue()
{
Expand All @@ -27,12 +32,27 @@ public void InsertNullValue()

using (ISession s = OpenSession())
{
StringClass b = (StringClass) s.CreateCriteria(
typeof(StringClass)).UniqueResult();
Assert.IsNull(b.StringValue);
s.Delete(b);
StringClass b = (StringClass) s.CreateCriteria(typeof(StringClass)).UniqueResult();
Assert.That(b.StringValue, Is.Null);
}
}

[Test]
public void InsertUnicodeValue()
{
const string unicode = "길동 최고 新闻 地图 ます プル éèêëîïôöõàâäåãçùûü бджзй αβ ខគឃ ضذخ";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is some characters taken from NHCH-43, then some European accentuated letters, some Cyrillic letters, two Greek ones, a bunch of Khmer ones (likely the script supported the latest on windows) and some Arabic ones (which is in my experience one of the worst to support due to right-to-left logic, but that is more a trouble for display than for storage; Khmer was easier to deal with even when not yet officially supported on Windows).

using (var s = OpenSession())
{
var b = new StringClass { StringValue = unicode };
s.Save(b);
s.Flush();
}

using (var s = OpenSession())
{
var b = s.Query<StringClass>().Single();
Assert.That(b.StringValue, Is.EqualTo(unicode));
}
}
}
}
}
5 changes: 4 additions & 1 deletion src/NHibernate.TestDatabaseSetup/TestDatabaseSetup.cs
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,10 @@ private static void SetupFirebird(Cfg.Configuration cfg)
{
Console.WriteLine(e);
}
FbConnection.CreateDatabase(connStr, forcedWrites:false);
// With UTF8 charset, string takes up to four times as many space, causing the
// default page-size of 4096 to no more be enough for index key sizes. (Index key
// size is limited to a quarter of the page size.)
FbConnection.CreateDatabase(connStr, pageSize:16384, forcedWrites:false);
}

private static void SetupSqlServerCe(Cfg.Configuration cfg)
Expand Down
16 changes: 14 additions & 2 deletions teamcity.build
Original file line number Diff line number Diff line change
Expand Up @@ -76,14 +76,14 @@
<target name="setup-teamcity-firebird32">
<property name="nhibernate.connection.driver_class" value="NHibernate.Driver.FirebirdClientDriver" />
<property name="nhibernate.dialect" value="NHibernate.Dialect.FirebirdDialect" />
<property name="nhibernate.connection.connection_string" value="DataSource=localhost;Database=nhibernate;User ID=SYSDBA;Password=masterkey;MaxPoolSize=200;" />
<property name="nhibernate.connection.connection_string" value="DataSource=localhost;Database=nhibernate;User ID=SYSDBA;Password=masterkey;MaxPoolSize=200;charset=utf8;" />
</target>

<target name="setup-teamcity-firebird64">
<property name="nunit-x64" value="true" />
<property name="nhibernate.connection.driver_class" value="NHibernate.Driver.FirebirdClientDriver" />
<property name="nhibernate.dialect" value="NHibernate.Dialect.FirebirdDialect" />
<property name="nhibernate.connection.connection_string" value="DataSource=localhost;Database=nhibernate;User ID=SYSDBA;Password=masterkey;MaxPoolSize=200;" />
<property name="nhibernate.connection.connection_string" value="DataSource=localhost;Database=nhibernate;User ID=SYSDBA;Password=masterkey;MaxPoolSize=200;charset=utf8;" />
</target>

<target name="setup-teamcity-sqlite32">
Expand Down Expand Up @@ -127,6 +127,9 @@
<property name="nhibernate.connection.connection_string" value="User ID=nhibernate;Password=nhibernate;Metadata Pooling=false;Self Tuning=false;Data Source=(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XE)))" />
<!-- Teamcity Oracle test database is configured with a non-Unicode encoding -->
<property name="nhibernate.oracle.use_n_prefixed_types_for_unicode" value="true" />
<!-- The default value of 4000 is too big for nvarchar2 with default database settings.
nvarchar2 may be used due to use_n_prefixed_types_for_unicode -->
<property name="nhibernate.query.default_cast_length" value="2000" />
<copy todir="${bin.dir}">
<fileset basedir="${root.dir}/lib/teamcity/oracle/x86">
<include name="*.dll"/>
Expand All @@ -143,6 +146,9 @@
<property name="nhibernate.connection.connection_string" value="User ID=nhibernate;Password=nhibernate;Metadata Pooling=false;Self Tuning=false;Data Source=(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XE)))" />
<!-- Teamcity Oracle test database is configured with a non-Unicode encoding -->
<property name="nhibernate.oracle.use_n_prefixed_types_for_unicode" value="true" />
<!-- The default value of 4000 is too big for nvarchar2 with default database settings.
nvarchar2 may be used due to use_n_prefixed_types_for_unicode -->
<property name="nhibernate.query.default_cast_length" value="2000" />
<copy todir="${bin.dir}">
<fileset basedir="${root.dir}/lib/teamcity/oracle/x64">
<include name="*.dll"/>
Expand All @@ -158,6 +164,9 @@
<property name="nhibernate.connection.connection_string" value="User ID=nhibernate;Password=nhibernate;Metadata Pooling=false;Self Tuning=false;Data Source=(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XE)))" />
<!-- Teamcity Oracle test database is configured with a non-Unicode encoding -->
<property name="nhibernate.oracle.use_n_prefixed_types_for_unicode" value="true" />
<!-- The default value of 4000 is too big for nvarchar2 with default database settings.
nvarchar2 may be used due to use_n_prefixed_types_for_unicode -->
<property name="nhibernate.query.default_cast_length" value="2000" />
</target>

<target name="setup-teamcity-oracle-managed64">
Expand All @@ -167,6 +176,9 @@
<property name="nhibernate.connection.connection_string" value="User ID=nhibernate;Password=nhibernate;Metadata Pooling=false;Self Tuning=false;Data Source=(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XE)))" />
<!-- Teamcity Oracle test database is configured with a non-Unicode encoding -->
<property name="nhibernate.oracle.use_n_prefixed_types_for_unicode" value="true" />
<!-- The default value of 4000 is too big for nvarchar2 with default database settings.
nvarchar2 may be used due to use_n_prefixed_types_for_unicode -->
<property name="nhibernate.query.default_cast_length" value="2000" />
</target>

<target name="setup-teamcity-mysql">
Expand Down