Skip to content

NH-3919 - Clean up and harmonize datetime types with regards to different dialects #703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Oct 4, 2017

Conversation

fredericDelaporte
Copy link
Member

@fredericDelaporte fredericDelaporte commented Sep 24, 2017

NH-3919 - Clean up and harmonize datetime types with regards to different dialects.

Possible breaking changes:

  • NHibernate type DateTimeType, which is the default for a .Net DateTime, does no longer cut fractional seconds. Use DateTimeNoMsType if you wish to have fractional seconds cut. It applies to its Local/Utc counterparts too.
  • LocalDateTimeType and UtcDateTimeType do no more accept being set with value having a non-matching kind, they throw instead.
  • DbTimestamp will now round the retrieved value according to Dialect.TimestampResolutionInTicks.
  • When an object typed property is mapped to a NHibernate timestamp, setting an invalid object in the property will now throw at flush instead of replacing it with DateTime.Now.
  • SQL Server 2008+ dialects now use datetime2 instead of datetime for all date time types, including timestamp. This can be reverted with sql_types.keep_datetime setting.
  • SQL Server 2008+ timestamp resolution is now 100ns in accordance with datetime2 capabilities, down from 10ms previously. This can be reverted with sql_types.keep_datetime setting.
  • Oracle 9g+ dialects now use timestamp(7) for all date time types, instead of timestamp(4).
  • Oracle 9g+ timestamp resolution is now 100ns in accordance with timestamp(7) capabilities, down from 100µs previously.
  • String parameter length will no more be specified by the OdbcDriver.
  • IMapping interface has an additional Dialect member. ISessionFactoryImplementor has lost it, since it gains it back through IMapping.
  • IDriver.ExpandQueryParameters and DriverBase.CloneParameter take an additional argument.

@@ -21,11 +21,11 @@ public void ForeignKeyNames()


string script = string.Join("\n",
cfg.GenerateSchemaCreationScript(new MsSql2000Dialect()));
cfg.GenerateSchemaCreationScript(new MsSql2008Dialect()));
Copy link
Member Author

@fredericDelaporte fredericDelaporte Sep 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQL 2000 dialect was failing due to "unknow type datetime2"...

@@ -1,26 +1,21 @@
<Project Sdk="Microsoft.NET.Sdk">
<Import Project="../../build-common/NHibernate.props" />

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I revert those riders glitches, or just let it do that once and hopefully for all?

@@ -36,5 +74,26 @@ protected override void RegisterDefaultProperties()
base.RegisterDefaultProperties();
DefaultProperties[Environment.ConnectionDriver] = typeof(Sql2008ClientDriver).AssemblyQualifiedName;
}

public override string CurrentTimestampSQLFunctionName =>
"SYSDATETIME()";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously it was using CURRENT_TIMESTAMP, calling indeed getdate(), so yielding datetime instead of datetime2.

@@ -39,7 +39,7 @@ public override IType DataType
return fe.DataType;
}
ISQLFunction sf = Walker.SessionFactoryHelper.FindSQLFunction(Text);
return sf == null ? null : sf.ReturnType(null, null);
return sf?.ReturnType(null, Walker.SessionFactoryHelper.Factory);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unique case I have spotted where a methods chain of calls ending by calling IType.SqlTypes was not supplying an IMapping (the factory in this case).

@@ -16,7 +16,8 @@ public partial class ClassMetaType : AbstractType
{
public override SqlType[] SqlTypes(IMapping mapping)
{
return new SqlType[] { NHibernateUtil.String.SqlType };
var types = new[] { NHibernateUtil.String.SqlType };
return mapping?.Dialect.RefineSqlTypes(this, types) ?? types;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added that logic for each type, excepted those yielding no types and those building them from other types.

@@ -34,9 +34,9 @@ private IInternalLogger Log

/// <summary>
/// Initialize a new instance of the NullableType class using a
/// <see cref="SqlType"/>.
/// <see cref="NHibernate.SqlTypes.SqlType"/>.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was referencing the NullabeType.SqType property otherwise.

{
return new SqlType[] {SqlType};
var types = new[] { SqlType };
return mapping?.Dialect.RefineSqlTypes(this, types) ?? types;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That the main change for supporting the feature: this allow switching the NHibernate type default corresponding sql type for another.

Hibernate does not do that. They do not need. Java does not know DbType.DateTime and DbType.DateTime2, it only knows Types.TIMESTAMP. Hibernate just need to tell this one should be mapped to sql type datetime2 in SQL 2008 dialect.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They call it on nullSafeGet/nullSafeSet/extract(no equivalent on our side)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. It does not seems near to be ported currently, since it depends on their type descriptor which is not ported currently, and uses instance overrides for doBind which relies on members of the calling class, pattern not doable that way in .Net. (We could likely use method delegates instead.) And I have not checked if their override is taken into account also for hbm2ddl, as mine.

{
var validator = new SchemaValidator(cfg);
validator.Validate();
Copy link
Member Author

@fredericDelaporte fredericDelaporte Sep 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delegating to Validate for checking we have the expected type in database. Which means being confident Validate correctly knows if it should be DateTime or DateTime2.
It appeared to me overly complicated to test this manually for all dialects, so I delegated to Validate. I have furthermore checked in a debug session directly in the db what type it was to ascertain we were now having datetime2 with SQL2008.

}

private void AssertSqlType(ClientDriverWithParamsStats driver)
private void AssertSqlType(ClientDriverWithParamsStats driver, int expectedCount)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking the exact usage count by the way.

}

[Test]
public void QueryUseExpectedSqlType()
{
if (!TestDialect.SupportsNonDataBoundCondition)
Assert.Ignore("Dialect does not support the test query");
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debuging the failure, the expected DbType and SqlType usages count is found (5), but Firebird does not accept the last condition. Removing it allows the query to be executed, but I want this condition for avoiding having the Hql parser determining the type for all parameters from the properties they are compared to, and instead use the type supplied by the parameter. So I ignore this test for Firebird.

@fredericDelaporte
Copy link
Member Author

Last commit is part 2. I intend to add more tests though.

var grandchild = new Node {Name = "grandchild"};
var root = new Node
{
Created = RoundForDialect(DateTime.Now),
Copy link
Member Author

@fredericDelaporte fredericDelaporte Sep 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just fixing the trouble in test. It could be fixed at the type level by overriding IsEqual (and preferably setter and getter too). But this would be a possible breaking change for timestamp when used a as simple property, because this will cause it to use the dialect declared accuracy instead of the actual database accuracy.

Copy link
Member Author

@fredericDelaporte fredericDelaporte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Odbc causes issue with milliseconds due to NH-3895 which does not seem actually fixed when the Loader is used.

@@ -6,23 +6,23 @@
<id name="Id" type="Int32">
<generator class="assigned" />
</id>
<property name="Name" length="2000" />
<property name="Name" length="1999" />
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With below change to expand parameters, the size defined by OdbcDriver now actually ends in final parameters, and it causes issues for nvarchar(2000) and above: they get converted to NText by ODBC, causing:

ERROR [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]The data types nvarchar and ntext are incompatible in the equal to operator.

So here this is just a workaround for this test which was not about long string. But I fear there is a serious trouble with Odbc, and we should consider removing that size specification which was not actually taken into accont until below change.

Copy link
Member Author

@fredericDelaporte fredericDelaporte Sep 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have searched about this trouble of ODBC switching to ntext for nvarchar >= 2000. Only to find someone confirming it here, but no one seem to have a clue about why is it like this.

@@ -231,7 +231,7 @@ public void RemoveUnusedCommandParameters(DbCommand cmd, SqlString sqlString)
.ForEach(unusedParameterName => cmd.Parameters.RemoveAt(unusedParameterName));
}

public virtual void ExpandQueryParameters(DbCommand cmd, SqlString sqlString)
public virtual void ExpandQueryParameters(DbCommand cmd, SqlString sqlString, SqlType[] parameterTypes)
Copy link
Member Author

@fredericDelaporte fredericDelaporte Sep 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method in the case of ODBC was replacing the original parameters by new one, partially clone, leaving aside Scale, Precision and Size set by OdbcDriver.InitializeParamater. So I wonder if those changes in InitializeParameter were tested. It does not seem to be the case. When looking at NH-3895 which was supposed to be fixed by one of those changes, there are no test in the commit.

Due to ExpandQueryParameters, Scale is lost. Due to the change in DateTimeType, we now have milliseconds, and this has triggered the NH-3895 bug for numerous tests. So I have changed ExpandQueryParameters for it to create parameters the same way as original one, calling InitializeParamater. This fixes NH-3895. But it has triggered the trouble explained in previous comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having dug into files history, the size setting has been added with 03eb58a in January 2010, while the ExpandQueryParameters was not yet there. This one has been added in July 2010 with 131b97e. So, not having Size/Precision/Scale actually specified for Odbc parameters (and all other drivers not handling named parameters - DB2, Ifx, Ingres, OleDb, OracelLite, SybaseAsa - if they do set them in InitializeParameter,) is an old regression.

Copy link
Member Author

@fredericDelaporte fredericDelaporte Sep 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But later on NH-3036 reverted the size setting for sql client due to issues with like (which may requires larger size than the actual column size), so I think we should remove this Size setting from Odbc parameters too.
In fact Odbc seems to re-adjust size anyway according to some comment in Set of string type.

//TODO: Fix me after 4.6.2 update. Size and Precision has been added to DbParameter
var p = dataParameter as IDbDataParameter;
if (p != null)
return p.DbType + " (" + p.Size + ":" + p.Scale + ":" + p.Precision + ")";
Copy link
Member Author

@fredericDelaporte fredericDelaporte Sep 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have I missed something or this todo was wrong on the version number? Precision was added in 4.5.1, and others are older.

@@ -29,14 +29,14 @@ public void SimpleSelectTest()
long simple1Key = 15;
Simple simple1 = new Simple();
simple1.Address = "Street 12";
simple1.Date = DateTime.Now;
simple1.Date = RoundForDialect(DateTime.Now);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many tests comparing date before and after saving are changed to round them to the dialect known accuracy. Previously they were rounded to seconds due to DateTimeType Hibernate like semantic. Now that NHibernate does no more round to second for this type, we need to take into account the database accuracy.

@@ -40,31 +38,33 @@ public void SelectConditionalValuesTest()
.Select(o => new
{
o.Color,
AliveDays = (int)(DateTime.Now - o.Birthdate).TotalDays,
AliveDays = (now - o.Birthdate).TotalDays,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test had start being flaky with DateTimeType no more truncating to seconds. The diff followed by TotalDays does not count boundaries but a fractional days count. We should not truncate it to int but better instead compare with a tolerance.

@@ -23,14 +24,14 @@ public void Next()
[Test]
public void Seed()
{
DateTimeType type = NHibernateUtil.DateTime;
var type = NHibernateUtil.DateTime2;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yuck. Was not testing what it was supposed to.

/// TestFixtures for the <see cref="DateTimeNoMsType"/>.
/// </summary>
[TestFixture]
public class DateTimeNoMsTypeFixture
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact this is what was DateTimeTypeFixture.

// This test fails against the ODBC driver. The driver would need to be override to allow longer parameter sizes than the column.
AssertExpectedFailureOrNoException(
exception,
(Sfi.ConnectionProvider.Driver is OdbcDriver));
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was no more failing since 131b97e which was losing the Size set on parameter. But since "expand parameter" is now fixed, it would fail again if we do not do NH-4083.


namespace NHibernate.Type
{
/// <summary>
/// This is almost the exact same type as the DateTime except it can be used
/// in the version column, stores it to the accuracy the database supports,
/// and will default to the value of DateTime.Now if the value is null.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default to the value of DateTime.Now if the value is null: wrong, since it was "implemented" into Set which is not called when the value is null.
Moreover since when this comment was written, it seems DateTimeType has gained the ability to serve as version too.

/// and will default to the value of DateTime.Now if the value is null.
/// This is almost the exact same type as the <see cref="DateTimeType" /> excepted it can be used
/// in the version column, stores it to the accuracy the database supports
/// and will default to the value of <see cref="TimestampType.Now" /> if the value is not valid.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may remove that since I do not think it was intended, the aim for the code being to replace null values, but in a method which is never called on null. It coud be a breaking change for edge cases, where a timestamp is mapped to an object property in which the app may put non DateTime data...

I have left the accuracy part in case for some reason we were to reduce DateTimeType one again.

}
/// <inheritdoc />
protected override DateTime GetDateTimeToSet(object value, ISessionImplementor session) =>
(value as DateTime?) ?? Now;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This override should be removed if we want to clean the failed attempt at replacing null values by Now. By the way if keeping it, we should probably call Seed instead of Now.

return '\'' + value.ToString() + '\'';
}
public override string ToString(object val) =>
((DateTime) val).ToString("O");
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the only significant change of Timestamp compared to the new DateTimeType.

var dbValue = base.GetDateTimeToSet(value, session);
if (dbValue.Kind != DateTimeKind.Utc)
throw new ArgumentException("Kind is NOT Utc", nameof(value));
return dbValue;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With Timestamp.ToString, that is all the differences with UtcDateTimeType. UtcDateTimeType does not throw if it receives a non UTC value: it silently switch its kind to UTC.

/// for dialects supporting datetime2.
/// </summary>
[Obsolete("Migrate SQL datetime type to datetime2 instead.")]
public const string SqlTypesKeepDateTime = "sql_types.keep_datetime";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DateTime2 being unsupported by OdbcDriver, we may use it for allowing using 2008 dialect with Odbc. So I think I will remove the obsolete attribute.

The subject is a bit deeper since Odbc does not understand Time either, and likely Date too: we should have a mean to switch those types too. So we can also decide that only dialects up to 2005 are compatible with ODBC.
(By the way this page states ODBC does support SQL2008 types, but how are we supposed to do in a .Net context, mystery.)

An other option would be to try moving the type override from dialect to driver. But currently IMapping interface which is where I pull the dialect from for refining types is not much suitable for having the driver, unless we add the ability for Configuration.Mapping to instantiate a driver.

I think I will just remove the Obsolete attribute.

@fredericDelaporte fredericDelaporte force-pushed the NH-3919 branch 2 times, most recently from 509b41d to ee12f78 Compare September 27, 2017 18:31
@fredericDelaporte
Copy link
Member Author

All date time types are now tested. One more bug to fix by the way.
I intend to test what it gives using timestamp as version on a SQL Server db having datetime columns instead of datetime2: it will surely cause stale update exceptions, which will be the main breaking change for this switch from datetime to datetime2.

@hazzik
Copy link
Member

hazzik commented Sep 27, 2017

Such a Pandora's box.

@fredericDelaporte
Copy link
Member Author

If it is not 5.0 it should be 6, too much changes for a minor.

@hazzik
Copy link
Member

hazzik commented Sep 27, 2017

If it is not 5.0 it should be 6, too much changes for a minor.

I think this is better to leave it for 6. A good exercise for a types refactoring. At least much better than trying to rush in 5.0

hazzik
hazzik previously approved these changes Oct 4, 2017
case DbType.Time:
return true;
}
return false;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I have been a bit quick with the "refactor extract method here".

@fredericDelaporte fredericDelaporte merged commit c7845d8 into nhibernate:master Oct 4, 2017
@fredericDelaporte fredericDelaporte deleted the NH-3919 branch October 4, 2017 13:56
@hazzik
Copy link
Member

hazzik commented Oct 4, 2017

I just realized that "NoMs" types are not needed anymore.

@fredericDelaporte
Copy link
Member Author

Why, because we have scale? Scale is not understood and handled by every database.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants