Encoding problem

[StyXman]

I'm not sure this is a proper way to use TagReferences, but it's definitely unexpected. This time I'm using GitPython installed by pypi.

I have this nice tag:

In [8]: tag
Out[8]: <git.TagReference "refs/tags/PROMOTED_1501131729_MKT15_01_12_QU_1">

I can get a lot of info out of it:

In [9]: tag.object.hexsha
Out[9]: u'dca63c5c7e6aab3cd4934e60230ec3419ab87071'

In [12]: tag.name
Out[12]: 'PROMOTED_1501131729_MKT15_01_12_QU_1'

In [13]: tag.object
Out[13]: <git.TagObject "dca63c5c7e6aab3cd4934e60230ec3419ab87071">

In [14]: tag.ref
TypeError: PROMOTED_1501131729_MKT15_01_12_QU_1 is a detached symbolic reference as it points to 'dca63c5c7e6aab3cd4934e60230ec3419ab87071'

But this fails:

In [15]: tag.commit
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-15-2431a6e80cf9> in <module>()
----> 1 tag.commit

/home/mdione/local/lib/python2.7/site-packages/git/refs/tag.pyc in commit(self)
     29         elif obj.type == "tag":
     30             # it is a tag object which carries the commit as an object - we can point to anything
---> 31             return obj.object
     32         else:
     33             raise ValueError("Tag %s points to a Blob or Tree - have never seen that before" % self)

/home/mdione/local/lib/python2.7/site-packages/gitdb/util.pyc in __getattr__(self, attr)
--> 237         self._set_cache_(attr)
    238         # will raise in case the cache was not created
    239         return object.__getattribute__(self, attr)

/home/mdione/local/lib/python2.7/site-packages/git/objects/tag.pyc in _set_cache_(self, attr)
     54         if attr in TagObject.__slots__:
     55             ostream = self.repo.odb.stream(self.binsha)
---> 56             lines = ostream.read().decode(defenc).splitlines()
     57
     58             obj, hexsha = lines[0].split(" ")       # object <hexsha>

/usr/lib/python2.7/encodings/utf_8.pyc in decode(input, errors)
     14
     15 def decode(input, errors='strict'):
---> 16     return codecs.utf_8_decode(input, errors, True)
     17
     18 class IncrementalEncoder(codecs.IncrementalEncoder):

UnicodeDecodeError: 'utf8' codec can't decode byte 0xa8 in position 108: invalid start byte

Unluckily this is happening with an internal repo and I don't know how to even try to reproduce with a public one. Meanwhile I can workaround it by using tag.object.hexsha, which is what I wanted.

[Byron]

Thanks for posting this issue !

It seems that the TagObject's information can't be decoded as it contains a non-utf-8 encoding which is unexpected.
Maybe it is safer to not attempt to decode anything, and leave that to the client, who could read the bytes of the associated tag-object and parse them with a suitable encoding in mind.

Even though you have already discovered a workaround, the original problem remains. A proper fix would re-evaluate the current code and prefer to work on bytes instead of a decoded string.

[StyXman]

In fact, no, tag.object.hexsha is not what I'm looking for.

[StyXman]

More data: technically this is an encoding error in the data itself:

In [11]: stream= pricing.odb.stream(tag.object.binsha)

In [12]: stream.read()
Out[12]: 'object 4b50858c4debda3ad5d6ea5b7a485cd4eb5ecc73\ntype commit\ntag PROMOTED_1501131729_MKT15_01_12_QU_1\ntagger \xa8John Doe <jdoe@megacorp.com> 1421167136 +0100\n\nMerged CIU_MKT1501_28 to remote master\n'

You can see the offensive character just before the tagger's name (technically being part of it). In the other hand, I don't know even if git handles this, but what happens when different objects are encoded with different encodings? I'm pretty sure git objects do not store this kind of info...

[Byron]

Using the tag.object it should be straightforward to obtain the raw-bytes stored in the tag-object, in case this is what you are actually looking for. Those represent a few formatted lines of information, which could be parsed with code similar to the one currently in use.
Parsing can only safely operate on bytes though, as the encoding seems not to be UTF-8 at all times.

Even if parsing is made to work at some point, right now the tagger-name are expected to be str/unicode instances, which couldn't be obtained if the encoding of the underlying bytes are unknown.

[StyXman]

What about using decode(defenc, 'ignore')? I hope it doesn't break anything else. I'll try that locally.

[Byron]

Great idea !
Of course it's questionable whether the program should silently drop information, instead of loudly abort operation as it currently does.
It seems that it's generally unwise to make assumptions about the encoding in TagObjects, so the implementation should leave it to the client to deal with that and provide byte-strings only.

[StyXman]

But that would be against your policy of handling as much as possible as unicode (if I correctly understood #312)...

[StyXman]

BTW, that fixed my particular problem, but I guess you don't want the PR just yet...

[Byron]

But how would you want to produce proper unicode strings if the encoding is unclear ? It's unsafe to try it, which is showing in this example.
The truth is that I am not entirely sure how git itself handles encodings, and it might be that GitPython actually went down a wrong path by trying to just decode textual data as UTF-8. The latter works most of the time, but that's not really good enough.

Maybe a suitable solution would be to allow the client to set the decode-behaviour on a per-repository basis to control whether .decode(defenc, 'ignore') is acceptable.

Doing this sounds like quite some work - and as it stands, the unicode handling in GitPython seems flawed by design :(.

[StyXman]

I think git just doesn't handle encoding at all. In any case, any free form byte sequences (strings) are strings for user consumption: tag names, logs comments, etc. Even filenames are, I'm sure, not converted in any way. In fact, most (Unix/Linux) filesystems know nothing about encoding: it's possible to handle filenames encoded in one encoding in a system using another encoding, simply because filenames are treated as byte sequences with no specific meaning or encoding.

[CepGamer]

I have encountered similar problem - when invoking diff on a file that contains wrong utf8 sequence in this locale, GitPython fails with UnicodeDecodeError. Backtrace follows:

File "/usr/lib/python2.7/site-packages/gitupstream/gitupstream.py", line 175, in update
diff = self._repo.git.diff('--full-index', self._mainline, self._rebased)
File "/usr/lib/python2.7/site-packages/git/cmd.py", line 431, in
return lambda _args, *_kwargs: self._call_process(name, _args, *_kwargs)
File "/usr/lib/python2.7/site-packages/git/cmd.py", line 802, in _call_process
return self.execute(make_call(), **_kwargs)
File "/usr/lib/python2.7/site-packages/git/cmd.py", line 610, in execute
stdout_value = stdout_value.decode(defenc)
File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 225: invalid continuation byte

Will this issue include my error or I need to create another one? Maybe you could help me with the solution?

[Byron]

@StyXman You are totally right. As stated previously, fixing this in GitPython may be a breaking change to some, as bytes would be returned instead of unicode. This make me somewhat reluctant to attempt such a change, but I should check how much is actually affected.

@CepGamer You can pass the stdout_as_string=False keyword argument when executing .git.diff (i.e. .git.diff(..., stdout_as_string=False)), or use GitPython's own diffing facilities.

[maikelsteneker]

I believe I ran into a similar issue. When querying the commit message for a commit, the following exception is thrown:

ERROR:git.objects.commit:Failed to decode message '...' using encoding UTF-8
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/git/objects/commit.py", line 500, in _deserialize
    self.message = self.message.decode(self.encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf8 in position 126: invalid start byte

Unfortunately, I cannot share the exact commit message or the repository. I did not succeed in reproducing it in a test repository.
Perhaps an option would be to provide an option to disable decoding?