dump and manual restore causes broken external auth sources.

[TwoTwenty]

• Gitea version (or commit ref): 1.14.6
• Git version:
• Operating system: ubuntu 2004
• Database (use [x]):
  • PostgreSQL
  • [ x] MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • [x ] No

...

using gitea dump to backup my system then restored it I realized that all the external authentication is broken.

During the restore process I deleted the old DB and restored it the with the command provided in the docs.
mysql --default-character-set=utf8mb4 -u$USER -p$PASS $DATABASE <gitea-db.sql

now when users from external auth try to login they get error 500 and also when trying to edit the sources from my local admin account they also error 500.

This is reproducible with my setup but I have not tried to setup external auth with a vanilla install. please let me know if more details are needed.

[zeripath]

We need logs.

[TwoTwenty]

When I try to log a user in that belongs to the authentication source

2021/08/26 12:39:45 models/oauth2.go:89:GetActiveOAuth2ProviderLoginSources() [I] [SQL] SELECT `id`, `type`, `name`, `is_actived`, `is_sync_enabled`, `cfg`, `created_unix`, `updated_unix` FROM `login_source` WHERE (is_actived = ? and type = ?) [true 6] - 1.089409ms
2021/08/26 12:39:45 ...dels/login_source.go:390:ActiveLoginSources() [I] [SQL] SELECT `id`, `type`, `name`, `is_actived`, `is_sync_enabled`, `cfg`, `created_unix`, `updated_unix` FROM `login_source` WHERE (is_actived = ? and type = ?) [true 7] - 750.341µs
2021/08/26 12:39:45 ...rm.io/xorm/engine.go:1226:Get() [I] [SQL] SELECT `id`, `lower_name`, `name`, `full_name`, `email`, `keep_email_private`, `email_notifications_preference`, `passwd`, `passwd_hash_algo`, `must_change_password`, `login_type`, `login_source`, `login_name`, `type`, `location`, `website`, `rands`, `salt`, `language`, `description`, `created_unix`, `updated_unix`, `last_login_unix`, `last_repo_visibility`, `max_repo_creation`, `is_active`, `is_admin`, `is_restricted`, `allow_git_hook`, `allow_import_local`, `allow_create_organization`, `prohibit_login`, `avatar`, `avatar_email`, `use_custom_avatar`, `num_followers`, `num_following`, `num_stars`, `num_repos`, `num_teams`, `num_members`, `visibility`, `repo_admin_change_team_access`, `diff_view_style`, `theme`, `keep_activity_private` FROM `user` WHERE `lower_name`=? LIMIT 1 [mgregoir] - 1.016018ms
2021/08/26 12:39:45 ...dels/login_source.go:844:UserSignIn() [I] [SQL] SELECT `id`, `type`, `name`, `is_actived`, `is_sync_enabled`, `cfg`, `created_unix`, `updated_unix` FROM `login_source` WHERE `id`=? LIMIT 1 [5] - 736.54µs
2021/08/26 12:39:45 ...uters/routes/base.go:153:1() [E] PANIC: runtime error: invalid memory address or nil pointer dereference
	/usr/local/go/src/runtime/panic.go:212 (0x43de1a)
	/usr/local/go/src/runtime/signal_unix.go:734 (0x457c72)
	/source/modules/auth/ldap/ldap.go:149 (0x11b2c37)
	/source/modules/auth/ldap/ldap.go:239 (0x11b3e78)
	/source/models/login_source.go:502 (0x1473b38)
	/source/models/login_source.go:765 (0x14756ba)
	/source/models/login_source.go:851 (0x1475f7b)
	/source/routers/user/auth.go:175 (0x1fac1e9)
	/source/modules/web/route.go:64 (0x1f894fb)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/mux.go:436 (0x1b3ea0a)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/web/route.go:103 (0x1f89bd6)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/web/route.go:103 (0x1f89bd6)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/web/route.go:103 (0x1f89bd6)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/web/route.go:103 (0x1f89bd6)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/middleware/get_head.go:37 (0x21bba21)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/context/context.go:704 (0x1b5e0a1)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/gitea.com/go-chi/captcha/captcha.go:245 (0x15d0073)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/base.go:87 (0x21c2ea1)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/base.go:87 (0x21c2ea1)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/public/public.go:86 (0x13fa6a7)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/public/public.go:86 (0x13fa6a7)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/base.go:199 (0x21c4bb0)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/gitea.com/go-chi/session/session.go:256 (0x154d82e)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/mux.go:70 (0x1b3c56a)
	/source/vendor/github.com/go-chi/chi/mux.go:311 (0x1b4295b)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/mux.go:436 (0x1b3ea0a)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/web.go:107 (0x21c5ffd)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/base.go:38 (0x21c1c3b)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/middleware/strip.go:30 (0x21bc327)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/chi-middleware/proxy/middleware.go:37 (0x21b798e)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/web.go:63 (0x21c5b3c)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/mux.go:87 (0x1b3c2f0)
	/source/modules/web/route.go:298 (0x1f88a33)
	/source/vendor/github.com/gorilla/context/context.go:141 (0x11b9793)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/usr/local/go/src/net/http/server.go:2867 (0x7c1542)
	/usr/local/go/src/net/http/server.go:1932 (0x7bca6c)
	/usr/local/go/src/runtime/asm_amd64.s:1371 (0x47a4a0)
	
2021/08/26 12:39:45 ...les/public/public.go:165:handle() [I] [Static] Serving /css/index.css
2021/08/26 12:39:45 ...les/public/public.go:165:handle() [I] [Static] Serving /css/theme-arc-green.css
2021/08/26 12:39:45 ...les/public/public.go:165:handle() [I] [Static] Serving /js/index.js
2021/08/26 12:39:45 ...les/public/public.go:165:handle() [I] [Static] Serving /img/500.png
2021/08/26 12:39:45 ...les/public/public.go:165:handle() [I] [Static] Serving /img/logo.svg
2021/08/26 12:39:45 ...s/issue_stopwatch.go:49:GetUserStopwatches() [I] [SQL] SELECT `id`, `issue_id`, `user_id`, `created_unix` FROM `stopwatch` WHERE (stopwatch.user_id = ?) [71] - 982.277µs
2021/08/26 12:39:46 ...les/public/public.go:165:handle() [I] [Static] Serving /serviceworker.js
2021/08/26 12:39:49 ...s/issue_stopwatch.go:49:GetUserStopwatches() [I] [SQL] SELECT `id`, `issue_id`, `user_id`, `created_unix` FROM `stopwatch` WHERE (stopwatch.user_id = ?) [14] - 2.85ms

When trying to edit the authentication source

2021/08/26 12:43:48 models/user.go:1340:getUserByID() [I] [SQL] SELECT `id`, `lower_name`, `name`, `full_name`, `email`, `keep_email_private`, `email_notifications_preference`, `passwd`, `passwd_hash_algo`, `must_change_password`, `login_type`, `login_source`, `login_name`, `type`, `location`, `website`, `rands`, `salt`, `language`, `description`, `created_unix`, `updated_unix`, `last_login_unix`, `last_repo_visibility`, `max_repo_creation`, `is_active`, `is_admin`, `is_restricted`, `allow_git_hook`, `allow_import_local`, `allow_create_organization`, `prohibit_login`, `avatar`, `avatar_email`, `use_custom_avatar`, `num_followers`, `num_following`, `num_stars`, `num_repos`, `num_teams`, `num_members`, `visibility`, `repo_admin_change_team_access`, `diff_view_style`, `theme`, `keep_activity_private` FROM `user` WHERE `id`=? LIMIT 1 [71] - 2.453225ms
2021/08/26 12:43:48 ...s/issue_stopwatch.go:67:HasUserStopwatch() [I] [SQL] SELECT `id`, `issue_id`, `user_id`, `created_unix` FROM `stopwatch` WHERE (user_id = ?) LIMIT 1 [71] - 751.128µs
2021/08/26 12:43:48 ...dels/login_source.go:413:GetLoginSourceByID() [I] [SQL] SELECT `id`, `type`, `name`, `is_actived`, `is_sync_enabled`, `cfg`, `created_unix`, `updated_unix` FROM `login_source` WHERE `id`=? LIMIT 1 [5] - 694.52µs
2021/08/26 12:43:48 ...uters/routes/base.go:153:1() [E] PANIC: runtime error: invalid memory address or nil pointer dereference
	/usr/local/go/src/runtime/panic.go:212 (0x43de1a)
	/usr/local/go/src/runtime/signal_unix.go:734 (0x457c72)
	/source/models/login_source.go:288 (0x20632ba)
	/source/routers/admin/auths.go:309 (0x20632cb)
	/source/modules/web/route.go:64 (0x1f894fb)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/mux.go:436 (0x1b3ea0a)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/web/route.go:103 (0x1f89bd6)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/web/route.go:103 (0x1f89bd6)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/web/route.go:103 (0x1f89bd6)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/web/route.go:103 (0x1f89bd6)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/middleware/get_head.go:37 (0x21bba21)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/context/context.go:704 (0x1b5e0a1)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/gitea.com/go-chi/captcha/captcha.go:245 (0x15d0073)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/base.go:94 (0x21c2fc1)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/base.go:94 (0x21c2fc1)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/public/public.go:86 (0x13fa6a7)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/modules/public/public.go:86 (0x13fa6a7)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/base.go:199 (0x21c4bb0)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/gitea.com/go-chi/session/session.go:256 (0x154d82e)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/mux.go:70 (0x1b3c56a)
	/source/vendor/github.com/go-chi/chi/mux.go:311 (0x1b4295b)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/mux.go:436 (0x1b3ea0a)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/web.go:107 (0x21c5ffd)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/base.go:38 (0x21c1c3b)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/middleware/strip.go:30 (0x21bc327)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/chi-middleware/proxy/middleware.go:37 (0x21b798e)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/routers/routes/web.go:63 (0x21c5b3c)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/source/vendor/github.com/go-chi/chi/mux.go:87 (0x1b3c2f0)
	/source/modules/web/route.go:298 (0x1f88a33)
	/source/vendor/github.com/gorilla/context/context.go:141 (0x11b9793)
	/usr/local/go/src/net/http/server.go:2049 (0x7bdf83)
	/usr/local/go/src/net/http/server.go:2867 (0x7c1542)
	/usr/local/go/src/net/http/server.go:1932 (0x7bca6c)
	/usr/local/go/src/runtime/asm_amd64.s:1371 (0x47a4a0)
	
2021/08/26 12:43:48 models/user.go:1340:getUserByID() [I] [SQL] SELECT `id`, `lower_name`, `name`, `full_name`, `email`, `keep_email_private`, `email_notifications_preference`, `passwd`, `passwd_hash_algo`, `must_change_password`, `login_type`, `login_source`, `login_name`, `type`, `location`, `website`, `rands`, `salt`, `language`, `description`, `created_unix`, `updated_unix`, `last_login_unix`, `last_repo_visibility`, `max_repo_creation`, `is_active`, `is_admin`, `is_restricted`, `allow_git_hook`, `allow_import_local`, `allow_create_organization`, `prohibit_login`, `avatar`, `avatar_email`, `use_custom_avatar`, `num_followers`, `num_following`, `num_stars`, `num_repos`, `num_teams`, `num_members`, `visibility`, `repo_admin_change_team_access`, `diff_view_style`, `theme`, `keep_activity_private` FROM `user` WHERE `id`=? LIMIT 1 [71] - 815.1µs
2021/08/26 12:43:49 ...s/issue_stopwatch.go:49:GetUserStopwatches() [I] [SQL] SELECT `id`, `issue_id`, `user_id`, `created_unix` FROM `stopwatch` WHERE (stopwatch.user_id = ?) [14] - 1.015585ms
2021/08/26 12:43:49 models/user.go:1340:getUserByID() [I] [SQL] SELECT `id`, `lower_name`, `name`, `full_name`, `email`, `keep_email_private`, `email_notifications_preference`, `passwd`, `passwd_hash_algo`, `must_change_password`, `login_type`, `login_source`, `login_name`, `type`, `location`, `website`, `rands`, `salt`, `language`, `description`, `created_unix`, `updated_unix`, `last_login_unix`, `last_repo_visibility`, `max_repo_creation`, `is_active`, `is_admin`, `is_restricted`, `allow_git_hook`, `allow_import_local`, `allow_create_organization`, `prohibit_login`, `avatar`, `avatar_email`, `use_custom_avatar`, `num_followers`, `num_following`, `num_stars`, `num_repos`, `num_teams`, `num_members`, `visibility`, `repo_admin_change_team_access`, `diff_view_style`, `theme`, `keep_activity_private` FROM `user` WHERE `id`=? LIMIT 1 [71] - 1.176479ms
2021/08/26 12:43:49 ...s/issue_stopwatch.go:67:HasUserStopwatch() [I] [SQL] SELECT `id`, `issue_id`, `user_id`, `created_unix` FROM `stopwatch` WHERE (user_id = ?) LIMIT 1 [71] - 712.921µs
2021/08/26 12:43:51 ...les/public/public.go:165:handle() [I] [Static] Serving /serviceworker.js
2021/08/26 12:43:54 ...s/issue_stopwatch.go:49:GetUserStopwatches() [I] [SQL] SELECT `id`, `issue_id`, `user_id`, `created_unix` FROM `stopwatch` WHERE (stopwatch.user_id = ?) [81] - 2.579257ms
2021/08/26 12:43:54 ...dels/notification.go:734:GetUIDsAndNotificationCounts() [I] [SQL] SELECT user_id, count(*) AS count FROM notification WHERE user_id IN (SELECT user_id FROM notification WHERE updated_unix >= ? AND updated_unix < ?) AND status = ? GROUP BY user_id [1630007022 1630007032 1] - 1.207019ms
2021/08/26 12:43:59 ...s/issue_stopwatch.go:49:GetUserStopwatches() [I] [SQL] SELECT `id`, `issue_id`, `user_id`, `created_unix` FROM `stopwatch` WHERE (stopwatch.user_id = ?) [14] - 2.474893ms
2021/08/26 12:43:59 ...s/issue_stopwatch.go:49:GetUserStopwatches() [I] [SQL] SELECT `id`, `issue_id`, `user_id`, `created_unix` FROM `stopwatch` WHERE (stopwatch.user_id = ?) [71] - 793.313µs

[zeripath]

Now that's more helpful.

It would be useful to examine the login_source table in particular row 5.

Likely what has happened is that the cfg field is empty or corrupted somehow.

You may be better off deleting the login_source and recreating it.

[TwoTwenty]

I cannot edit the login source to delete it. but I can do it from sql... would it be ok to delete them, recreate them and then update their IDs? I have many users and would like it all to jive.

In fact I have 8 different LDAP sources ... and they are all broken... 5 is just the one that user was authenticating against which is broken.

I could share the data privately in row 5 if you like just not on git... do you use element/matrix ?

I am @two:gnu.fun

Not certain what you mean by cfg field ? potentially I am missing it? since I have been using gitea since it forked from gogs.
by cfg field do you mean in the app.ini?

Pretty sure I can correct this without help but I would be glad to share the data, if you think this might be an issue... if not on element I can clean the data of private bits and share it here?

[zeripath]

I cannot edit the login source to delete it. but I can do it from sql... would it be ok to delete them, recreate them and then update their IDs? I have many users and would like it all to jive.

Ah you have a lot of users you want to keep...

OK. Create a new source with the all the details - mark it inactive - then in the db copy over its cfg on to the broken.

In fact I have 8 different LDAP sources ... and they are all broken... 5 is just the one that user was authenticating against which is broken.

You'll have to do the same thing.

I could share the data privately in row 5 if you like just not on git... do you use element/matrix ?

That might be useful

I am @two:gnu.fun

Not certain what you mean by cfg field ? potentially I am missing it? since I have been using gitea since it forked from gogs.
by cfg field do you mean in the app.ini?

I meant in the db

[zeripath]

Ok this is annoying because I definitely fixed an issue with this early on in 1.14.x but it appears that it came back.

The problem is that the ToDB on the login_source is not being called on dump.

https://gitea.com/xorm/xorm/pulls/1903

Fixed this but it appears that it's been refactored away at some point.

[zeripath]

OK so testing reveals that the dump works on 1.14.0, 1.14.1, and 1.14.2 and is broken on 1.14.3.

The bad commit is: ad54f00

Upgrade xorm to v1.1.0 (#15869) (#15885)

[zeripath]

However, things are correct on v1.16.0-dev-159-gcd8db3a83.

The fixing commit is: 7224cfc

Upgrade xorm to v1.2.2 (#16663)