-
-
Notifications
You must be signed in to change notification settings - Fork 108
Implementation to aggregate statistical information #12 #128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 30 commits
e3fd895
db0afac
04f645c
6ff7f21
453d28f
3669f2a
b3d39bc
f3580de
3bc50c1
dfdc16c
c93dfb8
9eb61aa
a4cb3ec
d75d2fe
8ba4506
b1a5ad1
7197558
983d828
cf033f8
70ef722
5168879
ba6fa8f
59fe612
ec719e1
0024192
b7e8213
58bfebd
ab9736d
740ca73
86b1f6c
563bd68
f44f78e
0d72302
754cf52
0366800
9adb714
d392c68
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -12,4 +12,10 @@ deploy: | |
run: | ||
- "bundle exec rails db:migrate" | ||
- "bundle exec rails dojos:update_db_by_yaml" | ||
|
||
rvm: | ||
- 2.4.0 | ||
cache: | ||
- bundler | ||
script: | ||
- RAILS_ENV=test bundle exec rake db:migrate --trace | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. travisだとデフォルトで There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 😅 |
||
- bundle exec rake |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,6 +3,9 @@ class Dojo < ApplicationRecord | |
NUM_OF_WHOLE_DOJOS = "1,250" | ||
NUM_OF_JAPAN_DOJOS = Dojo.count.to_s | ||
|
||
has_one :dojo_event_service | ||
has_many :event_history | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. おぉ😓 |
||
|
||
serialize :tags | ||
default_scope -> { order(order: :asc) } | ||
before_save { self.email = self.email.downcase } | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
class DojoEventService < ApplicationRecord | ||
belongs_to :dojo | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
class EventHistory < ApplicationRecord | ||
belongs_to :dojo | ||
|
||
validates :dojo_name, presence: true | ||
validates :service_name, presence: true, uniqueness: { scope: :event_id } | ||
validates :service_group_id, presence: true | ||
validates :event_id, presence: true | ||
validates :event_url, presence: true, uniqueness: true, allow_nil: true | ||
validates :participants, presence: true | ||
validates :evented_at, presence: true | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,5 +11,9 @@ class Application < Rails::Application | |
# Settings in config/environments/* take precedence over those specified here. | ||
# Application configuration should go into files in config/initializers | ||
# -- all .rb files in that directory are automatically loaded. | ||
|
||
# Timezone | ||
config.time_zone = 'Asia/Tokyo' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
ENV['TZ'] = 'Asia/Tokyo' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Heroku、travis-ciのほうで設定した方がいいかも? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Herokuのenvはどなたかにお願いするとして、そうしましょうか。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Heroku の env 確認しましたが、既に入ってました ;) ✅ $ heroku config
...
TZ: Asia/Tokyo There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ご確認ありがとうございます!😁 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ご確認ありがとうございます!😁 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed 0366800 |
||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
class CreateEventHistories < ActiveRecord::Migration[5.0] | ||
def change | ||
create_table :event_histories do |t| | ||
t.references :dojo, foreign_key: true, index: true, null: false | ||
t.string :dojo_name, null: false | ||
t.string :service_name, null: false | ||
t.integer :service_group_id, null: false | ||
t.integer :event_id, null: false | ||
t.string :event_url, unique: true, null: false | ||
t.integer :participants, null: false | ||
t.datetime :evented_at, null: false | ||
t.timestamps | ||
|
||
t.index [:service_name, :event_id], unique: true | ||
t.index [:evented_at, :dojo_id] | ||
end | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
class CreateDojoEventServices < ActiveRecord::Migration[5.0] | ||
def change | ||
create_table :dojo_event_services do |t| | ||
t.references :dojo, foreign_key: true, index: true, null: false | ||
t.string :name | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. NOT NULL制約ついてる There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. connpassとdoorkeeper以外で開催されているところの扱いをどうすべきか判断がつかなかった(どれくらいあるのかもわからなかった)のでなんとなく制約を付けませんでした。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
悩ましいですね🤔 NOT NULL 制約って、「もし必要になったら外す」みたいな運用だとマズイですかね? 😅 それで問題なければ、制約つけておいても良いのかなと思いました >< There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. あとから外しても問題ないなので、制約を付けるようにします。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed 9adb714 |
||
t.string :url | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. つかわれてなさそう...? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. connpass / doorkeeperの各dojoごとのurlを入れようかと思って用意しています。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. なるほど、イベントページのトップページへのリンク等表示したいこともありそうなので、あったほうが良さそうですね。 |
||
t.integer :group_id | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. serviceによっては There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. そうですね、同感です。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed 9adb714 |
||
t.timestamps | ||
end | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
module Statistics; end | ||
|
||
require_relative 'statistics/client' | ||
require_relative 'statistics/aggregation' |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,66 @@ | ||
module Statistics | ||
class Aggregation | ||
class << self | ||
def run(date:) | ||
cnps_dojos = Dojo.joins(:dojo_event_service).where(dojo_event_services: { name: 'connpass' }).to_a | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ここで There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ご指摘のとおりですね。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed d392c68 |
||
drkp_dojos = Dojo.joins(:dojo_event_service).where(dojo_event_services: { name: 'doorkeeper' }).to_a | ||
|
||
Connpass.run(cnps_dojos, date) | ||
Doorkeeper.run(drkp_dojos, date) | ||
end | ||
end | ||
|
||
class Connpass | ||
class << self | ||
def run(dojos, date) | ||
cnps = Client::Connpass.new | ||
params = { | ||
yyyymm: "#{date.year}#{date.month}" | ||
} | ||
|
||
dojos.each do |dojo| | ||
cnps.fetch_events(params.merge(series_id: dojo.dojo_event_service.group_id)).each do |e| | ||
next unless e.dig('series', 'id') == dojo.dojo_event_service.group_id | ||
|
||
EventHistory.create!(dojo_id: dojo.id, | ||
dojo_name: dojo.name, | ||
service_name: dojo.dojo_event_service.name, | ||
service_group_id: dojo.dojo_event_service.group_id, | ||
event_id: e['event_id'], | ||
event_url: e['event_url'], | ||
participants: e['accepted'], | ||
evented_at: Time.zone.parse(e['started_at'])) | ||
end | ||
end | ||
end | ||
end | ||
end | ||
|
||
class Doorkeeper | ||
class << self | ||
def run(dojos, date) | ||
drkp = Client::Doorkeeper.new | ||
params = { | ||
since_at: date.beginning_of_month, | ||
until_at: date.end_of_month | ||
} | ||
|
||
dojos.each do |dojo| | ||
drkp.fetch_events(params.merge(group_id: dojo.dojo_event_service.group_id)).each do |e| | ||
next unless e['group'] == dojo.dojo_event_service.group_id | ||
|
||
EventHistory.create!(dojo_id: dojo.id, | ||
dojo_name: dojo.name, | ||
service_name: dojo.dojo_event_service.name, | ||
service_group_id: dojo.dojo_event_service.group_id, | ||
event_id: e['id'], | ||
event_url: e['public_url'], | ||
participants: e['participants'], | ||
evented_at: Time.zone.parse(e['starts_at'])) | ||
end | ||
end | ||
end | ||
end | ||
end | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,105 @@ | ||
module Statistics | ||
class Client | ||
class_attribute :debug | ||
self.debug = false | ||
|
||
def initialize(endpoint, &block) | ||
@conn = connection_for(endpoint, &block) | ||
end | ||
|
||
def get(path, params) | ||
@conn.get(path, params).body | ||
end | ||
|
||
private | ||
|
||
def connection_for(endpoint, &block) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. こちらで使っています〜 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. あ、確かに。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixed 754cf52 |
||
Faraday.new(endpoint) do |f| | ||
f.response :logger if self.class.debug | ||
f.response :json, :content_type => /\bjson$/ | ||
f.response :raise_error | ||
|
||
yield f if block_given? | ||
|
||
f.adapter Faraday.default_adapter | ||
end | ||
end | ||
|
||
class Connpass | ||
ENDPOINT = 'https://connpass.com/api/v1'.freeze | ||
|
||
def initialize | ||
@client = Client.new(ENDPOINT) | ||
end | ||
|
||
def search(keyword:) | ||
@client.get('event/', { keyword: keyword, count: 100 }) | ||
end | ||
|
||
def fetch_events(series_id:, yyyymm: nil) | ||
params = { | ||
series_id: series_id, | ||
start: 1, | ||
count: 100 | ||
} | ||
params[:ym] = yyyymm if yyyymm | ||
events = [] | ||
|
||
loop do | ||
part = @client.get('event/', params) | ||
|
||
break if part['results_returned'].zero? | ||
|
||
events.push(*part.fetch('events')) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. こことL99、count: 100なので大丈夫そうですがevents個数がものすごく多いとスタック溢れそうですね There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. そうですねー。 |
||
|
||
break if part.size < params[:count] | ||
|
||
break if params[:start] + params[:count] > part['results_available'] | ||
|
||
params[:start] += params[:count] | ||
end | ||
|
||
events | ||
end | ||
end | ||
|
||
class Doorkeeper | ||
ENDPOINT = 'https://api.doorkeeper.jp'.freeze | ||
|
||
def initialize | ||
@client = Client.new(ENDPOINT) do |c| | ||
c.authorization(:Bearer, ENV.fetch('DOORKEEPER_API_TOKEN')) | ||
end | ||
@default_since = Time.zone.parse('2010-07-01') | ||
@default_until = Time.zone.now.end_of_day | ||
end | ||
|
||
def search(keyword:) | ||
@client.get('events', q: keyword, since: @default_since) | ||
end | ||
|
||
def fetch_events(group_id:, since_at: @default_since, until_at: @default_until) | ||
params = { | ||
page: 1, | ||
since: since_at, | ||
until: until_at | ||
} | ||
events = [] | ||
|
||
loop do | ||
part = @client.get("groups/#{group_id}/events", params) | ||
|
||
break if part.size.zero? | ||
|
||
events.push(*part.map { |e| e['event'] }) | ||
|
||
break if part.size < 25 # 25 items / 1 request | ||
|
||
params[:page] += 1 | ||
end | ||
|
||
events | ||
end | ||
end | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
require_relative '../statistics.rb' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. はい、そのとおりです〜。😌 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. なるほど 🦀 |
||
|
||
namespace :statistics do | ||
desc '月次のイベント履歴を集計します' | ||
task :aggregation, [:yyyymm] => :environment do |tasks, args| | ||
date = Time.current.prev_month.beginning_of_month | ||
if args[:yyyymm].present? | ||
date = %w(%Y%m %Y/%m %Y-%m).map do |fmt| | ||
begin | ||
Time.zone.strptime(args[:yyyymm], fmt) | ||
rescue ArgumentError | ||
end | ||
end.compact.first | ||
end | ||
|
||
raise ArgumentError, "Invalid format: #{args[:yyyymm]}" if date.nil? | ||
|
||
|
||
EventHistory.where(evented_at: date.beginning_of_month..date.end_of_month).delete_all | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. その月の分まるごと作り直す感じなんですねー。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. あまり無いとは思いますがたまにやり直したいケースがあったりするのでそのようにしました。 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. いいと思います! |
||
|
||
Statistics::Aggregation.run(date: date) | ||
end | ||
|
||
desc 'キーワードからイベント情報を検索します' | ||
task :search, [:keyword] => :environment do |tasks, args| | ||
raise ArgumentError, 'Require the keyword' if args[:keyword].nil? | ||
|
||
require 'pp' | ||
|
||
puts 'Searching Connpass' | ||
pp Statistics::Client::Connpass.new.search(keyword: args[:keyword]) | ||
|
||
puts 'Searching Doorkeeper' | ||
pp Statistics::Client::Doorkeeper.new.search(keyword: args[:keyword]) | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
require 'rails_helper' | ||
require 'statistics' | ||
|
||
RSpec.describe Statistics::Aggregation do | ||
include_context 'Use stubs for Faraday' | ||
|
||
before(:all) do | ||
Dojo.delete_all | ||
DojoEventService.delete_all | ||
EventHistory.delete_all | ||
end | ||
|
||
after do | ||
Dojo.delete_all | ||
DojoEventService.delete_all | ||
EventHistory.delete_all | ||
end | ||
|
||
describe '.run' do | ||
before do | ||
d1 = Dojo.create(name: 'Dojo1', email: 'info@dojo1.com', description: 'CoderDojo1', tags: %w(CoderDojo1), url: 'https://dojo1.com') | ||
d2 = Dojo.create(name: 'Dojo2', email: 'info@dojo2.com', description: 'CoderDojo2', tags: %w(CoderDojo2), url: 'https://dojo2.com') | ||
DojoEventService.create(dojo_id: d1.id, name: 'connpass', group_id: 9876) | ||
DojoEventService.create(dojo_id: d2.id, name: 'doorkeeper', group_id: 5555) | ||
end | ||
|
||
subject { Statistics::Aggregation.run(date: Time.current) } | ||
|
||
it do | ||
expect{ subject }.to change{EventHistory.count}.from(0).to(2) | ||
end | ||
end | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
travisのログみた感じだと
.ruby-version
があればここ指定する必要はないみたいです。https://travis-ci.org/coderdojo-japan/coderdojo.jp/jobs/260468747
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
なるほど。
fixed 563bd68
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
複数バージョンのRubyに対してテスト実行したいときは便利そうですが今は使ってるRuby1つしかないので消しちゃって大丈夫そうですね 👍