はじめに
こちらも使ってみてからだいぶたってるのでうる覚えですが自分用にメモっておきます！
ということで、td-agent(fluentd)関連の記事としてTreasure Dataを使う・使ってみたメモの⑤です
関連記事はこちら
→① - とにかく使ってみる
→② - Application Supportを使ってみる(fluent-logger-perl)
→③ - Application Supportを使ってみる(fluent-logger-php)
→④ - tdコマンドでいろいろやってみる
補足
①～④でいくつかやりましたが、
公式のこちらを見ればだいたい使い方は書いてあります
http://docs.treasuredata.com/articles/rest-api#prerequisites
そんなにテクニカルでも有用でもないカモですがこちらにメモまとめをメモっておきました →td-agent(fluentd)を本番サ―ビスで小さく使ってみたメモ－まとめ
流れ
1. apikeyの取得
ここは自分の記事をまんま流用です
→td-agent(fluentd)の運用でTreasure Data使ってみるメモ② - Application Supportを使ってみる(fluent-logger-perl)
→td-agent(fluentd)の運用でTreasure Data使ってみたメモ③ - Application Supportを使ってみる(fluent-logger-php)
APIを叩くにはapikeyを書く必要があるのでapikeyを取得します。
apikey自体は取得というよりはTreasure Dataにsign upしていれば既にできています
→sign upについてはこちら
ってコトでhttps://console.treasuredata.com/にログインした状態で
アカウント名クリック > API Keysのランにパスワードを入力 > Show Keysでapikeyが表示されます
2.api叩いてTreasure DataのデータをGETしてみる
ってことでさっそく
基本
コマンドラインからはこんな感じで叩きます
# curl -H "AUTHORIZATION: TD1 {取得したapikey}" "http://api.treasuredata.com/v3/{apiに応じたuri}"
databaseの一覧を取得
# curl -H "AUTHORIZATION: TD1 {取得したapikey}"\
"http://api.treasuredata.com/v3/database/list"
----結果----
{"databases":[{"name":"hoge_db","count":5000,"created_at":"2014-05-20 12:50:31 UTC","updated_at":"2014-05-20 12:50:31 UTC","organization":null},{"name":"sample_db","count":5000,"created_at":"2014-05-16 11:53:52 UTC","updated_at":"2014-05-16 11:53:52 UTC","organization":null},{"name":"test_db","count":6,"created_at":"2014-05-19 10:55:12 UTC","updated_at":"2014-05-19 10:55:12 UTC","organization":null}]}%
tableの一覧を取得
# curl -H "AUTHORIZATION: TD1 {取得したapikey}"\
"http://api.treasuredata.com/v3/table/list/sample_db"
----結果----
{"tables":[{"id":127770,"name":"test_table","estimated_storage_size":0,"counter_updated_at":null,"last_log_timestamp":null,"expire_days":null,"type":"log","count":0,"created_at":"2014-05-16 12:03:15 UTC","updated_at":"2014-05-16 12:03:15 UTC","schema":"[]"},{"id":127765,"name":"www_access","estimated_storage_size":0,"counter_updated_at":"2014-05-16T11:53:53Z","last_log_timestamp":"2013-09-07T01:13:45Z","expire_days":null,"type":"log","count":5000,"created_at":"2014-05-16 11:53:52 UTC","updated_at":"2014-05-16 11:53:53 UTC","schema":"[[\"host\",\"string\"],[\"path\",\"string\"],[\"method\",\"string\"],[\"referer\",\"string\"],[\"code\",\"long\"],[\"agent\",\"string\"],[\"user\",\"string\"],[\"size\",\"long\"]]"}],"database":"sample_db"}%
queryの結果(jobの結果)を取得
jobのidについてはTreasureDataのconsoleにログインし、jobsタブで確認できます
# curl -H "AUTHORIZATION: TD1 {取得したapikey}"\
"http://api.treasuredata.com/v3/job/result/10612780?format=csv"
----結果----
2013-09-06,/category/software,200
2013-09-06,/category/electronics,200
2013-09-06,/category/jewelry,200
# curl -H "AUTHORIZATION: TD1 {取得したapikey}"\
"http://api.treasuredata.com/v3/job/result/10612780?format=json"
----結果----
["2013-09-06","/category/software","200"]
["2013-09-06","/category/electronics","200"]
["2013-09-06","/category/jewelry","200"]
3. phpでjob実行 -> phpでapi叩いてjob結果表示してみる
phpライブラリのインストール
こちらを見ればサンプルがあります
→https://github.com/crocos/php-treasure-data
crocosさんさすがです！
# sudo pear channel-discover openpear.org Channel "openpear.org" is already initialized ※エラー # sudo pear install openpear/TreasureData Failed to download openpear/TreasureData within preferred state "stable", latest release is version 0.1.1, stability "alpha", use "channel://openpear.org/TreasureData-0.1.1" to install install failed
現時点(2014/05/21)では安定版ではないようでそのままやるとエラーになってしまうので
アルファ(もしくはベータ)の指定をしてインストールします
# sudo pear install openpear/TreasureData-alpha downloading TreasureData-0.1.1.tgz ... Starting to download TreasureData-0.1.1.tgz (2,890 bytes) ....done: 2,890 bytes install ok: channel://openpear.org/TreasureData-0.1.1
phpスクリプト書いて試す
一応インストールされた場所を確認
# locate TreasureData /usr/local/php/lib/php/TreasureData /usr/local/php/lib/php/TreasureData/API /usr/local/php/lib/php/TreasureData/API.php /usr/local/php/lib/php/TreasureData/Autoload.php /usr/local/php/lib/php/TreasureData/Exception.php /usr/local/php/lib/php/TreasureData/API/Base.php /usr/local/php/lib/php/TreasureData/API/Job.php
ってことで確認をもとにスクリプト書く。 書いたのはサンプルまんまですがこんな感じ
treasuredata_php_library_test.php
実行中にjob確認してみる
結果
# php treasuredata_php_library_test.php [DEBUG] API request to http://api.treasure-data.com/v3/job/issue/hive/sample_db?query=SELECT+v%5B%27code%27%5D+as+code%2C+count%281%29+FROM+www_access+GROUP+BY+v%5B%27code%27%5D [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]queued [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]running [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]running [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]running [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]running [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]running [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]running [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]running [DEBUG] API request to http://api.treasure-data.com/v3/job/show/10638615 [STATUS]success [DEBUG] API request to http://api.treasure-data.com/v3/job/result/10638615 404,17 500,2 200,4981
まとめ
ってことでREST APIを使って、TreasureDataに貯めたデータをいろいろできそうです！
