Solaris10でfluent-agent-liteを動かす
Solaris10な環境でfluent-agent-liteを動かすためのメモです。
目的がDBサーバ向けのログ監視向けのログ転送用で
という理由でfluent-agent-liteでとりあえずサクっと作れたら嬉しいという感じです。
構築方針を「1ディレクトリに必要なものを全部ぶち込む」とするため、install.sh などもちょっと手を入れます。
●明示的にダウンロードするもの
- Perl (5.16系では起動時にエラーが起きてしまったので、動く中での新しめの 5.14系で行きます
- cpanm (fluent-agent-lite の bin/install.sh で落としますが、curl になっているので事前に落としておきます
- coreutils & xz (xz は coreutils のアーカイブ展開用、coreutils内で必要なのはtailのみ
- fluent-agent-lite
●大雑把な作り方
・ダウンロード Solaris10はcurlはデフォルトでは無いんですよね。 wegt http://www.cpan.org/src/5.0/perl-5.14.4.tar.gz wget http://ftp.gnu.org/gnu/coreutils/coreutils-8.21.tar.xz wget http://tukaani.org/xz/xz-5.0.5.tar.gz wget --no-check-certificate http://xrl.us/cpanm wget --no-check-certificate https://github.com/tagomoris/fluent-agent-lite/archive/master.zip mv master fluent-agent-lite.v1.0.zip ・環境変数 export PATH=/usr/sfw/bin:/usr/sfw/sbin:/usr/xpg4/bin:/usr/ccs/bin:/usr/sbin:/usr/bin ・perl gtar xzf perl-5.14.4.tar.gz cd perl-5.14.4 ./Configure -Dcc=gcc -des -Dprefix=/usr/local/fluent-agent-lite-v1.0 gmake gmake install ・xz coreutils-8.21.tar.xzが展開できればいいので、installはしません。してもいいけれど。 gtar xzf xz-5.0.5.tar.gz cd xz-5.0.5 ./configure gmake LD_LIBRARY_PATH=./src/liblzma/.libs/ ./src/xz/.libs/xz -d ../coreutils-8.21.tar.xz ・coreutils 欲しいのはtailだけなのでmake後単独でコピります。 make checkするとFAILしたりするので、微妙なものを入れないという意味もあります。tail部分はクリアしてる様子。 gtar xf coreutils-8.21.tar chown -R mhnsr:other coreutils-8.21 cd coreutils-8.21 su mhnsr ./configure --prefix=/usr/local/fluent-agent-lite-v1.0 gmake exit cp -p src/tail /usr/local/fluent-agent-lite-v1.0/bin/ chown root:root /usr/local/fluent-agent-lite-v1.0/bin/tail ・cpanm chmod +x cpanm cp -p cpanm /usr/local/fluent-agent-lite-v1.0/bin/ ・fluent-agent-lite unzip fluent-agent-lite.v1.0.zip cd fluent-agent-lite-master/ cp -p bin/install.sh bin/install.sh.org vi bin/install.sh cp -p lib/Fluent/AgentLite.pm lib/Fluent/AgentLite.pm.org vi lib/Fluent/AgentLite.pm cp -p package/fluent-agent-lite.init package/fluent-agent-lite.init.org vi package/fluent-agent-lite.init bin/install.sh ・起動 cd /usr/local/fluent-agent-lite-v1.0/ vi fluent-agent-lite.conf ./fluent-agent-lite start
●diff周り
install.sh、fluent-agent-lite.init の変更のやり方がだめだめです。とりあえず動かすことを優先した結果。
もうちょいうまく直せて、Solaris11系辺りでも問題なく動くようなら修正の提案してみたいところです。
#diff -u bin/install.sh bin/install.sh.org --- bin/install.sh Fri Dec 13 02:20:29 2013 +++ bin/install.sh.org Wed Dec 11 16:24:40 2013 @@ -15,7 +15,7 @@ if [ "x"$PREFIX = "x" ]; then PREFIX= fi -INSTALLDIR=/usr/local/fluent-agent-lite-v1.0 +INSTALLDIR=$PREFIX/usr/local/fluent-agent-lite if [ "x"$PERL_PATH = "x" ]; then PERL_PATH="perl" @@ -31,17 +31,17 @@ cd $INSTALLDIR -#curl -s -L http://xrl.us/cpanm > $INSTALLDIR/bin/cpanm -#chmod +x $INSTALLDIR/bin/cpanm +curl -s -L http://xrl.us/cpanm > $INSTALLDIR/bin/cpanm +chmod +x $INSTALLDIR/bin/cpanm $PERL_PATH $INSTALLDIR/bin/cpanm -n -lextlib inc::Module::Install $PERL_PATH $INSTALLDIR/bin/cpanm -n -lextlib --reinstall --installdeps . cd $SOURCEDIR -#mkdir -p $PREFIX/etc/init.d -cp package/fluent-agent-lite.init $INSTALLDIR/fluent-agent-lite -chmod +x $INSTALLDIR/fluent-agent-lite +mkdir -p $PREFIX/etc/init.d +cp package/fluent-agent-lite.init $PREFIX/etc/init.d/fluent-agent-lite +chmod +x $PREFIX/etc/init.d/fluent-agent-lite -if [ ! -f $INSTALLDIR/fluent-agent-lite.conf -o "x"$CLEAN = x"y" ]; then - cp package/fluent-agent-lite.conf $INSTALLDIR/fluent-agent-lite.conf +if [ ! -f $PREFIX/etc/fluent-agent-lite.conf -o "x"$CLEAN = x"y" ]; then + cp package/fluent-agent-lite.conf $PREFIX/etc/fluent-agent-lite.conf fi diff -u lib/Fluent/AgentLite.pm lib/Fluent/AgentLite.pm.org --- lib/Fluent/AgentLite.pm Fri Dec 13 02:22:23 2013 +++ lib/Fluent/AgentLite.pm.org Wed Dec 11 16:24:40 2013 @@ -58,7 +58,7 @@ *pack_drainlog = *pack_drainlog_json; } - srand (time ^ $PID ^ unpack("%L*", `ps -ef | gzip`)); + srand (time ^ $PID ^ unpack("%L*", `ps axww | gzip`)); bless $self, $this; } diff -u package/fluent-agent-lite.init package/fluent-agent-lite.init.org --- package/fluent-agent-lite.init Fri Dec 13 02:26:42 2013 +++ package/fluent-agent-lite.init.org Wed Dec 11 16:24:40 2013 @@ -7,7 +7,7 @@ prog="fluent-agent-lite" -scriptpath="/usr/local/fluent-agent-lite-v1.0/bin/"$prog +scriptpath="/usr/local/fluent-agent-lite/bin/"$prog if [ ! -x $scriptpath ]; then echo "Not installed properly" @@ -14,7 +14,7 @@ exit 1 fi -CONFIG_FILE=/usr/local/fluent-agent-lite-v1.0/fluent-agent-lite.conf +CONFIG_FILE=/etc/fluent-agent-lite.conf if [ ! -r $CONFIG_FILE ]; then echo "Config file does not exists: "$CONFIG_FILE exit 1 @@ -21,7 +21,7 @@ fi source $CONFIG_FILE -PID_FILE=/usr/local/fluent-agent-lite-v1.0/fluent-agent-lite.pid +PID_FILE=/var/run/fluent-agent-lite.pid RETVAL=0 @@ -47,7 +47,7 @@ if [ "x"$FORCE_START = "x" -o "x"$FORCE_START = "xno" -o "x"$FORCE_START = "x0" -o "x"$FORCE_START = "xfalse" ]; then for (( i = 0; i < $lines; i++ )); do lineno=$((i + 1)) - line=$(echo "$LOGS" | grep -v '^#' | tail -$lineno | head -n 1) + line=$(echo "$LOGS" | grep -v '^#' | tail -n $lineno | head -n 1) path=$(echo $line | awk '{print $2;}') if [ ! -r "$path" ]; then echo "$path does not exist or is not readable." @@ -238,7 +238,7 @@ prepare_build lineno=$((pos + 1)) - line=$(echo "$LOGS" | grep -v '^#' | tail -$lineno | head -n 1) + line=$(echo "$LOGS" | grep -v '^#' | tail -n $lineno | head -n 1) tag=$(echo $line | awk '{print $1;}') tag=$TAG_PREFIX_STRING$tag @@ -268,7 +268,6 @@ *) echo $"Usage: $prog {start|stop|restart|reload|status}" exit 1 - ;; esac exit $RETVAL
●install.sh実行時イメージ
#bin/install.sh --> Working on inc::Module::Install Fetching http://www.cpan.org/authors/id/A/AD/ADAMK/Module-Install-1.06.tar.gz ... OK ==> Found dependencies: YAML::Tiny --> Working on YAML::Tiny Fetching http://www.cpan.org/authors/id/E/ET/ETHER/YAML-Tiny-1.56.tar.gz ... OK ==> Found dependencies: Module::Build::Tiny --> Working on Module::Build::Tiny Fetching http://www.cpan.org/authors/id/L/LE/LEONT/Module-Build-Tiny-0.030.tar.gz ... OK ==> Found dependencies: TAP::Harness::Env, CPAN::Meta, ExtUtils::Helpers, ExtUtils::InstallPaths, TAP::Harness, JSON::PP, ExtUtils::Config --> Working on TAP::Harness::Env Fetching http://www.cpan.org/authors/id/L/LE/LEONT/Test-Harness-3.30.tar.gz ... OK Configuring Test-Harness-3.30 ... OK Building Test-Harness-3.30 ... OK Successfully installed Test-Harness-3.30 --> Working on CPAN::Meta Fetching http://www.cpan.org/authors/id/D/DA/DAGOLDEN/CPAN-Meta-2.133380.tar.gz ... OK Configuring CPAN-Meta-2.133380 ... OK ==> Found dependencies: List::Util, Parse::CPAN::Meta, version, CPAN::Meta::YAML, JSON::PP, CPAN::Meta::Requirements --> Working on List::Util Fetching http://www.cpan.org/authors/id/P/PE/PEVANS/Scalar-List-Utils-1.35.tar.gz ... OK Configuring Scalar-List-Utils-1.35 ... OK Building Scalar-List-Utils-1.35 ... OK Successfully installed Scalar-List-Utils-1.35 (upgraded from 1.22) --> Working on Parse::CPAN::Meta Fetching http://www.cpan.org/authors/id/D/DA/DAGOLDEN/Parse-CPAN-Meta-1.4409.tar.gz ... OK Configuring Parse-CPAN-Meta-1.4409 ... OK ==> Found dependencies: CPAN::Meta::YAML, JSON::PP --> Working on CPAN::Meta::YAML Fetching http://www.cpan.org/authors/id/D/DA/DAGOLDEN/CPAN-Meta-YAML-0.010.tar.gz ... OK Configuring CPAN-Meta-YAML-0.010 ... OK Building CPAN-Meta-YAML-0.010 ... OK Successfully installed CPAN-Meta-YAML-0.010 --> Working on JSON::PP Fetching http://www.cpan.org/authors/id/M/MA/MAKAMAKA/JSON-PP-2.27203.tar.gz ... OK Configuring JSON-PP-2.27203 ... OK Building JSON-PP-2.27203 ... OK Successfully installed JSON-PP-2.27203 Building Parse-CPAN-Meta-1.4409 ... OK Successfully installed Parse-CPAN-Meta-1.4409 (upgraded from 1.40) --> Working on version Fetching http://www.cpan.org/authors/id/J/JP/JPEACOCK/version-0.9904.tar.gz ... OK Configuring version-0.9904 ... OK Building version-0.9904 ... OK Successfully installed version-0.9904 (upgraded from 0.82) --> Working on CPAN::Meta::Requirements Fetching http://www.cpan.org/authors/id/D/DA/DAGOLDEN/CPAN-Meta-Requirements-2.125.tar.gz ... OK Configuring CPAN-Meta-Requirements-2.125 ... OK Building CPAN-Meta-Requirements-2.125 ... OK Successfully installed CPAN-Meta-Requirements-2.125 Building CPAN-Meta-2.133380 ... OK Successfully installed CPAN-Meta-2.133380 --> Working on ExtUtils::Helpers Fetching http://www.cpan.org/authors/id/L/LE/LEONT/ExtUtils-Helpers-0.021.tar.gz ... OK Configuring ExtUtils-Helpers-0.021 ... OK Building ExtUtils-Helpers-0.021 ... OK Successfully installed ExtUtils-Helpers-0.021 --> Working on ExtUtils::InstallPaths Fetching http://www.cpan.org/authors/id/L/LE/LEONT/ExtUtils-InstallPaths-0.010.tar.gz ... OK Configuring ExtUtils-InstallPaths-0.010 ... OK ==> Found dependencies: ExtUtils::Config --> Working on ExtUtils::Config Fetching http://www.cpan.org/authors/id/L/LE/LEONT/ExtUtils-Config-0.007.tar.gz ... OK Configuring ExtUtils-Config-0.007 ... OK Building ExtUtils-Config-0.007 ... OK Successfully installed ExtUtils-Config-0.007 Building ExtUtils-InstallPaths-0.010 ... OK Successfully installed ExtUtils-InstallPaths-0.010 Configuring Module-Build-Tiny-0.030 ... OK Building Module-Build-Tiny-0.030 ... OK Successfully installed Module-Build-Tiny-0.030 Configuring YAML-Tiny-1.56 ... OK Building YAML-Tiny-1.56 ... OK Successfully installed YAML-Tiny-1.56 Configuring Module-Install-1.06 ... OK ==> Found dependencies: File::Remove, ExtUtils::MakeMaker, PAR::Dist, LWP::UserAgent, Module::ScanDeps, JSON --> Working on File::Remove Fetching http://www.cpan.org/authors/id/A/AD/ADAMK/File-Remove-1.52.tar.gz ... OK Configuring File-Remove-1.52 ... OK Building File-Remove-1.52 ... OK Successfully installed File-Remove-1.52 --> Working on ExtUtils::MakeMaker Fetching http://www.cpan.org/authors/id/B/BI/BINGOS/ExtUtils-MakeMaker-6.84.tar.gz ... OK Configuring ExtUtils-MakeMaker-6.84 ... OK Building ExtUtils-MakeMaker-6.84 ... OK Successfully installed ExtUtils-MakeMaker-6.84 (upgraded from 6.56) --> Working on PAR::Dist Fetching http://www.cpan.org/authors/id/R/RS/RSCHUPP/PAR-Dist-0.49.tar.gz ... OK Configuring PAR-Dist-0.49 ... OK Building PAR-Dist-0.49 ... OK Successfully installed PAR-Dist-0.49 --> Working on LWP::UserAgent Fetching http://www.cpan.org/authors/id/G/GA/GAAS/libwww-perl-6.05.tar.gz ... OK Configuring libwww-perl-6.05 ... OK ==> Found dependencies: URI::Escape, LWP::MediaTypes, Encode::Locale, HTTP::Status, File::Listing, HTTP::Negotiate, HTML::Entities, HTTP::Daemon, Net::HTTP, URI, HTML::HeadParser, HTTP::Cookies, HTTP::Request::Common, HTTP::Date, HTTP::Response, HTTP::Request, WWW::RobotRules --> Working on URI::Escape Fetching http://www.cpan.org/authors/id/G/GA/GAAS/URI-1.60.tar.gz ... OK Configuring URI-1.60 ... OK Building URI-1.60 ... OK Successfully installed URI-1.60 --> Working on LWP::MediaTypes Fetching http://www.cpan.org/authors/id/G/GA/GAAS/LWP-MediaTypes-6.02.tar.gz ... OK Configuring LWP-MediaTypes-6.02 ... OK Building LWP-MediaTypes-6.02 ... OK Successfully installed LWP-MediaTypes-6.02 --> Working on Encode::Locale Fetching http://www.cpan.org/authors/id/G/GA/GAAS/Encode-Locale-1.03.tar.gz ... OK Configuring Encode-Locale-1.03 ... OK Building Encode-Locale-1.03 ... OK Successfully installed Encode-Locale-1.03 --> Working on HTTP::Status Fetching http://www.cpan.org/authors/id/G/GA/GAAS/HTTP-Message-6.06.tar.gz ... OK Configuring HTTP-Message-6.06 ... OK ==> Found dependencies: HTTP::Date, IO::HTML --> Working on HTTP::Date Fetching http://www.cpan.org/authors/id/G/GA/GAAS/HTTP-Date-6.02.tar.gz ... OK Configuring HTTP-Date-6.02 ... OK Building HTTP-Date-6.02 ... OK Successfully installed HTTP-Date-6.02 --> Working on IO::HTML Fetching http://www.cpan.org/authors/id/C/CJ/CJM/IO-HTML-1.00.tar.gz ... OK Configuring IO-HTML-1.00 ... OK Building IO-HTML-1.00 ... OK Successfully installed IO-HTML-1.00 Building HTTP-Message-6.06 ... OK Successfully installed HTTP-Message-6.06 --> Working on File::Listing Fetching http://www.cpan.org/authors/id/G/GA/GAAS/File-Listing-6.04.tar.gz ... OK Configuring File-Listing-6.04 ... OK Building File-Listing-6.04 ... OK Successfully installed File-Listing-6.04 --> Working on HTTP::Negotiate Fetching http://www.cpan.org/authors/id/G/GA/GAAS/HTTP-Negotiate-6.01.tar.gz ... OK Configuring HTTP-Negotiate-6.01 ... OK Building HTTP-Negotiate-6.01 ... OK Successfully installed HTTP-Negotiate-6.01 --> Working on HTML::Entities Fetching http://www.cpan.org/authors/id/G/GA/GAAS/HTML-Parser-3.71.tar.gz ... OK Configuring HTML-Parser-3.71 ... OK ==> Found dependencies: HTML::Tagset --> Working on HTML::Tagset Fetching http://www.cpan.org/authors/id/P/PE/PETDANCE/HTML-Tagset-3.20.tar.gz ... OK Configuring HTML-Tagset-3.20 ... OK Building HTML-Tagset-3.20 ... OK Successfully installed HTML-Tagset-3.20 Building HTML-Parser-3.71 ... OK Successfully installed HTML-Parser-3.71 --> Working on HTTP::Daemon Fetching http://www.cpan.org/authors/id/G/GA/GAAS/HTTP-Daemon-6.01.tar.gz ... OK Configuring HTTP-Daemon-6.01 ... OK Building HTTP-Daemon-6.01 ... OK Successfully installed HTTP-Daemon-6.01 --> Working on Net::HTTP Fetching http://www.cpan.org/authors/id/G/GA/GAAS/Net-HTTP-6.06.tar.gz ... OK Configuring Net-HTTP-6.06 ... OK Building Net-HTTP-6.06 ... OK Successfully installed Net-HTTP-6.06 --> Working on HTTP::Cookies Fetching http://www.cpan.org/authors/id/G/GA/GAAS/HTTP-Cookies-6.01.tar.gz ... OK Configuring HTTP-Cookies-6.01 ... OK Building HTTP-Cookies-6.01 ... OK Successfully installed HTTP-Cookies-6.01 --> Working on WWW::RobotRules Fetching http://www.cpan.org/authors/id/G/GA/GAAS/WWW-RobotRules-6.02.tar.gz ... OK Configuring WWW-RobotRules-6.02 ... OK Building WWW-RobotRules-6.02 ... OK Successfully installed WWW-RobotRules-6.02 Building libwww-perl-6.05 ... OK Successfully installed libwww-perl-6.05 --> Working on Module::ScanDeps Fetching http://www.cpan.org/authors/id/R/RS/RSCHUPP/Module-ScanDeps-1.12.tar.gz ... OK Configuring Module-ScanDeps-1.12 ... OK ==> Found dependencies: Test::Requires --> Working on Test::Requires Fetching http://www.cpan.org/authors/id/T/TO/TOKUHIROM/Test-Requires-0.07.tar.gz ... OK ==> Found dependencies: Module::Build --> Working on Module::Build Fetching http://www.cpan.org/authors/id/L/LE/LEONT/Module-Build-0.4203.tar.gz ... OK Configuring Module-Build-0.4203 ... OK ==> Found dependencies: Module::Metadata, Perl::OSType --> Working on Module::Metadata Fetching http://www.cpan.org/authors/id/E/ET/ETHER/Module-Metadata-1.000019.tar.gz ... OK Configuring Module-Metadata-1.000019 ... OK Building Module-Metadata-1.000019 ... OK Successfully installed Module-Metadata-1.000019 --> Working on Perl::OSType Fetching http://www.cpan.org/authors/id/D/DA/DAGOLDEN/Perl-OSType-1.006.tar.gz ... OK Configuring Perl-OSType-1.006 ... OK Building Perl-OSType-1.006 ... OK Successfully installed Perl-OSType-1.006 Building Module-Build-0.4203 ... OK Successfully installed Module-Build-0.4203 (upgraded from 0.3603) Configuring Test-Requires-0.07 ... OK Building Test-Requires-0.07 ... OK Successfully installed Test-Requires-0.07 Building Module-ScanDeps-1.12 ... OK Successfully installed Module-ScanDeps-1.12 --> Working on JSON Fetching http://www.cpan.org/authors/id/M/MA/MAKAMAKA/JSON-2.90.tar.gz ... OK Configuring JSON-2.90 ... OK Building JSON-2.90 ... OK Successfully installed JSON-2.90 Building Module-Install-1.06 ... OK Successfully installed Module-Install-1.06 38 distributions installed --> Working on . Configuring /usr/local/fluent-agent-lite-v1.0 ... OK ==> Found dependencies: Log::Minimal, JSON::XS, Data::MessagePack --> Working on Log::Minimal Fetching http://www.cpan.org/authors/id/K/KA/KAZEBURO/Log-Minimal-0.16.tar.gz ... OK Configuring Log-Minimal-0.16 ... OK Building Log-Minimal-0.16 ... OK Successfully installed Log-Minimal-0.16 --> Working on JSON::XS Fetching http://www.cpan.org/authors/id/M/ML/MLEHMANN/JSON-XS-3.01.tar.gz ... OK Configuring JSON-XS-3.01 ... OK ==> Found dependencies: Types::Serialiser, common::sense --> Working on Types::Serialiser Fetching http://www.cpan.org/authors/id/M/ML/MLEHMANN/Types-Serialiser-1.0.tar.gz ... OK Configuring Types-Serialiser-1.0 ... OK ==> Found dependencies: common::sense --> Working on common::sense Fetching http://www.cpan.org/authors/id/M/ML/MLEHMANN/common-sense-3.72.tar.gz ... OK Configuring common-sense-3.72 ... OK Building common-sense-3.72 ... OK Successfully installed common-sense-3.72 Building Types-Serialiser-1.0 ... OK Successfully installed Types-Serialiser-1.0 Building JSON-XS-3.01 ... OK Successfully installed JSON-XS-3.01 --> Working on Data::MessagePack Fetching http://www.cpan.org/authors/id/G/GF/GFUJI/Data-MessagePack-0.48.tar.gz ... OK Configuring Data-MessagePack-0.48 ... OK Building Data-MessagePack-0.48 ... OK Successfully installed Data-MessagePack-0.48 <== Installed dependencies for .. Finishing. 5 distributions installed
●fluent-agent-lite.confサンプル
# fluentd tag prefix of all LOGS TAG_PREFIX="" # fluentd message log attribute name (default: message) # FIELD_NAME="message" ### LOGS: tag /path/to/log/file # # LOGS=$(cat <<"EOF" # apache2 /var/log/apache2/access.log # # yourservice /var/log/yourservice/access.log # EOF # ) LOGS=$(cat <<"EOF" syslog /var/adm/messages EOF ) ### or, read from external file # LOGS=$(cat /etc/fluent-agent.logs) # SERVERNAME[:PORTNUM] # port number is optional (default: 24224) PRIMARY_SERVER="primary.fluentd.local:24224" ### or, PRIMARY SERVER LIST FILE of servers # PRIMARY_SERVERS_LIST="/etc/fluent-agent.servers.primary" # secondary server setting is optional... # SECONDARY_SERVER="secondary.fluentd.local:24224" # SECONDARY_SERVERS_LIST is available as like as PRIMARY_SERVERS_LIST # max bytes to try read as one action from tail (default: 1MB) # READ_BUFFER_SIZE=1048576 # PROCESS_NICE default: 0 # PROCESS_NICE=-1 # TAIL_INTERVAL=0.1 # TAIL_PATH=/usr/bin/tail TAIL_PATH=/usr/local/fluent-agent-lite-v1.0/bin/tail # Tag , data and interval of ping message (not to output ping message when tag not specified) # PING_TAG=ping # PING_DATA=`hostname` # PING_INTERVAL=60 # Tag name of 'drain_log' (none: not to output drain_log) # DRAIN_LOG_TAG= # connection keepalive time in seconds. 0 means infinity (DEFAULT: 1800) # KEEPALIVE_TIME=1800 # LOG_PATH=/tmp/fluent-agent.log LOG_PATH=/usr/local/fluent-agent-lite-v1.0/fluent-agent.log # LOG_VERBOSE=false # PERL_PATH=/usr/bin/perl PERL_PATH=/usr/local/fluent-agent-lite-v1.0/bin/perl
RHEL6系 の authorized_keys と SELinux
Cent6系の初期インストール直後、SELinux on の状態で公開鍵認証の authorized_keys 設定するときよく忘れるのでメモしておきます。orz
とりあえず restorecon -R -v ~/.ssh を覚えておくこと。
mkdir .ssh chmod 700 .ssh vi .ssh/authorized_keys chmod 600 .ssh/authorized_keys restorecon -R -v ~/.ssh
※/usr/bin/ssh-copy-id もありますが、やってることは殆ど一緒でした。どちらでもいいのかなと。
詳しい解説はこの辺で。
http://2done.org/index.php?id=76
https://bugzilla.redhat.com/show_bug.cgi?id=499343
http://stackoverflow.com/questions/9741574/redhat-6-oracle-linux-6-is-not-allowing-key-authentication-via-ssh
MySQL5.1→5.6移行を試し中のまとめ
Solaris10 ZFS & MySQL5.1 で動かしてる環境を Linux & MySQL5.6 化しています。
データサイズが大きくZFS 側はファイルシステムをgzip圧縮で利用中なので、Disk容量的に5.6へ移行する際にもサイズの大きいテーブルは圧縮が必須で、ここはInnoDBの圧縮機能を利用しました。
時間的な猶予はわりとあったので、多少試行錯誤しつつなんとか 5.6 のスレーブを作るところまで完成したのでまとめておきます。
しかし、@sh2nd さんの記事に頼りきりでした。良質な記事ばかりでいつも助かっています。本当にありがとうございますとしか言いようがありません。
●InnoDB圧縮まわり
http://d.hatena.ne.jp/sh2/20090628
http://d.hatena.ne.jp/sh2/20090705
●5.6のパラメータまわり
http://d.hatena.ne.jp/sh2/20130806
■移行方針
- クライアントライブラリは変更せず使えればそれに越したことはない
- アプリケーション向けの挙動はなるべく変わらないようにしたい
- 文字コードは 5.1 時の utf8 維持
- データは引き継いだ環境を作っておいて、ホストのIPアドレス変更作業で短時間停止で入替を試みる
- 先日の"MySQL Casual Talks vol.5"や、"db tech showcase 東京 2013"で聞いた感じで、GTIDやPerformanceSchema 周りは一旦off
■構成遷移
0.元構成 Master(Solaris ZFS/5.1) | 非同期レプリケーション | Slave(Solaris ZFS/5.1) 1.新ホストに 5.1 をコピー Master(Solaris ZFS/5.1) | 非同期レプリケーション | Slave(Solaris ZFS/5.1) -- stop slave & zfs snap した静止点をコピー --> Slave(Linux XFS/5.1) 2.新ホストの 5.1 を1回Masterと同期 Master(Solaris ZFS/5.1) ---------+ | | 非同期レプリケーション 非同期レプリケーション | | Slave(Solaris ZFS/5.1) Slave(Linux XFS/5.1) 3.新ホストの 5.1 を 5.5 へアップグレード Master(Solaris ZFS/5.1) ---------+ | | 非同期レプリケーション stop slave & mysqlupgrade | | Slave(Solaris ZFS/5.1) Slave(Linux XFS/5.5) 4.新ホストの 5.5 を replicate-do-db で必要なDBに絞ってレプリケーション再開 Master(Solaris ZFS/5.1) ---------+ | | 非同期レプリケーション 非同期レプリケーション(replicate-do-dbでDB絞り) | | Slave(Solaris ZFS/5.1) Slave(Linux XFS/5.5) ※mysqlデータベース構造が同期されないように replicate-do-db でデータ移行が必要なDBに絞りました 5.新ホストの 5.5 で、ALTER TABLE ~ ROW_FORMAT=compressed KEY_BLOCK_SIZE=n で圧縮 Master(Solaris ZFS/5.1) ---------+ | | 非同期レプリケーション stop slave -> start slave io_thread -> ALTER TABLE | | Slave(Solaris ZFS/5.1) Slave(Linux XFS/5.5) ※ALTER 中はレプリケーションのSQLスレッドを停止、IOスレッドはバイナリログ確保のために稼動させ続けました 6.圧縮の終わった 5.5 でSQLスレッドを稼動させて同期させる Master(Solaris ZFS/5.1) ---------+ | | 非同期レプリケーション 非同期レプリケーション(replicate-do-dbでDB絞り) | | Slave(Solaris ZFS/5.1) Slave(Linux XFS/5.5) ※InnoDBテーブル圧縮状態 ※後続の 5.6 側レプリケーションに備えて、5.5 側では log_slave_updates=ON しておきます 7.Linux 5.5 を一旦停止して、5.6 向けにデータコピーして5.6へアップグレード Master(Solaris ZFS/5.1) ---------+ | | 非同期レプリケーション 非同期レプリケーション(replicate-do-dbでDB絞り) | | Slave(Solaris ZFS/5.1) Slave(Linux XFS/5.5) | 5.5 のコールドバックアップ状態のファイルをコピー & msyqlupgrade | Slave(Linux XFS/5.6) 8.Linux 5.6 側を 5.5 と非同期レプリケーションを張る Master(Solaris ZFS/5.1) ---------+ | | 非同期レプリケーション 非同期レプリケーション(replicate-do-dbでDB絞り) | | Slave(Solaris ZFS/5.1) Slave(Linux XFS/5.5) | 非同期レプリケーション(replicate-do-dbでDB絞り) | Slave(Linux XFS/5.5)
■5.6 の my.cnf サンプル
[mysqld_safe] basedir = /usr/local/mysql [mysqld] ## Base socket = /tmp/mysql56.sock port = 33306 user = mysql character_set_server = utf8 collation_server = utf8_general_ci default_time_zone = SYSTEM default_storage_engine = InnoDB innodb_file_per_table = 1 skip-character-set-client-handshake skip-name-resolve explicit_defaults_for_timestamp = OFF #sql_mode = TRADITIONAL #transaction_isolation = READ-COMMITTED ## File Location basedir = /usr/local/mysql datadir = /usr/local/mysql/data innodb_data_home_dir = /usr/local/mysql/data innodb_log_group_home_dir = /usr/local/mysql/data slave_load_tmpdir = /usr/local/mysql/mysqltmp tmpdir = /usr/local/mysql/mysqltmp innodb_data_file_path = ibdata1:1G:autoextend ## InnoDB #innodb_buffer_pool_dump_at_shutdown = ON #innodb_buffer_pool_load_at_startup = ON innodb_buffer_pool_size = 10240M #innodb_checksum_algorithm = crc32 innodb_file_format = Barracuda innodb_flush_method = O_DIRECT innodb_large_prefix = ON innodb_log_buffer_size = 8M innodb_log_file_size = 1G innodb_log_files_in_group = 3 #innodb_online_alter_log_max_size = 1G innodb_print_all_deadlocks = ON innodb_flush_log_at_trx_commit = 1 innodb_read_io_threads = 16 innodb_write_io_threads = 16 innodb_io_capacity = 5000 innodb_thread_concurrency = 0 innodb_autoextend_increment = 256 skip-innodb-doublewrite ## PerformanceSchema performance_schema = OFF ## MyISAM key_buffer_size = 512M ## Memory max_heap_table_size = 256M tmp_table_size = 256M ## Binary Log binlog_format = MIXED log-bin = mysql-bin binlog_checksum = none #enforce_gtid_consistency = ON expire_logs_days = 2 #gtid_mode = ON log_slave_updates = ON master_info_repository = TABLE #master_verify_checksum = ON max_binlog_size = 1G relay_log_info_repository = TABLE relay_log_recovery = ON #report_host = <hostname> #report_port = 3306 server_id = 3 #slave_net_timeout = 30 sync_binlog = 1 ## Slow Query and General Log log_output = FILE log_queries_not_using_indexes = ON log_slow_slave_statements = ON log_throttle_queries_not_using_indexes = 60 long_query_time = 60.0 slow_query_log = ON #general_log = 1 ## Connection Management #log_warnings = 2 max_allowed_packet = 16M #max_connect_errors = 999999999 max_connections = 200 table_open_cache = 10000 table_open_cache_instances = 16 ## Memory Allocation per Connection read_buffer_size = 2M read_rnd_buffer_size = 2M sort_buffer_size = 2M ## Replication read_only replicate-do-db=hogehoge #plugin-load=rpl_semi_sync_master=semisync_master.so #plugin-load=rpl_semi_sync_slave=semisync_slave.so #//master settings #rpl_semi_sync_master_enabled=1 #rpl_semi_sync_master_timeout=5000 [client] port = 33306 socket = /tmp/mysql56.sock [mysqldump] quick max_allowed_packet = 16M
■5.6 のエラーログサンプル
2013-11-xx 16:33:12 34945 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='127.0.0.1', master_port= 3306, master_log_file='mysql-bin.000008', master_log_pos= 71975, master_bind=''. New state master_host='xxx.xxx.xxx.xxx', master_port= 3306, master_log_file='mysql-bin.000008', master_log_pos= 71975, master_bind=''. 2013-11-xx 16:33:24 34945 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2013-11-xx 16:33:24 34945 [Note] Slave I/O thread: connected to master 'slave@xxx.xxx.xxx.xxx:3306',replication started in log 'mysql-bin.000008' at position 71975 2013-11-xx 16:33:24 34945 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000008' at position 71975, relay log './mysqlhost4-relay-bin.000001' position: 4 2013-11-xx 16:33:24 34945 [Warning] Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Error_code: 1193 2013-11-xx 16:33:24 34945 [Warning] Slave I/O: Unknown system variable 'SERVER_UUID' on master. A probable cause is that the variable is not supported on the master (version: 5.5.34-log), even though it is on the slave (version: 5.6.14-log), Error_code: 1193 2013-11-xx 15:17:07 34945 [Note] /usr/local/mysql/bin/mysqld: Normal shutdown 2013-11-xx 15:17:07 34945 [Note] Giving 2 client threads a chance to die gracefully 2013-11-xx 15:17:07 34945 [Note] Event Scheduler: Purging the queue. 0 events 2013-11-xx 15:17:07 34945 [Note] Shutting down slave threads 2013-11-xx 15:17:07 34945 [Note] Error reading relay log event: slave SQL thread was killed 2013-11-xx 15:17:08 34945 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 2013-11-xx 15:17:08 34945 [Note] Slave I/O thread killed while reading event 2013-11-xx 15:17:08 34945 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.000015', position 45513016 2013-11-xx 15:17:08 34945 [Note] Forcefully disconnecting 0 remaining clients 2013-11-xx 15:17:08 34945 [Note] Binlog end 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'partition' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'ARCHIVE' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'BLACKHOLE' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_FIELDS' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_INDEXES' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_SYS_TABLES' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_FT_CONFIG' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_FT_DELETED' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_METRICS' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_CMPMEM' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_CMP_RESET' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_CMP' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_LOCK_WAITS' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_LOCKS' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'INNODB_TRX' 2013-11-xx 15:17:08 34945 [Note] Shutting down plugin 'InnoDB' 2013-11-xx 15:17:08 34945 [Note] InnoDB: FTS optimize thread exiting. 2013-11-xx 15:17:08 34945 [Note] InnoDB: Starting shutdown... 2013-11-xx 15:17:45 34945 [Note] InnoDB: Shutdown completed; log sequence number 4314504063076 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA' 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'CSV' 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'MEMORY' 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'MRG_MYISAM' 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'MyISAM' 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'sha256_password' 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'mysql_old_password' 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'mysql_native_password' 2013-11-xx 15:17:45 34945 [Note] Shutting down plugin 'binlog' 2013-11-xx 15:17:45 34945 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete 131126 15:17:46 mysqld_safe mysqld from pid file /data/data/mysqlhost4.pid ended 131126 15:26:34 mysqld_safe Starting mysqld daemon with databases from /data/data 2013-11-xx 15:26:34 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2013-11-xx 15:26:34 4434 [Note] Plugin 'FEDERATED' is disabled. 2013-11-xx 15:26:34 4434 [Note] InnoDB: The InnoDB memory heap is disabled 2013-11-xx 15:26:34 4434 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2013-11-xx 15:26:34 4434 [Note] InnoDB: Compressed tables use zlib 1.2.3 2013-11-xx 15:26:34 4434 [Note] InnoDB: Using Linux native AIO 2013-11-xx 15:26:34 4434 [Note] InnoDB: Using CPU crc32 instructions 2013-11-xx 15:26:34 4434 [Note] InnoDB: Initializing buffer pool, size = 10.0G 2013-11-xx 15:26:35 4434 [Note] InnoDB: Completed initialization of buffer pool 2013-11-xx 15:26:35 4434 [Note] InnoDB: Highest supported file format is Barracuda. 2013-11-xx 15:26:36 4434 [Note] InnoDB: 128 rollback segment(s) are active. 2013-11-xx 15:26:36 4434 [Note] InnoDB: Waiting for purge to start 2013-11-xx 15:26:36 4434 [Note] InnoDB: 5.6.14 started; log sequence number 4314504063076 2013-11-xx 15:26:36 4434 [Note] Server hostname (bind-address): '*'; port: 33306 2013-11-xx 15:26:36 4434 [Note] IPv6 is not available. 2013-11-xx 15:26:36 4434 [Note] - '0.0.0.0' resolves to '0.0.0.0'; 2013-11-xx 15:26:36 4434 [Note] Server socket created on IP: '0.0.0.0'. 2013-11-xx 15:26:36 4434 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqlhost4-relay-bin' to avoid this problem. 2013-11-xx 15:26:36 4434 [Warning] Recovery from master pos 45513016 and file mysql-bin.000015. 2013-11-xx 15:26:36 4434 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. 2013-11-xx 15:26:36 4434 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000015' at position 45513016, relay log './mysqlhost4-relay-bin.000033' position: 4 2013-11-xx 15:26:36 4434 [Note] Slave I/O thread: connected to master 'slave@xxx.xxx.xxx.xxx:3306',replication started in log 'mysql-bin.000015' at position 45513016 2013-11-xx 15:26:36 4434 [Warning] Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Error_code: 1193 2013-11-xx 15:26:36 4434 [Warning] Slave I/O: Unknown system variable 'SERVER_UUID' on master. A probable cause is that the variable is not supported on the master (version: 5.5.34-log), even though it is on the slave (version: 5.6.14-log), Error_code: 1193 2013-11-xx 15:26:36 4434 [Note] Event Scheduler: Loaded 0 events 2013-11-xx 15:26:36 4434 [Note] /usr/local/mysql/bin/mysqld: ready for connections. Version: '5.6.14-log' socket: '/tmp/mysql56.sock' port: 23306 MySQL Community Server (GPL)
※余談
エラーログ見るとわりとボロボロです。レプリケーション周りの警告はまだ調べてません。
ALTER での圧縮ですが、135G のファイルが 10時間弱、650G のファイルが3日ほどかかりました。
iostat/vmstat を見ているとIOは余裕があったので、サーバのCPU環境にかなり依存するようです。
5.6.7 からは圧縮について調整するパラメータが追加されています。
今回は 5.5 上で圧縮したため使えませんでしたが、5.6系であればこの辺を調整してもう少し速度や圧縮率の調整ができそうです。
http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_compression_failure_threshold_pct
http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_compression_level
http://dev.mysql.com/doc/refman/5.6/en/innodb-parameters.html#sysvar_innodb_compression_pad_pct_max
MySQL Casual Talks vol.5 メモ書き
同じ日だったのでこちらにも参加。ついった上でのすごい人たちが沢山いてすごかった。後半になるにつれて力尽きて雑。
やっぱコード書けないと、、と相変わらず思うのでした。とりあえず 5.6 使っても良いけど、今のところGTIDはまだdisる方向。
※は個人所感
5.1 -> 5.5 or 5.6 案件が1個あるんだけど、とりあえず 5.6 ベースの GTID dis で作って移行テスト。
パラメータは sh2nd さんの資料も参考にささせてもらいつつ、移行方式が悩ましい。
まとめ
http://togetter.com/li/581621
@Yuryuさん
http://www.slideshare.net/Yuryu/mysql56gtid
※最初からGTIDをdisる話が勃発して、今日の話の流れが確定した瞬間w
@con_mameさん
http://www.slideshare.net/conmame/mysql56-27565355
5.6を一部プロダクショ、開発、管理、分析用で使ってる
Kage でSQLを新システム側に流して様子見た
※Kage すげー
スレーブのSKIPが簡単にできないのが一番困った
パフォーマンススキーマがメモリ食いまくる
バグたくさん踏んだ
bug.mysql.com みたほうがいい
性能落ちてる部分もある
@yoshi_kenさん
http://y-ken.hatenablog.com/entry/mysql-casual-talks-vol5-mysql-migration
トークナイザの変化についての対応
オプティマイザ対策(遅くなったものがある)
groonga側の不具合をだいぶ直してもらった
@yoku0825さん
http://www.slideshare.net/yoku0825/ss-27597161
酷いDDLをなんとかした話
※ソースアクセス権が無いのはうちも同じだけど、DDLを実装に近いフェーズでレビューできるのいいなあ。自分のトコはQA環境適用からじゃないと見えないので、問題があっても後戻りが多すぎるでひっくり返せないことが多々あるし…。
@kamipoさん
http://kamipo.github.io/talks/20131025-mysqlcasual5/#/title
mysql-build
プラグインも含めてビルド対応して、メンテ続けてるよ
あと、INDEX-FAST-SCAN高速化のお話
@do_akiさん
http://www.slideshare.net/do_aki/ss-27565738
マスターn:1レプリ大変だけど面白そう
@Rkajiyamaさん:
http://www.slideshare.net/rkajiyama/20131025-casual-uppt
http://labs.mysql.com/ の話
「They are NOT FIT FOR PRODUCTION.」
MySQL Multi-source Replication
do_akiさんに話されたのでほぼすっとばし
MySQL Utilities - Fabric
最終的にはこれのために以下も密接に絡む
・Multi-source Replication
・GTID
・mysqlfailover
※ロスレスレプリケーションへの実装変更もそうだと思う
MySQL JSON UDFs
山崎さんに丸投げ
Hadoop Applier
Insert Only!!
@songmuさん
http://songmu.github.io/slides/mysql-casual-5/
SQL::Translator::Diff
runcron
@nekogeruge_987さん
https://speakerdeck.com/player/bba5fed01f620131616c1edfe7f2e72f#
DB2からMySQL5.5移行を4時間でやる必要があった
Jnekins で移行JOBフロー組んで繰り返し実施して、最終的にはワンクリックで移行!
他のJOBフローミドルってあるんだろうか?
※DB2からというスゴイネタで、会場どよめきまくり
※JOBはJnekinsで良いと思うけれど、使ったことないけどこの辺かなあ。普段扱う機会があるのはOracleのDBMS_SCHDULER…orz
Jobscheduler
Hinemos
@monryさん
http://www.slideshare.net/monry84/20131025-my-sql-casual-talks-vol5
RDS費用のお話
@chobi_eさん
https://speakerdeck.com/chobie/sutorezienzindeasondemita
ランキングストレージエンジン作ってみた
Expert MySQLとか見て、Redisの実装持ってくる
MySQL User Forum (10/25) メモ書き
行ってきたときのメモ書きを後から読める程度にちょっとだけ手直ししたもの。
※は個人的に思ったこととかの殴り書き
一言でまとめると、5.7のおおよその概況を聞きつつ、Workbench と Utilities を使ってみようかなと思う会でありました。
梶山さん:States of Dolphin - MySQL 最新技術情報 2013 秋
- 品質押ししてた。開発者やQAチームの規模などが増えてるんだよー的な。
- 5.7 DMR で今やってること→リファクタと性能改善
- MySQL Fabric
シャーディング支援
Fabricサーバにシャーディング情報を持つ
MySQLコネクタが Fabric サーバと通信して、適切なMySQLサーバに接続させる形
※コネクションプール型の実装の場合どうするか? MySQLの場合はコネクションプールしないが普通なんだろうけれど。
- MySQL Cluster
SQLノードを MySQL 5.6 ベースで
外部キーが使えるように
山崎さん:MySQL Enterprise Monitor を使った MySQL サーバーの監視
- Enterprise Backup
物理バックアップだとmysqldumpよりだいぶデータ肥大化に対応できるよ
リストア早いよ(物理ベースだしね)
- Enterprise Edition
スレッドプールのスケーラビリティが高い
- Enterprise Audit
XML監査
※jsonで出せないかなあ?出せれば fluent に放り込んでHDFSに持っていくとか簡単にできるのにー。
エージェントレスも可能
※DBA視点での運用は一通り揃ってる的な感じ
※画面がOEMみたい...
※MySQL ってDevな人で基本簡単に運用し切れるよ!ってのがベースにあったとも思うんだけど、こういうのが出てくるとなると流石にそーいう規模ではなくなってきた感。
山崎さん:MySQL Workbench を使ったデータベース開発
※データモデルの部分だけでも使う価値があると思う
※接続先としては、開発環境向け
梶山さん:What's New in MySQL Database 5.7 & Fabric
- リリースまでの時間は18から24か月くらいの想定
5.6 2013/2 → 5.7 2014/8~?
- バージョン番号も未定で5.7は仮
- トランザクション性能だけでなく、コネクション確立高速化なども行っている
- 一時表の処理高速化
- 他セッション実行中SQLの横やりexplainできる
- JSON explain フォーマットもより見やすく変更される予定
- bugs.mysql.com
future request 出してもらえれば
自分も影響あるよ!ボタンができた(Affects me)
- Performance Schema
ストアド性能分析用サマリテーブルもできた
5.6の段階で InnoDBファイル移動、オンラインAlterTableができるようになった
5.7で更に性能をつきつめ
SQLスレッドが同一スキーマ内マルチスレッド実行可能になる
ロスレス準同期レプリケーション
今:マスタでコミットしてからスレーブ転送かけて、OKだったらアプリにACK
→スレーブ転送失敗したときに落ちるとスレーブにどこにもコミットデータがなくなるおそれ
変更:スレーブ送ってからコミットさせることで、どこかのスレーブには確実にバイナリログがあるようになる
- トリガ
制約チェックタイミングを改良して最後に回すような感じ
PK列の採番をトリガで設定するようなトリガは 5.6 だとエラってたけど、今のところは OKになる
→互換性維持するかオプション制御できるようにするかなどまだ方針未定なところ
- エラーログ
出力形式変更
- MySQL Utilities
zabbix2.0.9 インストール
マニュアルどおり。MySQLは、Hiveテスト環境作るために metastone 用で立てたのがあったので、とりあえず相乗り。
https://www.zabbix.com/documentation/jp/2.0/manual/installation/install_from_packages
http://dexlab.net/pukiwiki/index.php?Memo%2FLinux%2FZabbix#b22fad3e
Linuxホストの監視をGUIでポチポチしたいけれど、最低限やりたいテンプレートセットがイマイチ見つからず。
やりたいことは↓のような感じ。
・zabbixエージェントは使わずに snmpd で済ませる
・監視&グラフ化項目 CPU使用率 / ロードアベレージ / メモリ使用状況 / Disk使用率 / ネットワーク状況
・SSH ポート監視
・アプリ系ポート監視もできたらいいな
デフォルトの「Template SNMP OS Linux」だとグラフで Disk と NIC はあるのですが、CPU・メモリがパッと見では見つからず。
後トリガーが無い感じ。
色々、足しこむテンプレート作って追加する感じか、もっとWeb上さらって目的に近いテンプレートを見つけるか悩みどころ。
今日はここまで。
Linuxメモリ周りで流れてたものをぐぐった備忘録
スタート
http://togetter.com/li/572766?utm_source=dlvr.it&utm_medium=twitter
派生
http://mkosaki.blog46.fc2.com/blog-entry-510.htmlhttp://alpha-netzilla.blogspot.jp/2012/12/parameter-kernel.htmlhttps://groups.google.com/forum/#!topic/jlug.ml.debian.users/3Kus-uw43sAhttp://d.hatena.ne.jp/gunshot/20120105/p3http://servicesblog.redhat.com/2012/04/16/tuning-your-system-with-tuned/https://groups.google.com/forum/#!topic/android-group-japan/-1ID28WWEdUhttp://www.ibm.com/developerworks/jp/linux/library/l-linux-slab-allocator/http://www.coins.tsukuba.ac.jp/~yas/coins/os2-2011/2012-01-17/
備忘メモ:
tuned もうちょっと調べる
RedHat系で /proc/sys/kernel/softlockup_panic 0 デフォはたぶん無効なんだけど裏取り
slabinfo 周りの読み方というかこの辺の調整のやり方の調べるのやらなきゃ