tower.im 的基于 docker 的测试和开发环境简介

intro

以 tower.im 项目的实践为例，讲解如何搭建入门级别的基于 docker 的测试环境和本地开发环境；并回顾实践中遇到的若干坑。

架构简介

dockerize 以前开发和测试时遇到的问题

• 一台测试服务器上运行着多个测试环境，交叉共享服务、端口，配置复杂，管理混乱
• 新增一个测试环境很耗时间
• 本地测试困难

解决方案

why docker

• lightweight
• eco
• image easy to build 

steps

• containerize every service
• combine into a docker stack
• spin up a stack for every one and for local

Describe Single Service

Dockerfile https://docs.docker.com/engine/reference/builder/

application server:

FROM ruby:2.5.1

# less change part comes first, ends up deeper in layer stack
RUN apt-get update -qq && apt-get install -y \
  build-essential \
  libxml2-dev \
  libxslt1-dev \
  imagemagick \
  apt-transport-https

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - && \
  curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
  echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list

RUN apt-get update -qq && apt-get install -y \
  nodejs \
  yarn

ENV APP_HOME=/app \
  LANG=C.UTF-8 \
  LANGUAGE=C.UTF-8 \
  LC_ALL=C.UTF-8

RUN bundle config mirror.https://rubygems.org https://gems.ruby-china.org

RUN mkdir $APP_HOME
WORKDIR $APP_HOME
COPY . $APP_HOME

• omit CMD
• omit bundle, yarn, config

file center:

FROM ruby:2.5.1

RUN apt-get update -qq && apt-get install -y --no-install-recommends \
  build-essential \
  libxml2-dev \
  libxslt1-dev

ENV APP_HOME=/app
RUN mkdir $APP_HOME
WORKDIR $APP_HOME
COPY Gemfile* $APP_HOME/

RUN bundle config mirror.https://rubygems.org https://gems.ruby-china.com
RUN bundle

COPY . $APP_HOME

• include bundle because it rare changes
• omit CMD too

Describe Entire Infrastructure

Docker Stack: https://docs.docker.com/get-started/part5/

docker-compose.yml https://docs.docker.com/compose/compose-file/

version: '3'

services:
  web:
    image: 127.0.0.1:5000/falcon1:latest
    command: bundle exec puma -t 2:16 -p 3000
    env_file: .env.local
    volumes:
      - ./bundle:/app/vendor/bundle
      - ./node_modules:/app/node_modules
      - ./assets:/app/public/assets
    ports:
      - "127.0.0.1:3001:3000"
    deploy:
      replicas: 1

  sidekiq:
    image: 127.0.0.1:5000/falcon1:latest
    command: bundle exec sidekiq
    env_file: .env.local
    volumes:
      - ./bundle:/app/vendor/bundle
      - ./node_modules:/app/node_modules
      - ./assets:/app/public/assets
    deploy:
      replicas: 1

  db:
    image: mysql
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    env_file: mysql.env # MYSQL_ROOT_PASSWORD
    volumes:
      - mysql_data:/var/lib/mysql

  redis:
    image: redis:4.0.11

  memcached:
    image: memcached:1.5.12

  elasticsearch:
    image: elasticsearch:5.6.13
    volumes:
      - es_data:/usr/share/elasticsearch/data

volumes:
  mysql_data:
  es_data:

deployment

镜像仓库 registry: https://docs.docker.com/registry/

部署工具 mina https://github.com/mina-deploy/mina

• push code
• build on server # why
• push image
• update stack with new image

require 'mina/rails'
require 'mina/git'
require 'wannabe_bool'

set :domain, 'x47b.tower.im'
set :repository, 'git@github.com:mycolorway/tower.git'
set :branch, ENV['BRANCH'] || 'master'
set :docker_registry, '127.0.0.1:5000'

# Optional settings:
set :user, 'root'          # Username in the server to SSH to.
set :forward_agent, true     # SSH forward_agent.

# shared dirs and files will be symlinked into the app-folder by the 'deploy:link_shared_paths' step.
# override default shared_dirs, shared_files
# other wise unexpected things happen
# like symbol linked tmp/cache will break assets:precompile
set :shared_dirs, ['mina-hack']
# set :shared_files, []

task :configure_stage do
  set :deploy_to, "/www_space/#{fetch :application_name}"

  set :docker_repo, "#{fetch :docker_registry}/#{fetch :application_name}" \
    unless set? :docker_repo
  set :image, "#{fetch :docker_repo}:latest"

  set :build, true unless set? :build

  set :run_migrate, false unless set? :run_migrate
end

task :build do
  command %(docker build -t #{fetch :image} .)
  command %(docker push #{fetch :image})
end

desc 'Deploys the current version to the server.'
task :deploy do
  # uncomment this line to make sure you pushed your local branch to the remote origin
  invoke :'git:ensure_pushed'

  deploy do
    # Put things that will set up an empty directory into a fully set-up
    # instance of your project.
    invoke :'git:clone'
    invoke :'deploy:link_shared_paths'
    # 'deploy:link_shared_paths' uses `ln -sf` but docker dosn't suuport a
    #   symbol linked Dockerfile
    %w(
      Dockerfile
      .env.local
      config/database.yml
      config/yetting.yml
      config/newrelic.yml
      tower2_production.key
      tower2_production.iv
    ).each do |linked_path|
      command %(cp "#{fetch(:shared_path)}/#{linked_path}" "./#{linked_path}")
    end
    # invoke :'bundle:install' # in image
    # invoke :'rails:db_migrate' # use image
    # invoke :'rails:assets_precompile' # look for ':compile_assets' below

    if fetch :build
      invoke :build
    else
      command "docker pull #{fetch :docker_repo}"
    end

    invoke :'deploy:cleanup'

    on :launch do
      in_path fetch(:shared_path) do
        # NOTE: it's the key to set BUNDLE_PATH=vendor/bundle in .env.local
        mount_bundle = "-v #{fetch :deploy_to}/shared/bundle:/app/vendor/bundle"
        mount_node_modules = "-v #{fetch :deploy_to}/shared/node_modules:/app/node_modules"
        mount_assets = "-v #{fetch :deploy_to}/shared/assets:/app/public/assets"

        command 'docker run --env-file .env.local --rm '\
          "#{mount_bundle} #{fetch :image} bundle install"

        if fetch(:compile_assets)
          command 'docker run --env-file .env.local --rm '\
            "#{mount_bundle} #{mount_node_modules} #{fetch :image} "\
            'yarn install'
          command 'docker run --env-file .env.local --rm '\
            "#{mount_bundle} #{mount_node_modules} #{mount_assets} #{fetch :image} "\
            'bundle exec rake assets:precompile --trace BEFORE_CREATE_DB=1'
        end

        command %(docker stack deploy --compose-file docker-stack.yml #{fetch :application_name})
      end
    end
  end

  # you can use `run :local` to run tasks on local machine before of after the deploy scripts
  # run(:local){ say 'done' }
end

task :beta1 do
  set :application_name, 'beta1'
  set :domain, 'f117'
  set :run_migrate, false
  set :compile_assets, ENV['ASSETS'].to_b

  invoke :configure_stage
end

task :beta2 do
  set :application_name, 'beta2'
  set :domain, 'f117'
  set :run_migrate, false
  set :compile_assets, ENV['ASSETS'].to_b

  invoke :configure_stage
end

task :beta3 do
  set :application_name, 'beta3'
  set :domain, 'f117'
  set :run_migrate, false
  set :compile_assets, ENV['ASSETS'].to_b

  invoke :configure_stage
end

task :beta4 do
  set :application_name, 'beta4'
  set :domain, 'f117'
  set :run_migrate, false
  set :compile_assets, ENV['ASSETS'].to_b

  invoke :configure_stage
end

• bundle / yarn
• assets

to deploy:

bundle exec mina falcon1 deploy --VERBOSE BRANCH=featureX

Logging

ELK: https://www.elastic.co/elk-stack

内存：

elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.1
    environment:
      cluster.name: "elk_es"
      # bootstrap.memory_lock: "true"
      ES_JAVA_OPTS: "-Xmx4g -Xms4g"
      # disable X-Pack
      # see https://www.elastic.co/guide/en/x-pack/current/xpack-settings.html
      #     https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html#xpack-enabling
      xpack.security.enabled: "false"
      xpack.monitoring.enabled: "false"
      xpack.graph.enabled: "false"
      xpack.watcher.enabled: "false"
    volumes:
      - /ES/es_data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      # - 9300:9300
    ulimits:
      memlock:
        soft: -1
        hard: -1
    deploy:
      resources:
        limits:
          memory: 5g
        reservations:
          memory: 4g
      restart_policy:
        condition: on-failure

misc:

• delete old indices with curator
• docker logging driver https://docs.docker.com/compose/compose-file/#logging

Development Environment

docker-sync to MAKE DOCKER FAST AGAIN http://docker-sync.io/

version: "2"
options:
  compose-dev-file-path: 'docker-compose.yml'
  verbose: false
syncs:
  #IMPORTANT: ensure this name is unique and does not match your other application container name
  tower2-app-sync: #tip: add -sync and you keep consistent names as a convention
    sync_strategy: 'native_osx'
    src: './'
    watch_excludes:
      - tmp
      - log
      - .git
    sync_excludes:
      - tmp
      - log
      - .git

version: '3'

services:
  web:
    build:
      context: .
      
    command: bundle exec rails s Puma -p 3000 -b '0.0.0.0'
    volumes:
      - bundle:/bundle
      - assets:/app/tmp/cache/assets
      - node_modules:/app/node_modules
      - ./tmp/pids:/app/tmp/pids # so we can delete server.pid when things crash
      - tower2-app-sync:/app:nocopy
    ports:
      - "3000:3000"
    depends_on:
      - db
      - redis
      - memcached
      - elasticsearch
    stdin_open: true # required by interactive mode
    tty: true

  
  sidekiq:
    build:
      context: .
    command: bundle exec sidekiq
    volumes:
      - .:/app
      - bundle:/bundle

  db:
    image: mysql:5.6
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    environment:
      MYSQL_ROOT_PASSWORD: tower
    volumes:
      - mysql_data:/var/lib/mysql
    ports:
      - "3306:3306"

  redis:
    image: redis:4.0.11

  memcached:
    image: memcached:1.5.12

  elasticsearch:
    image: elasticsearch:5.6.13
    volumes:
      - es_data:/usr/share/elasticsearch/data

volumes:
  bundle:
  assets:
  node_modules:
  mysql_data:
  es_data:
  tower2-app-sync:
    external: true

经验教训

更快的部署

https://docs.docker.com/storage/storagedriver/#images-and-layers

• cache as much as possible
• less change, higher position in Dockerfile
  
• CN registry
• optional assets:precompile

misc

• 入门曲线和接受度
• git:ensure_pushed
• write comment
• docker-sync CPU consumption on switching branch

What's Next?

• UI for op
• automated load test