Never miss an update:

  • Forwarding a webhook to multiple endpoints

    We have a growing support team and recently started work on an internal tool to help them better serve our users, this combines data from our main application and Helpscout, which we use extensively. We initially worked with scheduled API imports but quickly felt the need for real-time data coming in from Helpscout via webhooks. Unfortunately they only support setting up a single endpoint for their webhooks, as do some other services. As this endpoint was already in use, this posed a problem.

    To solve this issue, we decided to build something to mirror these incoming webhooks to any number of endpoints. Normally with webhooks you will want to verify that an incoming request was actually from the configured service by using a shared secret key to hash the body with, so the goal was to mirror it exactly, using the same headers and POST body.

    Trying out Lambda

    We actually started out with a solution on AWS Lambda running on Node, but the problem there was that you do not get easy access to the headers and request body, and have to define mapping templates. That ended up slightly changing the body, escaping http:// as http:\/\/ for example, which of course changes the verification signature.

    So back to the drawing board, Lambda seemed really awesome as you don’t have any servers to take care of, and webhooks really seem to fit nicely, but the Amazon API Gateway that needs to be in front of it felt annoying and tough to configure, we really wanted easy access to the raw request, and configure everything in code over clicking through endless and identical-looking forms.

    Forwardhook

    We decided to write Forwardhook using Go and it’s been serving our needs since. We are running this on AWS Container Services using our provided docker image. We picked Go as we like the benefits of a single binary, cross-compiliation and the low resource usage.

    As the goal is quite simple, this tool is as well. It mirrors incoming requests, using the same headers and body, and will retry a failed connection up to 10 times. Failing requests (non-20x) are not retried. Besides this it’s fire and forget, so we are always returning a 200 to the original requester.

    Endpoints are configurable via FORWARDHOOK_SITES, which makes it easy to spin up multiple containers to point to other endpoints. We have two running at the moment for our Helpscout accounts as they are currently split up per country. A possible improvement could be to configure the endpoints via query parameters, to be able to run a single instance for however many webhooks you need. For now, we simply run two containers and listen on two seperate ports.

    So give it a spin if you ever need to mirror incoming webhooks to multiple locations! :-)

    Read more
  • Working at Springest (video)

    At Springest we pride ourselves on our company culture and Frissr made a nice video of it. A big part of the video is about what it’s like to work as a developer at Springest.

    Interested in working at Springest? Check out our vacancy for a senior ruby on rails developer.

    Read more
  • Exploring the Simple Form Gem

    During my first months at Springest, as a fast-learning junior developer, I will be diving into the workings and usage of popular gems and posting here my experiences with each of them once in a while. Simple Form is the gem of this week!

    Forms are essential for user input in web applications. However, they can become tedious to write and maintain because of the need to handle form control naming and its numerous attributes.

    That’s where the Simple Form gem comes in handy. It aims to be as flexible as possible while helping you with powerful components to create your forms. And don’t worry, because you will still be able to define your layout the way you do now.

    What We Are Building

    In this article we will be building a model-centric form for creating and editing companies including the following fields: a text field for its name, a text area for its description, a select box for its size (number of employees) and a checkboxes collection for its different categories.

    Defining Relationships Between Models

    To be able to build this form we will first need a company and a category models. Beside these two models we will also have a characterization join model, which will be in charge of connecting the company and the category models. Any particular row in the characterizations table will effectively join a row in the companies table and a row in the categories table.

    Check out the following diagram showing the models associations:

    The following code shows the corresponding validations and how the structure would look like:

    class Company < ActiveRecord::Base
      has_many :characterizations, dependent: :destroy
      has_many :categories, through: :characterizations
    
      SIZES = ["0-10", "11-30", "31-50", "More than 50"]
    
      validates :name, presence: true, uniqueness: true
      validates :description, presence: true
      validates :size, presence: true, inclusion: { in: SIZES }
    end
    
    class Category < ActiveRecord::Base
      has_many :characterizations, dependent: :destroy
      has_many :companies, through: :characterizations
    
      validates :name, presence: true, uniqueness: true
    end
    
    class Characterization < ActiveRecord::Base
      belongs_to :company
      belongs_to :category
    end

    Let’s Build the Form

    I will be guiding you through the different form helpers in Simple Form and how to use them compared to the built-in Rails Form Builder.

    The form will be scoped to the company model and to start using Simple Form you just have to add the helper it provides:

    <%= simple_form_for @company do |f| %>

    Adding a Text Field and a Text Area

    When using the Rails Form Builder you need to specify one by one the type for each of the different form helpers and add a corresponding label to them. For the text and the text area fields you then have to pass in the name of the attribute to be called on the company object (name and description in this case).

    <%= f.label :name %>
    <%= f.text_field :name %>
    
    <%= f.label :description %>
    <%= f.text_area :description %>

    One of the nicest things about Simple Form is that you don’t need to specify the type of the different form elements, but it will figure that out by itself looking at the column type in the database and use an appropriate input for the column. For example, a column created with type :text in the database (like the company’s description in our case) will use a textarea input by default. You can check here a complete list of the available input types and defaults for each column type.

    A label element will also be added by default to each of the fields, transforming the above code into this:

    <%= f.input :name %>
    <%= f.input :description %>

    Adding a Select Box

    In the case of a select box, Rails Form Builder will also need the options array to be passed in as a second parameter.

    <%= f.label :size %>
    <%= f.select :size, Company::SIZES %>

    With Simple Form if you want to create a select containing the different sizes of a company you can do it overriding the :collection option. Collections can be arrays or ranges, and when a :collection is given the :select input will be rendered by default, so no need to pass the as: :select option:

    <%= f.input :size, collection: Company::SIZES, prompt: "- Select number of employees" %>

    Adding a Checkboxes Collection

    To generate a collection of checkboxes with Rails Form Builder we will need to pass in the following parameters to the form helper:

    • Name of the attribute or method to be called on the company object (as in the previous form elements)
    • Options array or collection to be shown
    • Name of the attribute to be used as the checkbox value
    • Name of the attribute to be used as the checkbox label text
    <%= f.label :categories %><br>
    <%= f.collection_check_boxes(:category_ids, Category.all, :id, :name) %>

    To deal with these type of associations, Simple Form can generate either select inputs or a series of radios buttons or checkboxes. We will only need to pass in the categories collection as a parameter. The association helper would render a :select input by default, which can be changed into radio buttons or checkboxes like we did in this case:

    <%= f.association :categories, as: :check_boxes %>

    What We Got

    Our form will end up looking very short and simple indeed ☺ :

    <%= simple_form_for @company do |f| %>
      <%= f.input :name %>
      <%= f.input :description %>
      <%= f.input :size, collection: Company::SIZES, prompt: "- Select number of employees" %>
      <%= f.association :categories, as: :check_boxes %>
      <%= f.button :submit %>
    <% end %>

    What’s Next

    If you didn’t have enough and you are actually interested in doing more complicated things with Simple Form, you will be happy to hear that:

    Read more
  • Running a Rails application on AWS using Elastic Beanstalk and Docker

    Deploying to AWS or any other server environment can be a chore. Fortunately there are tools like Elastic Beanstalk and Docker to make life a little easier.

    Elastic Beanstalk

    Elastic Beanstalk offers a lot of bang for your buck. Deployment, auto scaling, load balancing and database management are just some of the features Elastic Beanstalk offers out-of-the-box.

    It’s hard to find a software platform that isn’t supported by Elastic Beanstalk, it currently offers support for Ruby, PHP, Python, Java, .NET, Node.JS and Go. But it doesn’t stop there, for every platform there is a multitude of platform versions and servers to run your application. Just for the Ruby environment you can choose between several versions of Ruby in combination with Passenger or Puma.

    But it doesn’t stop there! Elastic Beanstalk has one more trick up it’s sleeve: The ability to run your application inside a Docker container.

    Why Docker?

    There are a lot of advantages to using Docker for running your application. A very incomplete list:

    • Less overhead on the environment running your application.
    • Test and run your application in the same environment.
    • Easy to debug. You can export a faulty container, download it and start it locally.
    • Simplify configuration, less need for customising the host environment using .ebextensions.
    • Faster deployments and auto scaling.

    We had been experimenting with using Docker to run our stack for a while. Our first try was using Deis to run our staging environment. We saw a lot of potential and spent quite a lot of time on getting our stack to work but, in the end we simply didn’t feel comfortable with the level of stability Deis was offering.

    This made us little weary of trying it again on Elastic Beanstalk but, after some initial experimentation we were sold!

    If you are interested in knowing more about our experiences using Deis please let us know in the comments. If we receive enough interest we might devote an article to the subject.

    Our solution

    There’s more than one way to go about deploying an application on an Elastic Beanstalk Docker environment. First of all, you have the choice between running a single or multi-docker container setup. For our purpose the Single Docker setup works just fine.

    First, we are going to need a base docker image that we can use for our deployments:

    FROM springest/ruby:2.1.5
    
    RUN apt-get -y -q update && apt-get -y -q install build-essential python-dev python-pip && apt-get clean
    RUN pip install awscli awsebcli
    
    RUN echo 'gem: --no-ri --no-rdoc' > ~/.gemrc
    
    # Rubygems and bundler
    RUN gem update --system --no-ri --no-rdoc
    RUN gem install bundler --no-ri --no-rdoc
    
    RUN git config --global user.email "developers@awesome-app.com"
    RUN git config --global user.name "AwesomeApp"

    Build and push this to a Docker repository like Docker Hub or Quay.io:

    docker build -t awesome_app/base .
    docker tag awesome_app/base:latest awesome_app/base:0.1
    docker push awesome_app/base

    You should of course replace awesome_app with the name of your own organisation on the Docker repository.

    Next we will create the Dockerrun.aws.json file. This file holds the configuration needed for Elastic Beanstalk to deploy your Docker container:

    {
      "AWSEBDockerrunVersion": 1,
      "Ports": [
        {
          "ContainerPort": "3000"
        }
      ],
      "Volumes": [
        {
          "HostDirectory": "/var/app/current/webapp/tmp",
          "ContainerDirectory": "/webapp/tmp"
        }
      ],
      "Logging": "/webapp/log"
    }

    The file includes the following sections:

    • AWSEBDockerrunVersion - Specifies the version number. 1 for single and 2 for multi.
    • Ports - Which ports to expose on the Docker container.
    • Volumes - Which volumes should be mounted from host to the Docker container.
    • Logging - Where to mount the host log directory to the docker container. The host log directory can be found here: /var/log/eb-docker/containers/eb-current-app.

    If you are already familiar with the structure of the Dockerrun.aws.json you might notice we left out the Image section. The above Dockerfile will be used as a base for a Docker image that will be built during deployment by Elastic Beanstalk. For this we need to create a different Dockerfile which will be sent along with all other application files to Elastic Beanstalk during deployment.

    Create the following Dockerfile and place it in config/docker:

    FROM awesome_app/base:0.1
    
    ADD webapp /webapp/
    
    WORKDIR /webapp
    
    CMD bundle exec puma -C config/puma.rb
    
    EXPOSE 3000

    In this example we start the application by running puma but you could just as easily start any other application server.

    For the final step we will need to pull it all together. We will be deploying from inside the awesome-app/base:0.1 docker container. For this we will create a simple script called deploy inside the script directory:

    #!/bin/bash
    
    function check_environment_status {
      environment=$1
    
      echo "Checking if $environment is not updating..."
    
      STATUS=`eb status $environment | grep Status | sed 's/^[ ]*Status: \(.*\)$/\1/'`
    
      if [ "$STATUS" = "Updating" ]; then
        echo "Environment $environment is updating. Aborting deploy."
        exit 1
      fi
    }
    
    function deploy_to_environment {
      environment=$1
    
      echo "Deploying to $environment..."
    
      git add --all .
    
      git commit -q -m "Deploying to ${environment}"
    
      eb deploy --nohang --quiet
    }
    
    BASE_DIR=$PWD
    DEPLOY_DIR="$BASE_DIR/deploy"
    
    EC2_REGION="eu-central-1"
    ENVIRONMENT_NAME="awesome-app"
    
    rm -rf $DEPLOY_DIR
    
    mkdir -p $DEPLOY_DIR/{webapp/public,.elasticbeanstalk}
    
    cd $DEPLOY_DIR
    
    cat > $DEPLOY_DIR/.elasticbeanstalk/config.yml <<- EBC
    branch-defaults:
      master:
        environment: ${ENVIRONMENT_NAME}
    global:
      application_name: awesome-app
      default_ec2_keyname: awesomeappkey
      default_platform: Docker 1.5.0
      default_region: ${EC2_REGION}
      profile: null
      sc: git
    EBC
    
    mkdir -p /root/.aws
    
    cat > /root/.aws/config <<- ACF
    [default]
    output = json
    region = eu-central-1
    aws_access_key_id = $AWS_KEY
    aws_secret_access_key = $AWS_SECRET_KEY
    ACF
    
    git init .
    
    check_environment_status ENVIRONMENT_NAME
    
    echo "Copying Elastic Beanstalk files..."
    cp -r $BASE_DIR/.ebextensions $DEPLOY_DIR
    cp -f $BASE_DIR/Dockerrun.aws.json $DEPLOY_DIR
    cp -f $BASE_DIR/config/docker/Dockerfile $DEPLOY_DIR
    
    echo "Copying application files..."
    cp $BASE_DIR/config.ru $DEPLOY_DIR/webapp/
    cp $BASE_DIR/Gemfile $DEPLOY_DIR/webapp/
    cp $BASE_DIR/Gemfile.lock $DEPLOY_DIR/webapp/
    cp $BASE_DIR/Rakefile $DEPLOY_DIR/webapp/
    cp -r $BASE_DIR/.bundle $DEPLOY_DIR/webapp
    cp -r $BASE_DIR/app $DEPLOY_DIR/webapp
    cp -r $BASE_DIR/config $DEPLOY_DIR/webapp
    cp -r $BASE_DIR/db $DEPLOY_DIR/webapp
    cp -r $BASE_DIR/lib $DEPLOY_DIR/webapp
    cp -r $BASE_DIR/public/assets $DEPLOY_DIR/webapp/public
    cp -r $BASE_DIR/public/images $DEPLOY_DIR/webapp/public
    cp -r $BASE_DIR/script $DEPLOY_DIR/webapp
    
    # Run bundle install here so we don't have to on every application server
    cd $DEPLOY_DIR/webapp
    bundle install --deployment -j4 --retry=3
    cd $DEPLOY_DIR
    
    # Remove any .git directory from gems that are downloaded from a git repository
    find $DEPLOY_DIR/webapp/vendor/bundle -name ".git" -type d | xargs rm -rf
    
    deploy_to_environment ENVIRONMENT_NAME
    
    rm -rf $DEPLOY_DIR

    In the example the EC2 region has been hardcoded to eu-central-1 but this can be changed to the region of your liking.

    The script creates two files needed by the Elastic Beanstalk CLI to deploy your application:

    • .elasticbeanstalk/config.yml - This file specifies the target environment for deployment
    • /root/.aws/config - This file specifies the AWS credentials file needed for authentication

    For the script to work it will need two environment variables:

    • $AWS_KEY
    • $AWS_SECRET_KEY

    These can be generated from the IAM console in your AWS console.

    Don’t forget to make the script executable:

    chmod +x script/deploy

    Finally, let’s run the script. First we will need to start a new docker container based on the awesome-app/base:0.1 image:

    docker run --rm -it -v $PWD:/webapp awesome_app/base:0.1 /bin/bash

    If everything went as planned you should be logged into the docker container as root.

    To run the script we first have to cd to the /webapp directory so we can run the deploy script:

    cd /webapp
    AWS_KEY=FOO AWS_SECRET_KEY=BAR script/deploy

    When the script is finished your application is pushed to Elastic Beanstalk for propagation to your EC2 instances, shaving a lot of time on spinning up new EC2 instances during auto scaling.

    What’s next

    Even though running the deployment from your local machine works perfectly fine, it would be a lot more efficient to let your CI environment take care of this. At Springest we use Wercker and their Ewok infrastructure to run our specs and deploy script.

    This article won’t go into detail how Wercker can be used to make the process even more efficient. If you are interested in the details please leave a comment. If we receive enough interest we might devote an article about it in the future.

    Read more
  • Deploy Notifications to NewRelic, AppSignal, and Slack with Elastic Beanstalk

    Note: This project is no longer maintained by our team, but it’s still cool to learn from. If you would like to take it over, email us, or check out one of its maintained forks.

    Amazon’s Elastic Beanstalk is a very nice platform to easily host simple applications, while it also gives you enough tools to customise your environment for more complex setups. One simple customisation we made here at Springest is a script to push deploy notifications to NewRelic, AppSignal, and our development Slack channel.

    Example Script for Slack

    For argument’s sake, we are going to use the Slack script as an example. The other scripts you can find in our repo.

    Below is an example script to post to Slack. It takes a couple of arguments. Read the usage info for more details. Note that this was heavily inspired by the scripts from @apancutt’s deploy notification script.

    Create a Webhook Integration in Slack

    First off, you need to create a webhook integration in Slack. You can do so by clicking the name of the channel you’re in and select Add a service integration.

    add a Slack webhook integration

    Create a file in script/slack_deploy.sh with the code below and make it executable with chmod +x script/slack_deploy.sh:

    #!/bin/bash
    #
    # File: ./script/slac_deploy.sh
    #
    # Notify the team through Slack that a deploy is done on Elastic
    # Beanstalk.
    #
    # Source: http://devblog.springest.com/deploy-notifications-to-newrelic-appsignal-and-slack-with-elastic-beanstalk/
    # Credits: https://github.com/apancutt/aws-eb-newrelic-deploynotify
    #
    # Usage: script/slack_deploy.sh -a TESTAPP -c aws-notifications -w https://hooks.slack.com/services/XXXXX/XXXXX/xxxxxxxxxxxxxxxxxxxxxxxx
    #
    
    version="1.0"
    
    function usage {
        echo "AWS Elastic Beanstalk Deployment Notifications for Slack (v${version})"
        echo
        echo "Usage: appsignal_deploy.sh -a <APP NAME> -c <SLACK CHANNEL> -k <API KEY> [options]"
        echo
        echo "Options:"
        echo
        echo "  -a  The name your the application in Elastic Beanstalk."
        echo "  -c  The channel to post to (without the hash)."
        echo "  -w  The webhook url to post to."
        echo "  -d  The name of the deployer (default: AWS Elastic Beanstalk)."
        echo "  -c  The icon to use (without the colons, default: package)."
        echo "  -e  Error if the HTTP request fails. Note that this will abort the deployment."
        echo "  -h  Displays this help message."
        echo "  -q  Quiet mode."
        echo "  -v  Display version information."
        echo
    }
    
    function info {
        echo "[INFO] ${@}"
    }
    
    function warn {
        echo "[WARN] ${@}"
    }
    
    function error {
        echo "[ERROR] ${@}" >&2
        exit 1
    }
    
    app_name=""
    channel=""
    webhook_url=""
    icon=""
    environment=$DEPLOY_STACK_NAME
    deployer=""
    verbose=1
    error_on_fail=0
    
    if [[ ${#} == 0 ]]; then
        usage
        exit 1
    fi
    
    while getopts "a:c:w:d:i:ehk:qv" option; do
        case "${option}" in
            a) app_name="${OPTARG}";;
            c) channel="${OPTARG}";;
            w) webhook_url="${OPTARG}";;
            d) deployer="${OPTARG}";;
            i) icon="${OPTARG}";;
            e) error_on_fail=1;;
            h) usage; exit;;
            q) verbose=0;;
            v) echo "Version ${version}"; exit;;
            *) echo; usage; exit 1;;
        esac
    done
    
    if [[ -z "${app_name}" ]]; then
        error "The application name must be provided"
    fi
    
    if [[ -z "${channel}" ]]; then
        error "The channel must be provided"
    fi
    
    if [[ -z "${webhook_url}" ]]; then
        error "The webhook_url must be provided"
    fi
    
    if [[ -z "${deployer}" ]]; then
        deployer="AWS Elastic Beanstalk"
    fi
    
    if [[ -z "${icon}" ]]; then
        icon="package"
    fi
    
    if [[ -f REVISION ]]; then
        app_version=$(cat REVISION)
    else
        app_version="unknown"
        error "Unable to extract application version from source REVISION
    file"
    fi
    
    if [[ -z "${environment}" ]]; then
      environment="development"
    fi
    
    if [[ ${verbose} == 1 ]]; then
        info "Application name: ${app_name}"
        info "Application version: ${app_version}"
        info "Application environment: ${environment}"
        info "Webhook URL: ${webhook_url}"
        info "Channel: ${channel}"
        info "Icon: ${icon}"
        info "Sending deployment notification..."
    fi
    
    http_response=$(curl -X POST -s -d "{\"channel\":\"#${channel}\",\"icon_emoji\":\":${icon}:\",\"text\":\"${app_name} was successfully deployed to ${environment} by ${deployer}\",\"username\":\"${deployer}\",\"attachments\":[{\"fallback\":\"${app_name} was successfully deployed to ${environment} by ${deployer}\",\"color\":\"#8eb573\",\"fields\":[{\"title\":\"Environment:\",\"value\":\"${environment}\",\"short\":false},{\"title\":\"Version:\",\"value\":\"${app_version}\"}]}]}" "${webhook_url}")
    http_status=$(echo "${http_response}" | head -n 1)
    echo "${http_status}" | grep -q "ok"
    
    if [[ ${?} == 0 ]]; then
        if [[ ${verbose} == 1 ]]; then
            info "Deployment notification successfully sent (${app_name} v${app_version})"
        fi
    else
        msg="Failed to send deployment notification: ${http_status}"
        if [[ ${error_on_fail} == 1 ]]; then
            error "${msg}"
        else
            warn "${msg}"
        fi
    fi

    When you try to use the script above, it will fail with a:

    ⇒  script/slack_deploy.sh -a TESTAPP -c aws-notifications -w https://hooks.slack.com/services/XXXXX/XXXXX/xxxxxxxxxxxxxxxxxxxxxxxx
    [ERROR] Unable to extract application version from source REVISION

    So let’s fix that.

    Creating the REVISION file

    AppSignal and Newrelic need a revision for best results, so we’re going to create one. We don’t have access to the git history directly while we run these scripts on EB, so let’s make sure they get created while we have it.

    You can either (1) update the REVISION file before you deploy with eb deploy or (2) have your CI / deploy tool (we use Wercker) write and commit the file. Either way, this is the command to update the file:

    git rev-parse HEAD > REVISION

    Script sources

    You can find the above script, and the scripts for NewRelic and AppSignal in the repository we created for it. Let us know if you have any fixes or improvements!

    Running the scripts during deploys

    To run the script, you need to tell EB which command to run. Additionally, you’ll want to make sure that it only runs once per deploy, and not on every instance (no need to have 40 deploy messages if you have 40 instances running your web cluster ;)

    So we’re going to use container_commands for that. Create a file called .ebextensions/003_notifications.config and provide the following input:

    # NewRelic, AppSignal, and Slack deploy notifications
    container_commands:
      00_aws-eb-newrelic-deploynotify:
        command: "webapp/script/newrelic_deploy.sh -a ${NEWRELIC_APP_NAME} -k ${NEWRELIC_API_KEY}"
        leader_only: true
        ignoreErrors: true
      01_aws-eb-appsignal-deploynotify:
        command: "webapp/script/appsignal_deploy.sh -a ${APPSIGNAL_APP_NAME} -k ${APPSIGNAL_PUSH_API_KEY}"
        leader_only: true
        ignoreErrors: true
      02_aws-eb-slack-deploynotify:
        command: "webapp/script/slack_deploy.sh -a <APP_NAME> -w ${SLACK_WEBHOOK_URL} -c <CHANNEL> -i <EMOJI_ICON>"
        leader_only: true
        ignoreErrors: true

    Setting the necessary ENV vars on Elastic Beanstalk

    As you can see from the example above, we use a couple of environment variables to call these scripts. To set these you can just use the eb command line tool:

    eb setenv NEWRELIC_APP_NAME=myapp NEWRELIC_API_KEY=xxxxxxxxxx APPSIGNAL_APP_NAME=myapp APPSIGNAL_PUSH_API_KEY=xxxxxxxxxx APP_NAME=myapp SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXXXXXXXXXXXX/XXXXXXX/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    Results

    If all goes well, you will see the notification in Slack:

    example notification on Slack

    Enjoy!

    Read more

subscribe via RSS