Scalable “Black Box” Load Testing Using Selenium Grid and StackDriver On GKE

Scalable “Black Box” Load Testing Using Selenium Grid and StackDriver On GKE

Introduction

I was appointed once to serve as a DevOps coach by a services company operating in the Oil and Gas Industry. The mission statement had parts as diverse as re-architecting a dinosaur software from the “monolithic” era into microservices, designing the processes for migrating to this new architecture, and ensuring Continuous Delivery and Integration as for this new software factory environment. 

The client was a company that is a bit “distributed,” to say the least. With code writers spread across the UK, Tunisia, Oman, Pakistan, and China, you can imagine the challenges that stem from such a culturally and functionally diverse fauna. This situation also had an important implication: We as the “DevOps” team were not as close to the code source (pun intended)  as we probably needed to be.  The collective mind of that organization considered DevOps to be some fancy Sysadmin running Docker and Kubernetes. 

But like we’re about to see, it takes so much more to intimidate a member of the Guild!

Rationale for the Black Box Load Testing

My team had to contribute CI/CD pipelines, Infrastructure Automation, and some Quality Enforcements at many levels to some Kubernetes hosted application – all while not having any “power of decision” on any code at all.

This application was rather complicated, and for the most part, APIs had little to no documentation. So, the best we could do for our automated integration testing endeavors was going “black-box”: Browser automation with Selenium all the way!

Having a well-established team of product managers at hand, we succeeded in translating many of the application user stories to Selenium cases, which made it possible to go to production with decent integration testing.

Then came the time when management needed to size the resources required for this application – probably out of the need to get a forecast of the costs and give the application subscriptions a relevant price tag.

Weirdly enough, my team was put in charge of that, and not the “coders.” Remember, that application was a complete mystery for us. The only way we could go about it was the “black-box” approach again.

thanks to our Selenium cases, we could mimic human users interacting with the application. , We went on to set up a system that would spawn many such interactions, all running in parallel. As the software was being stormed this way, we would monitor different system metrics through Stackdriver – as we’re in a Google Kubernetes Engine environment.

Using this approach, we were not only able to deliver on the original assignment, but we could also stress-test the application.

All along the way of delivering this system, we ran through some practical considerations: how to scale the parallel Selenium Test runners? How to read the Stack driver Metrics? With no further Ado, let’s dive right in!

The Environment

We used Kubernetes in a Google Cloud-Based environment. However, the concepts we discuss in this post can be easily ported to any other Kubernetes set-up. The most significant change would be replacing Stackdriver with another monitoring solution. (If you want to learn about other monitoring options, you can draw inspiration from the following post)

To showcase our approach, we’ll be using two Kubernetes clusters:

The following figure describes the setup we’ll use to showcase our system:

PoC Architecture Schma

Deploying The Guestbook Demo Application

For the target app, we’ll deploy the Guestbook application included in the Kubernetes Examples github repository (This is an invaluable resource and we will even use it later to set up our Selenium Grid platform!)

Make sure you have set up a GKE cluster. To be able to deploy the Guestbook application, nodes of type n1-standard-1 will be sufficient. Also,make sure the Kubernetes Engine Monitoring is Activated as per this documentation (it’s the default on new GKE engine versions). This is necessary for us to access dashboards relevant with the Kubernetes Workload items.

Make kubectl connect to your cluster. Please follow this documentation if you’re not sure how to do it.

Now you’re connected, clone the Kubernetes Examples github repository, then go under the guestbook folder:

git clone https://github.com/kubernetes/examples.git
cd examples/guestbook

In this folder, you’ll find a set of deployment files. We will install all of the app components using the artifacts under all-in-one folder:

cd all-in-one

Edit the guestbook-all-in-one.yaml file to set the service type of service frontend to LoadBalancer. This is necessary as we’re going to target the application from outside this cluster, so we need it to be publicly accessible:

apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: guestbook
    tier: frontend
spec:
  # comment or delete the following line if you want to use a LoadBalancer
  # type: NodePort 
  # if your cluster supports it, uncomment the following to automatically create
  # an external load-balanced IP for the frontend service.
  type: LoadBalancer

  …

Deploy the app:

kubectl create -f guestbook-all-in-one.yaml

Wait for the public IP of the app to be available – this is the EXTERNAL-IP attribute of the frontend service:

kubectl get svc -W

Once you get the External IP, write it down. We’ll use it as a target for our Selenium Load Testing.Now let’s inspect the application. It is a Javascript frontend, exposing a text box and a blue ‘Submit’ button. Whenever you write something and hit that button, what you just typed is persisted to a Redis store. The frontend also displays the contents of that same Redis store under the button, so what you wrote will also show up there.

Guest Book Frontend

Our tests will consist of using Selenium to mimic the user interaction we described. The Selenium script will write something in the text box, hit the button, and verify that that same message is displayed at the bottom of the page.

But before getting there, let’s set up our Scalable Selenium testing platform.

Browser automation at scale on Selenium Grid

Selenium is a browser automation framework. It lets you mimic a user typing, clicking, scrolling, and interacting in any way with a web application from a browser. Along the way, Selenium can check conditions and take screenshots – making it an essential library for many software quality tasks.

In our case, we happen to have quite a comprehensive set of Selenium tests at hand. To go about stress testing our application, we were tempted to shoot some of them in parallel at the main URL. But things were not that easy.

While scripting the parallel execution was possible, adopting this manual approach might not scale: Imagine having to manage running hundreds of Selenium processes by hand. Where are you going to run them? On your laptop? Will it be able to run a couple of hundred browsers simultaneously? On the cloud? On a single machine? On several? On Kubernetes? How are you going to orchestrate them? How will you be collecting the outcome of each Selenium process?

To address all of these scalability concerns, we are going to use Selenium Grid – running on Kubernetes.

According to the Selenium Grid documentation:

…Its aim is to provide an easy way to run tests in parallel on multiple machines.

Selenium Grid makes this possible by exposing a clustered architecture.

In this architecture, users, scripts, CI/CD pipelines, whoever, submit test requests to a master, called Selenium Hub. The Selenium Hub manages a set of worker nodes, called the Selenium Nodes. The Selenium Hub orders the actual fulfillment of the test requests on the Selenium Nodes. It’s these Selenium Nodes who spawn the browser instances and run the user interactions, communicating their state and the test results to the Selenium Hub.

This way, you can centrally command an operation of massive parallel testing, preparing as much capacity as you need, and letting the Selenium Hub master manage the test execution.

Let’s install a Selenium Grid cluster.

Make sure you have a GKE cluster. For this one, we must choose nodes optimized for computing. Instances like c2-standard-4 would be sufficient.

Assuming you connected your kubectl command to this GKE cluster, let’s install Selenium Grid. For this, we’ll reuse the repository we cloned in the previous paragraph. From the root of the Kubernetes examples folder, head over to the staging/selenium folder:

cd staging/selenium

We’ll follow the steps as described in the README to install the different components: Install the Master, Selenium Hub:

kubectl create --filename=selenium-hub-deployment.yaml
kubectl create --filename=selenium-hub-svc.yaml

Verify that the selenium-hub service is up:

kubectl get pods

NAME                            READY   STATUS    RESTARTS   AGE
selenium-hub-7ff45ff687-pb8cz   1/1     Running   0          43s

Then We’ll Install 4 Chrome Selenium Nodes. These nodes will spawn the actual Chrome instances to fulfil test requests ordered by the Selenium Hub on behalf of external test scripts.

For this, we’ll edit the selenium-node-chrome-deployment.yaml to change the replicas property to 4:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-node-chrome
  labels:
    app: selenium-node-chrome
spec:
  replicas: 4
  ...

Then we’ll deploy the Chrome Selenium Nodes using:

kubectl create --filename=selenium-node-chrome-deployment.yaml

Let’s verify that the Selenium chrome nodes are up. We’ll check them at the selenium hub interface. Begin by doing:

# We get the Selenium Hub Pod name
export PODNAME=`kubectl get pods --selector="app=selenium-hub" --output=template --template="{{with index .items 0}}{{.metadata.name}}{{end}}"`
# Then we port-forward the traffic from this pod to our machine
kubectl port-forward $PODNAME 4444:4444

Let’s head our browser over to http://localhost:4444/grid/console. We can see our four Chrome Selenium Nodes being managed by the Selenium Hub we just accessed:

Selenium Grids Nodes

Deploying The Selenium Tests as a Kubernetes Job

Now that our scalable Selenium Grid infrastructure is ready, we can run the actual load test.

For this, we’ll need the test to be written as a browser interaction sequence and submitted to the Selenium Grid via the Selenium Hub.

We’ll host the test as a docker image, and then we’ll write a Kubernetes job definition that’ll run many instances of this image in parallel. Each one of these executions will be “ordered” from Selenium Hub, which will command its actual roll-out from one of the four chrome nodes we deployed previously. This Kubernetes job is just a statement of (parallel) work submitted to the Selenium Hub.

To accomplish this, we wrote a Python script that connects a Selenium driver to a Selenium Hub as a remote executor. The script asks that a browser visits the application’s main page. It creates a random text message, orders that message to be typed at the text box element, hits the submit button, then asks to verify that this same message is printed at the bottom of the web page.

As examapled by following guest_book_add_msg.py script:

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.keys import Keys
from os import environ
from datetime import datetime

def run_tests(test_target,browser,command_executor):

  print("Initializing the Selenium driver to connect to Selenium Hub")  
  driver = webdriver.Remote(
    command_executor=  command_executor,
    desired_capabilities=getattr(DesiredCapabilities, "browser")
  )

  print("Connecting to {}".format(test_target))
  driver.get(test_target)

  print("Getting the Text Box element")  
  text_box =
driver.find_elements_by_xpath("/html/body/div/form/fieldset/input")[0]
  message = "Message-{}".format(datetime.now())

  print("Sending message {}".format(message))
  text_box.send_keys(message)
  text_box.send_keys(Keys.ENTER)

  print("Getting the button element")
  button =
driver.find_elements_by_xpath("/html/body/div/form/fieldset/button")[0]
  print("Clicking on that button")
  button.click()

  print("Finding the placeholder")
  placeholder = driver.find_elements_by_xpath("/html/body/div/div")[0] 

  print("Asserting the placeholder contains the message")
  assert message in placeholder.text

  driver.quit()
  print("Browser %s checks out!" % browser)

if __name__ == "__main__":

  # We get our parameters from environment variables
  test_target = environ["TEST_TARGET"] 
  browser = environ["BROWSER"] 

  command_executor = environ["COMMAND_EXECUTOR"]
  run_tests(test_target,browser,command_executor)

For this script to work we need to install the Selenium Python package. To automate this dependency installation process in the next step, let’s create a requirements.txt file:

selenium

We’ll now build this Selenium test into a Docker image. Let’s save the following under Dockerfile, and make sure it sits next to the guest_book_add_msg.py script and the requirements.txt file:

FROM python:3

WORKDIR /usr/src/app

COPY requirements.txt ./

|RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "guest_book_add_msg.py"]

Now let’s build and tag the image. Note the Docker tag scheme we must follow, so we are able to push this image to the Google Cloud Container Registry.

docker build . -t us.gcr.io/your-project-ID/guest-book-add-msg
docker push us.gcr.io/your-project-ID/guest-book-add-msg

At this point, we have a docker image that contains the automated Selenium “test order” to be submitted to Selenium Hub. To run it, we’ll create a Kubernetes job spawning 50 parallel executions of this “test order.” Save the following under guest-book-stress-test-job.yaml and change the corresponding parameters to reflect your specific configuration:

apiVersion: batch/v1
kind: Job
metadata:
  name: guest-book-stress-test
spec:
  parallelism: 50 # Pods that can run in parallel
  completions: 50 # How many completions do we want to achieve. in this case, if equals to parallelism, we want to have at least same number running in parallel.
  template:
    spec:
      containers:
      - name: guest-book-add-msg
        image: us.gcr.io/your-project-ID/guest-book-add-msg
        env:
            - name: TEST_TARGET
              value: http://ext_ip.of.guestbook.svc/
            - name: BROWSER
              value: CHROME
            - name: COMMAND_EXECUTOR
              value: http://selenium-hub:4444/wd/hub
      restartPolicy: Never
  backoffLimit: 4

Now create the job to generate 50 parallel executions of our test, to be run by the Selenium Chrome nodes:

kubectl create -f guest-book-stress-test-job.yaml

We can see the pods spawned by the job we just created:


StackDriver Monitoring

After having generated some (residual) traffic on our guestbook application, let’s head over to Stackdriver Monitoring and see the effects of that traffic on our cluster workload items.

At your Google Cloud Console, go to Monitoring:

Monitoring Menu Console

Click to the Dashboards Menu Item, then on the Kubernetes Engine Item on the central dashboards list:

Go To GKE Dashboard

You see your Kubernetes related Items. We are concerned about the “Workloads” view:

Go To Workloads

Expand the guest-book-app node on the dashboard, then the default namespace, then the frontend deployment, to see your pods:

Go To Frontend Pods

Finally, expand one of the pods, and click on one of the containers to get a resource consumptions history:

Container Usage history

Voilà. At this level, you should be able to measure the resource consumption profile of your application. The intuition we are following here is that any usage fluctuation must be related to the number of simultaneous users interacting with the system. This approach can be used as a fair basis to estimate infrastructure needs as a function of users’ numbers.

Conclusion

We’ve seen how you can reuse your Selenium integration tests to lead a “black-box” stress-testing on an application as well as how one can use Selenium Grid as a platform to conduct such an operation in a scalable way. We also discussed how you could take advantage of the new “Kubernetes Monitoring Engine” on the Google Cloud Platform to read metrics tightly integrated to the Google  Kubernetes Engine Workload Items.

In the end, let’s emphasize that we’ve gone this path because we were not so well acquainted with the internals of the application we had to monitor. In other situations where we would have had more knowledge about the APIs we’re facing, we could have taken advantage of more classical approaches based on platforms like Jmeter or Gatling.

But this is how we make it at The Greenfield Guild. If there is no path, we make one! With creativity and thanks to our experience in Kubernetes and GKE, we are able to devise an innovative approach that could achieve what seemed to be challenging, and I think this is the real value you get from – so cool yet modest – Pros 🙂