People use software as it facilitates the performance of their tasks. The more popular is the app (i.e. the more active users it has), the more profitable it is for you. But you need to have your application ready for the userbase growth.
In this post we'll talk about why and how we measure and check this "growth readiness" as a software testing services company.
Why it's important
Retail businesses lose money every year because of the poor performance of their online stores. When large promo and discount events like Black Friday happen, the number of users and their data requests multiplies manyfold. If your site or app goes down in this case, you'll lose profit exactly when you planned to increase it. And one-time profit loss is a lucky scenario, since collateral reputation losses might have much more longstanding consequences.
The same applies to any software business. In 2009, Microsoft opened access to the beta of Windows 7. The influx of users wanting to download it was so massive the servers could not stand it. Or you might have heard of the game "Among Us" that at some point became so popular its servers couldn't cope with so many users being online at the same time, and it's been a challenge for the developers to address that.
There's a rumor saying that when Google.com went down for 5 minutes in August 2013, that costed the company behind it $545,000. We can only imagine how expensive was the 2020 Google services outage.
No wonder there are also Ddos attacks targeted at getting servers down and get businesses out of the game this way.
Performance Testing and Its Key Performance Metrics
Performance testing is focused on three S's: speed, scalability, and stability. Unlike with usual quality assurance, we're looking for performance bottlenecks here rather than bugs. If the app runs slow or behaves inconsistently under expected workload, performance testing will reveal that.
At the EGO QA/testing services company, we have three key metrics used for measuring the web project:
- Number of requests per unit of time
- Amount of data per unit of time
- Percentage of errors
However, you can't define if the project is medium or high load based on this metrics only since request processing varies for different projects. As for EGO, a project is usually high load when its existing infrastructure can no longer cope with its daily tasks and requires non-trivial solutions.
Performance testing is required to test the system, server, or infrastructure efficiency under different conditions and loads. With it, you can prepare your application for the projected growth in the number of users.
Performance testing usually includes:
- Stress testing
- Load testing
- Configuration testing
- Stability testing
More rare cases include endurance, spike and volume testing.
Once such an extensive testing is done, you can analyze the results and update the system to increase its performance.
Performance Testing Tools EGO Uses
Below we'll list our most used tools for performance testing on the mid- and high-load projects and give short overviews of their pros and cons.
Apache Jmeter
This is one of the most popular tools for performance management and load testing. It'll suit for the project of any size, has lots of features and its customizable.
Pros:
- free and open source
- cross-platform (thanks to Java)
- easy to install and run
- user friendly UI
- test results are displayed visually
- HTTP, LDAP< JDBC, SOAP, JMC and FTP protocol support
- allows writing test cases
Cons:
- no JavaScript support
- large tests may not perform well every time
Grinder
One more free load testing framework. Based Java and Jyton (Java + Python), it also utilizes Clojure for scripting.
Teams of any size comfortable with writing scripts in Jyton/Closure usually find Grinder their best choice.
Pros:
- Free
- Works in any environment with Java virtual machine
- Easy to set parameters
- Works with many protocols
- Able to run third party libraries
Cons:
- You can't regulate load levels via its console
- Uploading large files can cause issues
- No advanced analysis available
Gatling
Written in Scala, Gatling is a powerful tool for load testing.
Pros:
- free and open source
- automatic report generation in the table form
- supports build automation tools
- interacts with Jenkins
- high performance with high loads
Cons:
- the load can't be shared between devices
- script recorder could be better
- you need to know Scala
Webload
Webload is an enterprise-level solution able to generate complex real-world load scenarios for stress testing. Works for both web and mobile apps and will fit companies of any size.
Pros:
- Provides cloud deployment
- Performance analytics is available
- Supports many web technologies
- Script creation is rich
- Automatic bottleneck detection
Cons:
- Some might consider it expensive
LoadNinja
We find it great as it doesn't require creating test scripts and thus saves us time. Suits medium and large companies best due to its cost.
Pros:
- cloud-based
- tests with many real browsers
- quick browser-based metrics retrieval
- real-time testing control functions
Cons:
- require JavaScript to be enabled
- cost
The above-mentioned tools are the ones we use at our penetration testing company, so you might find other companies have different lists.
EGO's Experience in Performance Testing
At EGO Creative Innovations, we do performance testing on most projects. Prior to the first release we prefer to make sure there are no long load and response times and no performance bottlenecks.
One of the first tools coming to our mind is Jmeter. With it, we've been counting how many users can log in and tap buttons at the same time. On Idencia's project we also measured synchronization speed between the web and mobile apps, and with another internal mobile app for corporate event management Jmeter helped measure the performance of the app screens loading at the same time.
On the project with the photo uploading functionality, we used a tool calculating device memory.
One of the most sophisticated performance testing approaches is deployed on the Amadeus project, a property management solution. There, we create automated performance test suites. They are launched on a virtual machine with enough resources, and this machine is in the Amadeus' network to eliminate network latency. That's how we calculate the performance threshold.
Usually we create a baseline performance test for every service. When changes potentially impacting performance are introduced to a service, this test is re-run to check if the performance hasn't dropped.
When applications don't need auto performance testing, our QA engineers check loading pages and pagination functionality and do other stuff that can be usually categorized under the umbrella term of "light performance testing".
At EGO, we also use Lighthouse for quick performance checks.
Bottom Line: Performance Testing in 2021
The COVID-19 pandemic brought even more users and businesses to the Internet. Because of that the load on popular web resources is expected to continue growing. Performance testing will remain necessary to maintain and increase the profitability of online businesses and quality assurance \ software testing company services will be in demand.
Although there is no shortage of performance testing tools, in 2021 companies will continue to prefer free and open source tools that can be modified according to their business needs. The highest demand will be for universal tools able to test both mobile and web applications. And since it's cumbersome to organize load testing via real devices, the popularity of cloud load testing tools will also grow.
But this is generalization. There's no way to figure out the best performance testing approach for your product without a deep look into it. And this is where's the problem: it seems like outsourcing quality assurance is unpredictable in terms of time and cost. To address that, we recently introduced our quality assurance plans for web and mobile. Contact us to learn more.
