Types Of Test Automaton Frameworks

There are several test automation frameworks available, among these the selection is made based on the factors such as reusability of both the scripts and the test assets. The different test automation frameworks available are as follows,

Ø  Test Script Modularity

Ø  Test Library Architecture

Ø  Data-Driven Testing

Ø  Keyword-Driven or Table-Driven Testing

Ø  Hybrid Test Automation

1.2.1 Test Script Modularity

The test script modularity framework is the most basic of the frameworks. It's a well-known programming strategy to build an abstraction layer in front of a component to hide the component from the rest of the application. This insulates the application from modifications in the component and provides modularity in the application design. When working with test scripts (in any language or proprietary environment) this can be achieved by creating small, independent scripts that represent modules, sections, and functions of the application-under-test. Then these small scripts are taken and combined them in a hierarchical fashion to construct larger tests. The use of this framework will yield a higher degree of modularization and add to the overall maintainability of the test scripts.

1.2.2 Test Library Architecture

The test library architecture framework is very similar to the test script modularity framework and offers the same advantages, but it divides the application-under-test into procedures and functions (or objects and methods depending on the implementation language) instead of scripts. This framework requires the creation of library files (SQABasic libraries, APIs, DLLs, and such) that represent modules, sections, and functions of the application-under-test. These library files are then called directly from the test case script. Much like script modularization this framework also yields a high degree of modularization and adds to the overall maintainability of the tests.

1.2.3 Data-Driven Testing

A data-driven framework is where test input and output values are read from data files (ODBC sources, CVS files, Excel files, DAO objects, ADO objects, and such) and are loaded into variables in captured or manually coded scripts. In this framework, variables are used for both input values and output verification values. Navigation through the program, reading of the data files, and logging of test status and information are all coded in the test script. This is similar to table-driven testing (which is discussed shortly) in that the test case is contained in the data file and not in the script; the script is just a "driver," or delivery mechanism, for the data. In data-driven testing, only test data is contained in the data files.

1.2.3.1 Merits of data-driven testing

The merits of the Data-Driven test automation framework are as follows,

Ø  Scripts may be developed while application development is still in progress

Ø  Utilizing a modular design, and using files or records to both input and verify data, reduces redundancy and duplication of effort in creating automated test scripts

Ø  If functionality changes, only the specific "Business Function" script needs to be updated

Ø  Data input/output and expected results are stored as easily maintainable text records.

Ø  Functions return "TRUE" or "FALSE" values to the calling script, rather than aborting, allowing for more effective error handling, and increasing the robustness of the test scripts. This, along with a well-designed "recovery" routine, enables "unattended" execution of test scripts.

1.2.3.2 Demerits of data-driven testing

The demerits of the Data-Driven test automation framework are as follows,

Ø  Requires proficiency in the Scripting language used by the tool (technical personnel)

Ø  Multiple data-files are required for each Test Case. There may be any number of data-inputs and verifications required, depending on how many different screens are accessed. This usually requires data-files to be kept in separate directories by Test Case

Ø  Tester must not only maintain the Detail Test Plan with specific data, but must also re-enter this data in the various required data-files

Ø  If a simple "text editor" such as Notepad is used to create and maintain the data-files, careful attention must be paid to the format required by the scripts/functions that process the files, or script-processing errors will occur due to data-file format and/or content being incorrect

1.2.4 Keyword-Driven Testing

This requires the development of data tables and keywords, independent of the test automation tool used to execute them and the test script code that "drives" the application-under-test and the data. Keyword-driven tests look very similar to manual test cases. In a keyword-driven test, the functionality of the application-under-test is documented in a table as well as in step-by-step instructions for each test. In this method, the entire process is data-driven, including functionality.

1.2.4.1 Example

In order to open a window, the following table is devised, and it can be used for any other application, just it requires just changing the window name.

Test Table for Opening a Window

Window	Control	Action	Arguments
Window Name	Menu	Click	File, Open
Window Name	Menu	Click	Close
Window Name	Pushbutton	Click	Folder Name
Window Name		Verify	Results

Once creating the test tables, a driver script or a set of scripts is written that reads in each step executes the step based on the keyword contained the Action field, performs error checking, and logs any relevant information.

1.2.4.2 Merits of keyword driven testing

The merits of the Keyword Driven Testing are as follows,

Ø  The Detail Test Plan can be written in Spreadsheet format containing all input and verification data.

Ø  If "utility" scripts can be created by someone proficient in the automated tool’s Scripting language prior to the Detail Test Plan being written, then the tester can use the Automated Test Tool immediately via the "spreadsheet-input" method, without needing to learn the Scripting language.

Ø  The tester need only learn the "Key Words" required, and the specific format to use within the Test Plan. This allows the tester to be productive with the test tool very quickly, and allows more extensive training in the test tool to be scheduled at a more convenient time.

1.2.4.3 Demerits of keyword driven testing

The demerits of the Keyword Driven Testing are as follows,

Ø  Development of "customized" (Application-Specific) Functions and Utilities requires proficiency in the tool’s Scripting language. (Note that this is also true for any method)

Ø  If application requires more than a few "customized" Utilities, this will require the tester to learn a number of "Key Words" and special formats. This can be time-consuming, and may have an initial impact on Test Plan Development. Once the testers get used to this, however, the time required to produce a test case is greatly improved.

1.2.5 Hybrid Test Automation Framework

The most commonly implemented framework is a combination of all of the above techniques, pulling from their strengths and trying to mitigate their weaknesses. This hybrid test automation framework is what most frameworks evolve into over time and multiple projects. The most successful automation frameworks generally accommodate both Keyword-Driven testing as well as Data-Driven scripts.

This allows data driven scripts to take advantage of the powerful libraries and utilities that usually accompany a keyword driven architecture. The framework utilities can make the data driven scripts more compact and less prone to failure than they otherwise would have been.

The utilities can also facilitate the gradual and manageable conversion of existing scripts to keyword driven equivalents when and where that appears desirable. On the other hand, the framework can use scripts to perform some tasks that might be too difficult to re-implement in a pure keyword driven approach, or where the keyword driven capabilities are not yet in place. The following sections describe its architecture, merits and demerits.

Automated Testing Best Practices

Introduction:

There are a lot of reasons as to why automated testing is beneficial, and by using these best practices in your automated testing you can ensure that your testing is successful and you get the maximum return on investment (ROI).

1. Decide what Test Cases to Automate
2. Test Early and Test Often
3. Select the Right Automated Testing Tool
4. Divide your Automated Testing Efforts
5. Create Good, Quality Test Data
6. Create Automated Tests that are Resistant to Changes in the UI

1. Decide What Test Cases to Automate

It is impossible to automate all testing, so it is important to determine what test cases should be automated first.

The benefit of automated testing is linked to how many times a given test can be repeated. Tests that are only performed a few times are better left for manual testing. Good test cases for automation are ones that are run frequently and require large amounts of data to perform the same action.

You can get the most benefit out of your automated testing efforts by automating:

• Repetitive tests that run for multiple builds.
• Tests that tend to cause human error.
• Tests that require multiple data sets.
• Frequently used functionality that introduces high risk conditions.
• Tests that are impossible to perform manually.
• Tests that run on several different hardware or software platforms and configurations.
• Tests that take a lot of effort and time when manual testing.

Success in test automation requires careful planning and design work. Start out by creating an automation plan. This allows you to identify the initial set of tests to automate, and serve as a guide for future tests. First, you should define your goal for automated testing and determine which types of tests to automate. There are a few different types of testing, and each has its place in the testing process. For instance, unit testing is used to test a small part of the intended application. Load testing is performed when you need to know how a web service responds under a heavy workload. To test a certain piece of the application’s UI, you would use functional or GUI testing.

After determining your goal and which types of tests to automate, you should decide what actions your automated tests will perform. Don’t just create test steps that test various aspects of the application’s behavior at one time. Large, complex automated tests are difficult to edit and debug. It is best to divide your tests into several logical, smaller tests. It makes your test environment more coherent and manageable and allows you to share test code, test data and processes. You will get more opportunities to update your automated tests just by adding small tests that address new functionality. Test the functionality of your application as you add it, rather than waiting until the whole feature is implemented.

When creating tests, try to keep them small and focused on one objective. For example, separate tests for read-only versus read/write tests. This allows you to use these individual tests repeatedly without including them in every automated test.

Once you create several simple automated tests, you can group your tests into one, larger automated test. You can organize automated tests by the application’s functional area, major/minor division in the application, common functions or a base set of test data. If an automated test refers to other tests, you may need to create a test tree, where you can run tests in a specific order.

2. Test Early and Test Often

To get the most out of your automated testing, testing should be started as early as possible and ran as often as needed. The earlier testers get involved in the life cycle of the project the better, and the more you test, the more bugs you find. Automated unit testing can be implemented on day one and then you can gradually build your automated test suite. Bugs detected early are a lot cheaper to fix than those discovered later in production or deployment.

3. Select the Right Automated Testing Tool

Selecting an automated testing tool is essential for test automation. There are a lot of automated testing tools on the market, and it is important to choose the automated testing tool that best suits your overall requirements.

Consider these key points when selecting an automated testing tool:

• Support for your platforms and technology. Are you testing .Net, C# or WPF applications and on what operating systems?
• Flexibility for testers of all skill levels. Can your QA department write automated test scripts or is there a need for keyword testing?
• Feature rich but also easy to create automated tests. Does the automated testing tool support record-and-playback test creation as well as manual creation of automated tests; does it include features for implementing checkpoints to verify values, databases, or key functionality of your application?
• Create automated tests that are reusable, maintainable and resistant to changes in the applications UI. Will my automated tests break if my UI changes?

For detailed information about selecting automated testing tools for automated testing, see Selecting Automated Testing Tools.

4. Divide Your Automated Testing Efforts

Usually, the creation of different tests is based on the QA engineers’ skill levels. It is important to identify the level of experience and skills for each of your team members and divide your automated testing efforts accordingly. For instance, writing automated test scripts requires expert knowledge of scripting languages. Thus, in order to perform these tasks, you should have QA engineers that know the script language provided by the automated testing tool.

Some team members may not be versed in writing automated test scripts. These QA engineers may be better at writing test cases. It is better when an automated testing tool has a way to create automated tests that do not require an in-depth knowledge of scripting languages, like TestComplete’s keyword tests feature. A keyword test (also known as keyword-driven testing) is a simple series of keywords with a specified action. With keyword tests, you can simulate keystrokes, click buttons, select menu items, call object methods and properties, and do a lot more. Keyword tests are often seen as an alternative to automated test scripts. Unlike scripts, they can be easily used by technical and non-technical users and allow users of all levels to create robust and powerful automated tests.

You should also collaborate on your automated testing project with other QA engineers in your department. Testing performed by a team is more effective for finding defects and the right automated testing tool allows you to share your projects with several testers.

5. Create Good, Quality Test Data

Good test data is extremely useful for data-driven testing. The data that should be entered into input fields during an automated test is usually stored in an external file. This data might be read from a database or any other data source like text or XML files, Excel sheets, and database tables. A good automated testing tool actually understands the contents of the data files and iterates over the contents in the automated test. Using external data makes your automated tests reusable and easier to maintain. To add different testing scenarios, the data files can be easily extended with new data without needing to edit the actual automated test.

Typically, you create test data manually and then save it to the desired data storage. However, TestComplete provides you with the Data Generator that assists you in creating Table variables and Excel files that store test data. This approach lets you generate data of the desired type (integer numbers, strings, boolean values and so on) and automatically save this data to the specified variable or file. Using this feature, you decrease the time spent on preparing test data for data-driven tests. For more information on generating test data with TestComplete, see the Using Data Generators section in TestComplete’s help.

Creating test data for your automated tests is boring, but you should invest time and effort into creating data that is well structured. With good test data available, writing automated tests becomes a lot easier. The earlier you create good-quality data, the easier it is to extend existing automated tests along with the application's development.

6. Create Automated Tests that are Resistant to Changes in the UI

Automated tests created with scripts or keyword tests are dependent on the application under test. The user interface of the application may change between builds, especially in the early stages. These changes may affect the test results, or your automated tests may no longer work with future versions of the application. The problem is automated testing tools use a series of properties to identify and locate an object. Sometimes a testing tool relies on location coordinates to find the object. For instance, if the control caption or its location has changed, the automated test will no longer be able to find the object when it runs and will fail. To run the automated test successfully, you may need to replace old names with new ones in the entire project, before running the test against the new version of the application. However, if you provide unique names for your controls, it makes your automated tests resistant to these UI changes and ensures that your automated tests work without having to make changes to the test itself. This also eliminates the automated testing tool from relying on location coordinates to find the control, which is less stable and breaks easily.

Test Automation for Web Applications

Test Automation for Web Applications

Mostly, software applications today are written as web-based applications to be run in an Internet browser.

The effectiveness of testing these applications varies widely among companies and organizations. In an era of highly interactive and responsive software processes where many organizations are using some form of Agile methodology, test automation is frequently becoming a requirement for software projects.

Test automation means using a software tool to run repeatable tests against the application to be tested. For regression testing this provides that responsiveness.

Test automation has specific advantages for improving the long-term efficiency of a software team’s testing processes.

Test automation supports:

• Frequent regression testing 
• Rapid feedback to developers 
• Virtually unlimited iterations of test case execution 
• Support for Agile and extreme development methodologies 
• Disciplined documentation of test cases 
• Customized defect reporting 
• Finding defects missed by manual testing

Why Automated Testing?

Why Automated Testing?

Every software development group tests its products, yet delivered software always has defects. Test engineers strive to catch them before the product is released but they always creep in and they often reappear, even with the best manual testing processes. 

Automated software testing is the best way to increase the effectiveness, efficiency and coverage of your software testing.

Manual software testing is performed by a human sitting in front of a computer carefully going through application screens, trying various usage and input combinations, comparing the results to the expected behavior and recording their observations. Manual tests are repeated often during development cycles for source code changes and other situations like multiple operating environments and hardware configurations.

An automated software testing tool is able to playback pre-recorded and predefined actions, compare the results to the expected behavior and report the success or failure of these manual tests to a test engineer. Once automated tests are created they can easily be repeated and they can be extended to perform tasks impossible with manual testing. Because of this, savvy managers have found that automated software testing is an essential component of successful development projects.

Automated software testing has long been considered critical for big software development organizations but is often thought to be too expensive or difficult for smaller companies to implement. SmartBear’s TestCompleteis affordable enough for single developer shops and yet powerful enough that our customer list includes some of the largest and most respected companies in the world.

Companies like Corel, Intel, Adobe, Autodesk, Intuit, McDonalds, Motorola, Symantec and Sony all use TestComplete.

What makes automated software testing so important to these successful companies?

1.      Automated Software Testing Saves Time and Money

Software tests have to be repeated often during development cycles to ensure quality. Every time source code is modified software tests should be repeated. For each release of the software it may be tested on all supported operating systems and hardware configurations. Manually repeating these tests is costly and time consuming. Once created, automated tests can be run over and over again at no additional cost and they are much faster than manual tests. Automated software testing can reduce the time to run repetitive tests from days to hours. A time savings that translates directly into cost savings.

2.      Automated Software Testing Improves Accuracy

Even the most conscientious tester will make mistakes during monotonous manual testing. Automated tests perform the same steps precisely every time they are executed and never forget to record detailed results.

3.      Automated Software Testing Increases Test Coverage

Automated software testing can increase the depth and scope of tests to help improve software quality. Lengthy tests that are often avoided during manual testing can be run unattended. They can even be run on multiple computers with different configurations. Automated software testing can look inside an application and see memory contents, data tables, file contents, and internal program states to determine if the product is behaving as expected. Automated software tests can easily execute thousands of different complex test cases during every test run providing coverage that is impossible with manual tests. Testers freed from repetitive manual tests have more time to create new automated software tests and deal with complex features.

4.      Automated Software Testing Does What Manual Testing Cannot

Even the largest software departments cannot perform a controlled web application test with thousands of users. Automated testing can simulate tens, hundreds or thousands of virtual users interacting with network or web software and applications.

5.      Automated Software Testing Helps Developers and Testers

Shared automated tests can be used by developers to catch problems quickly before sending to QA. Tests can run automatically whenever source code changes are checked in and notify the team or the developer if they fail. Features like these save developers time and increase their confidence.

6.      Automated Software Testing Improves Team Morale

This is hard to measure but we’ve experienced it first hand, automated software testing can improve team morale. Automating repetitive tasks with automated software testing gives your team time to spend on more challenging and rewarding projects. Team members improve their skill sets and confidence and, in turn, pass those gains on to their organization.

WHAT IS AUTOMATION?

WHAT IS AUTOMATION?

Automation is the use of control systems and information technologies to reduce the need for human work in the production of goods and services. In the scope of industrialization, automation is a step beyond mechanization. Whereas mechanization provided human operators with machinery to assist them with the muscular requirements of work, automation greatly decreases the need for human sensory and mental requirements as well. Automation plays an increasingly important role in the world economy and in daily experience.

Automation has had a notable impact in a wide range of industries beyond manufacturing (where it began). Once-ubiquitous telephone operators have been replaced largely by automated telephone switchboards and answering machines. Medical processes such as primary screening in electrocardiography or radiography and laboratory analysis of human genes, sera, cells, and tissues are carried out at much greater speed and accuracy by automated systems. Automated teller machines have reduced the need for bank visits to obtain cash and carry out transactions. In general, automation has been responsible for the shift in the world economy from industrial jobs to service jobs in the 20th and 21st centuries.

Advantages:

§  Replacing human operators in tasks that involve hard physical or monotonous work.

§  Replacing humans in tasks done in dangerous environments (i.e. fire, space, volcanoes, nuclear facilities, underwater, etc.)

§  Performing tasks that are beyond human capabilities of size, weight, speed, endurance, etc.

§  Economy improvement: Automation may improve in economy of enterprises, society or most of humanity.

Disadvantages:

§  Unemployment rate increases due to machines replacing humans and putting those humans out of their jobs.

§  Technical Limitation: Current technology is unable to automate all the desired tasks.

§  Security Threats/Vulnerability: An automated system may have limited level of intelligence, hence it is most likely susceptible to commit error.

§  Unpredictable development costs: The research and development cost of automating a process may exceed the cost saved by the automation itself.

High initial cost: The automation of a new product or plant requires a huge initial investment in comparison with the unit cost of the product, although the cost of automation is spread in many product batches.