Alsalam alikom wa ra7mat Allah wa barakatoh
Today, I was investigating -along with one of my friends- in Open Source GUI Testing tools... honestly there are quite a lot of them but non are smart enough.
Let's first define the goal of a GUI Testing Tool..
My Investigation scope didn't extend to commerical products (which I believe will have some more functionalities/polish interface...etc).
Here is a list of some apps that may cover some of these features for .NET Winforms:
As you can see, non of the above tools support our simple-7 rules, there are some available apps that uses Genetic algorithms to generate reasonable valid inputs but they do nothing but this :)
I believe the field is still widely open for those who are willing to contribute and put some innovative ideas over there.
Thanks,
Today, I was investigating -along with one of my friends- in Open Source GUI Testing tools... honestly there are quite a lot of them but non are smart enough.
Let's first define the goal of a GUI Testing Tool..
- Automated capture of different user scenarios,
- Generate editable script for different scenarios,
- Goal setting: define the success/failure state of a test run,
- Integratable with other Testing tools (NUnit, JUnit... etc) to provice a complete report about the test run,
- Requires 0-code changes,
- Can generate valid random inputs for testing (Textbox values mainly),
- Capture screenshots if something failed/succeeded/always,
My Investigation scope didn't extend to commerical products (which I believe will have some more functionalities/polish interface...etc).
Here is a list of some apps that may cover some of these features for .NET Winforms:
- ADLab Quail:
- Built on Windows Workflow Foundation technology
- No actions recording capabilities
- Able to take a snapshot of any control (a snapshot is a copy of the state of all/subset of properties)
- Goal setting by defining the desired state after specific action (button press ... etc)
- Requires adding 2 lines of code to the testee application to launch the testing tool
- NUnitForms:
- Integratable with NUnit famous unit testing tool,
- Supports recording actions and converting them to C# code
- Goal setting by adding assert statements
- Supports running script to automate testing (script and c# code are NOT interchangable)
- Can capture screenshots
- White:
- Easy to use .NET Lib for writing automated UI testing
- Easy to integrate with any Unit Testing Tool
- No recording
- No Screenshots
- Test Run:
- Free MSDN magazine code :)
- Lower level .NET code for writing automated UI testing (lower than White)
- No Recording
- No Screenshots
- A LOT of code
As you can see, non of the above tools support our simple-7 rules, there are some available apps that uses Genetic algorithms to generate reasonable valid inputs but they do nothing but this :)
I believe the field is still widely open for those who are willing to contribute and put some innovative ideas over there.
Thanks,
It looks like an "ideal" list :D
ReplyDelete- momken nsheel el point bta3et "Requires 0-code" ne7'alleha, requires less code...
in case of putting only assertion code that would be fine, (the other graphical option, is setting the form values, and clicking the "Valid" -button with a green check icon-, to identify a valid state, and vice-versa. I never heard about an easy tool :D )
- Integration is a perfect point, and even makes unit testing easier
- Generation of random inputs, looks be a "very" sophisticated option! :D ...not mandatory
Nize Bost
"ideal" list, that's exactly what it meant to be :D... I was trying what a GUI testing tool would look like in the perfect world...
ReplyDelete- 0-code is the optimal, less code is nice :D
I agree that currently we can deal with a few lines of code... :D
- ..
- Actually I read a paper (http://sigchi.org/chi96/proceedings/papers/Kasik/djk_txt.htm) which proposes to use Genetic Algorithms to do this heavy task..
I agree it's very sophisticated but this's how human needs evolve ;)
Thanks,
Modammed,
ReplyDeleteI'm an author of ADLab Quail.
Next release will have capturing screenshots build-in although it's easy to add it by yourself right now. Complete randomizing text inputs is more complicated and less useful then it's looks like, because you need to know which of random inputs suppose to fail (succeed), or which actions they need to cause.
I do, however, like idea of making a certain type of input easier (i.e. put random literal string to numeric only control etc). Not sure if it's needs to be a menu in design time or special activity.
As far as recording capabilities, I personally do not like it because it's makes too easy to create huge test scripts that clattered with unneeded code and, after changes in software, practically impossible to modify. I do prefer small, targeting scripts and Quail made just like I wanted it :)
Nevertheless, I really appreciate your input. if you think of any other features you would like to have please let me know.
Ilya
Ilya,
ReplyDeleteI would like to thank you for taking time to post a comment here.
- What I meant by capturing screenshot is not just a simple button to do it.. I meant to automatically capture the steps that lead to a Fail state (or success)... this way, it'll be easier to report a bug (along with the steps to regenerate it) later... if this can be added to the next release, it'll be a GR8 addition
- In some certain type of software, it's really important to automatically generating a LOT of valid inputs, to test performance/fault tolerenace... etc. I agree that generating all possible cases for the input may seem really complicated to achieve... and this is why men should do their job.
- Recording capabilities, your neat software doesn't generate scripts, that's why it's unique in design, what I imagine the recording will be is -thanks mohammed for the green icon thing-:
* I'll right click on my project in VS, choose Generate Quial Workflow
* My applicaiton will show up, with a red circle somewhere in the screen to tell me "I'm recording"
* I'll start reacting with the application in 2 ways:
*** Either by clicking a button/menu..etc and this will generate an Action Activity
*** Or, Selecting a control, contents, dialog then clicking the right green button or cross red button to indicate this is a good behaviour or a bad one..
* I'll stop the recording and the workflow will be generated for me
That may seem complicated but I feel it's still controllable and will enable us to easily generate test scripts...
I appreciate your reply,
Thanks,