Cell Phone Software Forum
| View previous topic :: View next topic |
| Author |
Message |
David Lightstone
Guest
|
 Posted: Thu May 24, 2007 5:59 pm Post subject: Testing in large scale projects |
 |
|
You can treat this as a troll (isn't everything posted here a troll of one
sort or another)
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level, so
no sense in wandering down that road. (I'm sure that some will try to do so)
To make matters a bit more interesting assume that the organization is 3 - 7
% understaffed (so the effort definitely is an aspect of a death march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need of
being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of tracing
mechanism that you want.
Subquestion - when you are done do you have verification of validation of
the requirements? |
|
| Back to top |
|
 |
| |
Ads |
Advertising
Sponsor
|
|
H. S. Lahman
Guest
|
| Posted: Thu May 24, 2007 8:16 pm Post subject: Re: Testing in large scale projects |
|
|
Responding to Lightstone...
| Quote: |
You can treat this as a troll (isn't everything posted here a troll of one
sort or another)
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level, so
no sense in wandering down that road. (I'm sure that some will try to do so)
|
One might argue that TDD is problematic at the system level, but I don't
see a problem with test-first. When one writes the tests and their
content seem like orthogonal issues. As a practical matter, if the shop
has an independent SQA group, their system tests will almost always be
designed before the system is close to ready for testing.
| Quote: |
To make matters a bit more interesting assume that the organization is 3 - 7
% understaffed (so the effort definitely is an aspect of a death march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need of
being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of tracing
mechanism that you want.
Subquestion - when you are done do you have verification of validation of
the requirements?
|
I am not sure what the point is here (i.e., how this relates to
test-first or staffing constraints). Requirements traceability is
largely orthogonal to design verification or requirements validation.
(In your opening you specified that the requirements had already been
decomposed to the unit test level, so somebody must have understood how
they trace though the design at the Systems Engineering level.)
*************
There is nothing wrong with me that could
not be cured by a capful of Drano.
H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH |
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
David Lightstone
Guest
|
| Posted: Fri May 25, 2007 3:12 am Post subject: Re: Testing in large scale projects |
|
|
"H. S. Lahman" <h.lahman@verizon.net> wrote in message
news:HTi5i.8393$TU1.519@trnddc07...
| Quote: |
Responding to Lightstone...
You can treat this as a troll (isn't everything posted here a troll of
one sort or another)
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level,
so no sense in wandering down that road. (I'm sure that some will try to
do so)
One might argue that TDD is problematic at the system level, but I don't
see a problem with test-first. When one writes the tests and their content
seem like orthogonal issues. As a practical matter, if the shop has an
independent SQA group, their system tests will almost always be designed
before the system is close to ready for testing.
|
You assume the "one writes the tests and their content seem like orthogonal
issues". Why is it that you believe such an assumption is reasonable?
| Quote: |
To make matters a bit more interesting assume that the organization is
3 - 7 % understaffed (so the effort definitely is an aspect of a death
march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need
of being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of tracing
mechanism that you want.
Subquestion - when you are done do you have verification of validation of
the requirements?
I am not sure what the point is here (i.e., how this relates to test-first
or staffing constraints). Requirements traceability is largely orthogonal
to design verification or requirements validation. (In your opening you
specified that the requirements had already been decomposed to the unit
test level, so somebody must have understood how they trace though the
design at the Systems Engineering level.)
|
Requirements having been decomposed and trace relations having been
established amongst the requirements, does not imply any of the following
(1) Requirements are complete
(2) Requirement are consistent
Ergo one must distinguish between efforts to confirm that the both design
and implementation is consistenty with the requirements, and that the
requirements themselves are correct. Verification vs validation
|
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
Guest
|
| Posted: Fri May 25, 2007 4:42 pm Post subject: Re: Testing in large scale projects |
|
|
In article <gTg5i.2730$u56.947@newssvr22.news.prodigy.net>, David Lightstone
says...
| Quote: |
You can treat this as a troll (isn't everything posted here a troll of one
sort or another)
|
But feeding trolls makes them nice and tender when you roast them... :-)
| Quote: |
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level, so
no sense in wandering down that road. (I'm sure that some will try to do so)
|
I am interested in this assertion: if it is too complex to test-first, how can
it be any easier to test-last?
For instance, if my requirement is is
R42: Transaction commitment, after hitting "save" should return <1sec for 99% of
cases.
Then I can directly write a test case.
Perhaps I am missing a deeper point about your assumed complexity in the system?
| Quote: |
To make matters a bit more interesting assume that the organization is 3 - 7
% understaffed (so the effort definitely is an aspect of a death march)
|
All the more reason to decide if you want 100% working of 50% of the
requirements or 95% working (i.e. not working well enough to roll out) for 95%
of the requirements.
If you sell to management - and are successful in your sales pitch! - that you
will create working software, that can deploy at any incremental time, they will
likely gain enough confidence to keep the project going until a logical
completion time.
If on the other hand, there are some set of external constraints that were not
elaborated in your initial mail, as to who made the commitment with 5%
understaffed and why they did that, then you may well be in for a rough ride.
Good Luck!
sdg |
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
David Lightstone
Guest
|
| Posted: Fri May 25, 2007 6:13 pm Post subject: Re: Testing in large scale projects |
|
|
<scottgregory@yahoo.com> wrote in message
news:f36ljg01a0t@drn.newsguy.com...
| Quote: |
In article <gTg5i.2730$u56.947@newssvr22.news.prodigy.net>, David
Lightstone
says...
You can treat this as a troll (isn't everything posted here a troll of one
sort or another)
But feeding trolls makes them nice and tender when you roast them... :-)
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level,
so
no sense in wandering down that road. (I'm sure that some will try to do
so)
I am interested in this assertion: if it is too complex to test-first,
how can
it be any easier to test-last?
|
There is a difference between possible and impossible to consider. Comparing
something that is impossible with something that is possible for purposes of
deciding which is easier is a no-brainer. Did you assume that it was
possible to test first?
| Quote: |
For instance, if my requirement is is
R42: Transaction commitment, after hitting "save" should return <1sec for
99% of
cases.
Then I can directly write a test case.
|
What must be built before you can run your test? How do you know it was
built? How do you know the preconditions for running your test work
correctly?
What must be tracked for this to happen? Note this applies to the test
fixtureing as well as the test subject.
| Quote: |
Perhaps I am missing a deeper point about your assumed complexity in the
system?
To make matters a bit more interesting assume that the organization is 3 -
7
% understaffed (so the effort definitely is an aspect of a death march)
All the more reason to decide if you want 100% working of 50% of the
requirements or 95% working (i.e. not working well enough to roll out) for
95%
of the requirements.
If you sell to management - and are successful in your sales pitch! - that
you
will create working software, that can deploy at any incremental time,
they will
likely gain enough confidence to keep the project going until a logical
completion time.
If on the other hand, there are some set of external constraints that were
not
elaborated in your initial mail, as to who made the commitment with 5%
understaffed and why they did that, then you may well be in for a rough
ride.
Good Luck!
sdg
|
|
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
H. S. Lahman
Guest
|
| Posted: Fri May 25, 2007 7:53 pm Post subject: Re: Testing in large scale projects |
|
|
Responding to Lightstone...
| Quote: |
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level,
so no sense in wandering down that road. (I'm sure that some will try to
do so)
One might argue that TDD is problematic at the system level, but I don't
see a problem with test-first. When one writes the tests and their content
seem like orthogonal issues. As a practical matter, if the shop has an
independent SQA group, their system tests will almost always be designed
before the system is close to ready for testing.
You assume the "one writes the tests and their content seem like orthogonal
issues". Why is it that you believe such an assumption is reasonable?
|
You forgot to include the 'when'. I believe it because I see no logical
connection between when one writes tests (before or after developing the
software) and what the tests actually test. IOW, one can do test-first
for /any/ application but the test content will be specific to the
application.
| Quote: |
To make matters a bit more interesting assume that the organization is
3 - 7 % understaffed (so the effort definitely is an aspect of a death
march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need
of being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of tracing
mechanism that you want.
Subquestion - when you are done do you have verification of validation of
the requirements?
I am not sure what the point is here (i.e., how this relates to test-first
or staffing constraints). Requirements traceability is largely orthogonal
to design verification or requirements validation. (In your opening you
specified that the requirements had already been decomposed to the unit
test level, so somebody must have understood how they trace though the
design at the Systems Engineering level.)
Requirements having been decomposed and trace relations having been
established amongst the requirements, does not imply any of the following
(1) Requirements are complete
(2) Requirement are consistent
Ergo one must distinguish between efforts to confirm that the both design
and implementation is consistenty with the requirements, and that the
requirements themselves are correct. Verification vs validation
|
While I agree that verification and validation are desirable, I see that
as pretty much Motherhood and Apple Pie. I still don't see the
connection between test-first/resources and
traceability/Doors/verification/validation (other than programmatic
testing is one tool for verification/validation). Nor do I see a
connection between test-first/resources and whether requirements are
complete and consistent. So I'm afraid I still don't understand what
your point is vis a vis the message subject.
Note, BTW, that programmatic testing does not address (1) at all and it
addresses (2) only very indirectly. Testing can be used to verify the
developer's design implementation (the product is what the developer
intended to build) or validate the software product against existing
requirements (the product is what the customer wanted built), but it
doesn't say much about the quality of the requirements that describe
what the customer wanted built.
*************
There is nothing wrong with me that could
not be cured by a capful of Drano.
H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH |
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
David Lightstone
Guest
|
| Posted: Sat May 26, 2007 1:49 am Post subject: Re: Testing in large scale projects |
|
|
"H. S. Lahman" <h.lahman@verizon.net> wrote in message
news:QDD5i.1780$XC3.1415@trnddc04...
| Quote: |
Responding to Lightstone...
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level,
so no sense in wandering down that road. (I'm sure that some will try to
do so)
One might argue that TDD is problematic at the system level, but I don't
see a problem with test-first. When one writes the tests and their
content seem like orthogonal issues. As a practical matter, if the shop
has an independent SQA group, their system tests will almost always be
designed before the system is close to ready for testing.
You assume the "one writes the tests and their content seem like
orthogonal issues". Why is it that you believe such an assumption is
reasonable?
You forgot to include the 'when'. I believe it because I see no logical
connection between when one writes tests (before or after developing the
software) and what the tests actually test. IOW, one can do test-first for
/any/ application but the test content will be specific to the
application.
|
I agree one can write tests at any time. Being able to implement them is a
separate question. The test first ideology implies immediate implementation
(assuming that the mantra have not reciently changed)
| Quote: |
To make matters a bit more interesting assume that the organization is
3 - 7 % understaffed (so the effort definitely is an aspect of a death
march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can
be tracked (progress)? i.e. mechanisms to both organize the material in
need of being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of
tracing mechanism that you want.
Subquestion - when you are done do you have verification of validation
of the requirements?
I am not sure what the point is here (i.e., how this relates to
test-first or staffing constraints). Requirements traceability is largely
orthogonal to design verification or requirements validation. (In your
opening you specified that the requirements had already been decomposed
to the unit test level, so somebody must have understood how they trace
though the design at the Systems Engineering level.)
Requirements having been decomposed and trace relations having been
established amongst the requirements, does not imply any of the following
(1) Requirements are complete
(2) Requirement are consistent
Ergo one must distinguish between efforts to confirm that the both design
and implementation is consistenty with the requirements, and that the
requirements themselves are correct. Verification vs validation
While I agree that verification and validation are desirable, I see that
as pretty much Motherhood and Apple Pie.
|
My intent was to explore the difference between the two. Whether the
selected testing strategy (choosen arbritarily by respondent) coupled with
the tracdeability would yield any meaningful conclusions as to whether
verification or validation had been achieved.
More importantly howe ver there was a significant logistics issue relating
to whether the perconditions for test performance had been achieved. These
logistics would relate to if and when the various implementation activities
completed. In a large system (and my interest was large systems) the
coordination aspects can become non-trivial
| Quote: |
I still don't see the connection between test-first/resources and
traceability/Doors/verification/validation (other than programmatic
testing is one tool for verification/validation). Nor do I see a
connection between test-first/resources and whether requirements are
complete and consistent. So I'm afraid I still don't understand what your
point is vis a vis the message subject.
Note, BTW, that programmatic testing does not address (1) at all and it
addresses (2) only very indirectly. Testing can be used to verify the
developer's design implementation (the product is what the developer
intended to build) or validate the software product against existing
requirements (the product is what the customer wanted built), but it
doesn't say much about the quality of the requirements that describe what
the customer wanted built.
|
On this I disagree. Verification simply states there is consistency between
a description and something that was built.
Validation says there is consistency between the description and what the
customer needed
|
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
Marra
Guest
|
| Posted: Sat May 26, 2007 3:54 am Post subject: Re: Testing in large scale projects |
|
|
After 30 years as a programmer I have found that testing us just down
to hard graft.
If you dont spend as much time testing as you did writing the software
then you are gonna have problems.
Testing:
1/ Try to make the software do what you want it to do
2/ Try to make the software do things wrong.
3/ Try all possible inputs and see if the outputs make sense.
4/ Now go back and do the same thing with a different person testing.
5/ Now go back and get a novice to test it. |
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
Gabriel Claramunt
Guest
|
| Posted: Sat May 26, 2007 6:50 am Post subject: Re: Testing in large scale projects |
|
|
A completelly side note:
7% understaffed is nothing.. wait till you have to work at 50% understaffed
Now, seems to me that 5%-7% falls pretty well under the uncertiainity of the
estimations... how do you know that you're not overestimated by a 5%? 
--
Gabriel Claramunt
http://gabrielsw.blogspot.com
"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:gTg5i.2730$u56.947@newssvr22.news.prodigy.net...
| Quote: |
You can treat this as a troll (isn't everything posted here a troll of one
sort or another)
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level,
so no sense in wandering down that road. (I'm sure that some will try to
do so)
To make matters a bit more interesting assume that the organization is 3 -
7 % understaffed (so the effort definitely is an aspect of a death march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need
of being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of tracing
mechanism that you want.
Subquestion - when you are done do you have verification of validation of
the requirements?
|
|
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
David Lightstone
Guest
|
| Posted: Sat May 26, 2007 4:12 pm Post subject: Re: Testing in large scale projects |
|
|
"Gabriel Claramunt" <gabclar@internet.com.uy> wrote in message
news:1gN5i.2623$FN5.1617@bignews7.bellsouth.net...
| Quote: |
A completelly side note:
7% understaffed is nothing.. wait till you have to work at 50%
understaffed
Now, seems to me that 5%-7% falls pretty well under the uncertiainity of
the estimations... how do you know that you're not overestimated by a
5%?
|
The estimate is based upon the size of a corporation in which I once worked,
the number of positions they currently are seeking to fill, the number of
positions they are rumored to have filled within the last 8 months, and the
attitude of the people with whom I have worked.
If 5 - 7% is not to your liking (and your reason is quite plausible) roughly
double it to 10% to 15%
Coordination activities deteriorate as the number of understaff increases.
Put out the fire becomes the modus operandi. Engineering discipline breaks
down as deadlines approach.
The original question stands
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need
of being tested and track development progress.
| Quote: |
--
Gabriel Claramunt
http://gabrielsw.blogspot.com
"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:gTg5i.2730$u56.947@newssvr22.news.prodigy.net...
You can treat this as a troll (isn't everything posted here a troll of
one sort or another)
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level,
so no sense in wandering down that road. (I'm sure that some will try to
do so)
To make matters a bit more interesting assume that the organization is
3 - 7 % understaffed (so the effort definitely is an aspect of a death
march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need
of being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of tracing
mechanism that you want.
Subquestion - when you are done do you have verification of validation of
the requirements?
|
|
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
Gabriel Claramunt
Guest
|
| Posted: Sun May 27, 2007 7:27 am Post subject: Re: Testing in large scale projects |
|
|
The project already started? In the middle? Trying to finish? Not started
yet?
If is a big project, an iterative-incremental process will help... something
along the lines of RUP maybe.
You will iterations to have short term goals and working software to
evaluate at the end of each one. The length will be determined by the team
size: you need to iterate as fast as possible, but big teams have more
communication/synchronization overhead.
With the focus on testing/tracking, a good idea is for each iteration use
something like release plan and release specification: the release plan
contains the requirements planned to implement on the iteration so the
testing team can start preparing the tests and the release specification is
the list of the requirements that actually got implemented and passed the
tests at the end of the iteration.
--
Gabriel Claramunt
http://gabrielsw.blogspot.com
"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:9vV5i.7077$4Y.637@newssvr19.news.prodigy.net...
| Quote: |
"Gabriel Claramunt" <gabclar@internet.com.uy> wrote in message
news:1gN5i.2623$FN5.1617@bignews7.bellsouth.net...
A completelly side note:
7% understaffed is nothing.. wait till you have to work at 50%
understaffed
Now, seems to me that 5%-7% falls pretty well under the uncertiainity of
the estimations... how do you know that you're not overestimated by a
5%? :-)
The estimate is based upon the size of a corporation in which I once
worked, the number of positions they currently are seeking to fill, the
number of positions they are rumored to have filled within the last 8
months, and the attitude of the people with whom I have worked.
If 5 - 7% is not to your liking (and your reason is quite plausible)
roughly double it to 10% to 15%
Coordination activities deteriorate as the number of understaff increases.
Put out the fire becomes the modus operandi. Engineering discipline breaks
down as deadlines approach.
The original question stands
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need
of being tested and track development progress.
--
Gabriel Claramunt
http://gabrielsw.blogspot.com
"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:gTg5i.2730$u56.947@newssvr22.news.prodigy.net...
You can treat this as a troll (isn't everything posted here a troll of
one sort or another)
In a large project (3 - 5 layers of requirements decomposition) there is
need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit level,
so no sense in wandering down that road. (I'm sure that some will try to
do so)
To make matters a bit more interesting assume that the organization is
3 - 7 % understaffed (so the effort definitely is an aspect of a death
march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can
be tracked (progress)? i.e. mechanisms to both organize the material in
need of being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of
tracing mechanism that you want.
Subquestion - when you are done do you have verification of validation
of the requirements?
|
|
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
David Lightstone
Guest
|
| Posted: Sun May 27, 2007 5:28 pm Post subject: Re: Testing in large scale projects |
|
|
"Gabriel Claramunt" <gabclar@internet.com.uy> wrote in message
news:DU66i.2373$YM5.934@bignews2.bellsouth.net...
| Quote: |
The project already started? In the middle? Trying to finish? Not started
yet?
|
This information is really not relevant. You are at liberty to make any
assumptions that you wish. (subject to the constraint that the organization
is understaffed by at least 10 - 15 %, that it is large scale having 3 - 5
layers of requirements decomposition)
| Quote: |
If is a big project, an iterative-incremental process will help...
something along the lines of RUP maybe.
You will iterations to have short term goals and working software to
evaluate at the end of each one. The length will be determined by the team
size: you need to iterate as fast as possible, but big teams have more
communication/synchronization overhead.
|
This is software project managment mantra. True in the abstract, but
meaningless in the practical. I'm certain that they were doing this. The
only question is the granularity of the increments and the proberbial. It is
really difficult making such plans when the time scale is to short (say the
proverbial extreme programming 2 weeks). You just never know what will be
ready for integration (ergo why I indicated in the initial post test first
was not appropriate at any level other than unit test)
| Quote: |
With the focus on testing/tracking, a good idea is for each iteration use
something like release plan and release specification: the release plan
contains the requirements planned to implement on the iteration so the
testing team can start preparing the tests and the release specification
is the list of the requirements that actually got implemented and passed
the tests at the end of the iteration.
|
As background for may specific task my initial plan called for developing a
taxonomy of the tests (ie both a test menu and a big glorified test
developement work break down, cutting accross the levels in the requirements
decomposition) The idea was to review it for correctness and completion.
Then triage the review results against the testing efforts of other groups
and organizations. Dead end, got the menu, could not get the review. In a
sense it was a duplication of effort. No time to duplicate. Tracking
strategies such as that one would not work.
| Quote: |
--
Gabriel Claramunt
http://gabrielsw.blogspot.com
"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in message
news:9vV5i.7077$4Y.637@newssvr19.news.prodigy.net...
"Gabriel Claramunt" <gabclar@internet.com.uy> wrote in message
news:1gN5i.2623$FN5.1617@bignews7.bellsouth.net...
A completelly side note:
7% understaffed is nothing.. wait till you have to work at 50%
understaffed
Now, seems to me that 5%-7% falls pretty well under the uncertiainity of
the estimations... how do you know that you're not overestimated by a
5%? :-)
The estimate is based upon the size of a corporation in which I once
worked, the number of positions they currently are seeking to fill, the
number of positions they are rumored to have filled within the last 8
months, and the attitude of the people with whom I have worked.
If 5 - 7% is not to your liking (and your reason is quite plausible)
roughly double it to 10% to 15%
Coordination activities deteriorate as the number of understaff
increases. Put out the fire becomes the modus operandi. Engineering
discipline breaks down as deadlines approach.
The original question stands
How should the development effort be structured so that the effort can
be
tracked (progress)? i.e. mechanisms to both organize the material in need
of being tested and track development progress.
--
Gabriel Claramunt
http://gabrielsw.blogspot.com
"David Lightstone" <david._NoSpamlightstone@prodigy.net> wrote in
message news:gTg5i.2730$u56.947@newssvr22.news.prodigy.net...
You can treat this as a troll (isn't everything posted here a troll of
one sort or another)
In a large project (3 - 5 layers of requirements decomposition) there
is need for unit testing, sub-system testing and system testing.
Ideally you would like a strong correlation between the tests and the
requirements with which they are associated.
It is too complex for test first to work at anything but the unit
level, so no sense in wandering down that road. (I'm sure that some
will try to do so)
To make matters a bit more interesting assume that the organization is
3 - 7 % understaffed (so the effort definitely is an aspect of a death
march)
So now for the subjects of amusement
How should the development effort be structured so that the effort can
be tracked (progress)? i.e. mechanisms to both organize the material in
need of being tested and track development progress.
Assume Doors as a requirement management tool. Set up any sort of
tracing mechanism that you want.
Subquestion - when you are done do you have verification of validation
of the requirements?
|
|
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
H. S. Lahman
Guest
|
| Posted: Sun May 27, 2007 7:25 pm Post subject: Re: Testing in large scale projects |
|
|
Responding to Lightstone...
| Quote: |
Note, BTW, that programmatic testing does not address (1) at all and it
addresses (2) only very indirectly. Testing can be used to verify the
developer's design implementation (the product is what the developer
intended to build) or validate the software product against existing
requirements (the product is what the customer wanted built), but it
doesn't say much about the quality of the requirements that describe what
the customer wanted built.
On this I disagree. Verification simply states there is consistency between
a description and something that was built.
Validation says there is consistency between the description and what the
customer needed
|
I prefer the more conventional definitions where verification ensures
the implementation satisfies the software design specification for the
current development stage while validation ensures the final software
product satisfies the original customer requirements. Thus white box
unit tests are classic verification while independent black box system
tests are classic validation.
In either case I don't see how programmatic testing can address whether
the customer requirements are complete. One can only design programmatic
tests for the requirements or design in hand. OTOH, non-programmatic
testing, such as SRS reviews, can determine the completeness of the
requirements. In addition the design process itself can identify missing
requirements when ambiguities arise.
Similarly, determining whether requirements are consistent is not
directly addressable by executing programmatic tests. A programmatic
test just demonstrates that the actual results match the expected
results for a given set of test inputs. It doesn't say anything about
whether those expected results are consistent with another test's
expected results.
OTOH, designing a test /suite/ can indirectly identify inconsistencies
in the requirements -- if the test suite designer evaluates consistency
between expected results for different testing goals. But in practice
that is very rarely done since programmatic tests are almost always
designed independently on a requirement-by-requirement basis, quite
often by different designers. Instead one usually relies on
non-programmatic testing and the software design process itself to
uncover requirements inconsistencies.
*************
There is nothing wrong with me that could
not be cured by a capful of Drano.
H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH |
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
David Lightstone
Guest
|
| Posted: Mon May 28, 2007 10:40 pm Post subject: Re: Testing in large scale projects |
|
|
"H. S. Lahman" <h.lahman@verizon.net> wrote in message
news:9qh6i.2118$9G3.873@trnddc07...
| Quote: |
Responding to Lightstone...
Note, BTW, that programmatic testing does not address (1) at all and it
addresses (2) only very indirectly. Testing can be used to verify the
developer's design implementation (the product is what the developer
intended to build) or validate the software product against existing
requirements (the product is what the customer wanted built), but it
doesn't say much about the quality of the requirements that describe what
the customer wanted built.
On this I disagree. Verification simply states there is consistency
between a description and something that was built.
Validation says there is consistency between the description and what the
customer needed
I prefer the more conventional definitions where verification ensures the
implementation satisfies the software design specification for the current
development stage while validation ensures the final software product
satisfies the original customer requirements. Thus white box unit tests
are classic verification while independent black box system tests are
classic validation.
|
Whether these definitions are conventional or not I can not state. You are
more than welcome to use them as you please. Below I will provide a quote
and identfication of its source. (Obtained by a quick Google search on the
expression "software validation"). The reference is to the web page
http://www.fda.gov/cdrh/comp/guidance/938.html#_Toc517237938 The Quote from
that web page is
---- begin quote -----
Software verification provides objective evidence that the design outputs of
a particular phase of the software development life cycle meet all of the
specified requirements for that phase. Software verification looks for
consistency, completeness, and correctness of the software and its
supporting documentation, as it is being developed, and provides support for
a subsequent conclusion that software is validated. Software testing is one
of many verification activities intended to confirm that software
development output meets its input requirements. Other verification
activities include various static and dynamic analyses, code and document
inspections, walkthroughs, and other techniques.
Software validation is a part of the design validation for a finished
device, but is not separately defined in the Quality System regulation. For
purposes of this guidance, FDA considers software validation to be
"confirmation by examination and provision of objective evidence that
software specifications conform to user needs and intended uses, and that
the particular requirements implemented through software can be consistently
fulfilled." In practice, software validation activities may occur both
during, as well as at the end of the software development life cycle to
ensure that all requirements have been fulfilled. Since software is usually
part of a larger hardware system, the validation of software typically
includes evidence that all software requirements have been implemented
correctly and completely and are traceable to system requirements. A
conclusion that software is validated is highly dependent upon comprehensive
software testing, inspections, analyses, and other verification tasks
performed at each stage of the software development life cycle. Testing of
device software functionality in a simulated use environment, and user site
testing are typically included as components of an overall design validation
program for a software automated device.
---- end quote ----
| Quote: |
In either case I don't see how programmatic testing can address whether
the customer requirements are complete. One can only design programmatic
tests for the requirements or design in hand. OTOH, non-programmatic
testing, such as SRS reviews, can determine the completeness of the
requirements. In addition the design process itself can identify missing
requirements when ambiguities arise.
Similarly, determining whether requirements are consistent is not directly
addressable by executing programmatic tests. A programmatic test just
demonstrates that the actual results match the expected results for a
given set of test inputs. It doesn't say anything about whether those
expected results are consistent with another test's expected results.
OTOH, designing a test /suite/ can indirectly identify inconsistencies in
the requirements -- if the test suite designer evaluates consistency
between expected results for different testing goals. But in practice that
is very rarely done since programmatic tests are almost always designed
independently on a requirement-by-requirement basis, quite often by
different designers. Instead one usually relies on non-programmatic
testing and the software design process itself to uncover requirements
inconsistencies.
*************
There is nothing wrong with me that could
not be cured by a capful of Drano.
H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH
|
|
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
Vladimir Trushkin
Guest
|
| Posted: Tue May 29, 2007 5:21 pm Post subject: Re: Testing in large scale projects |
|
|
On May 24, 4:59 pm, "David Lightstone"
<david._NoSpamlightst...@prodigy.net> wrote:
| Quote: |
How should the development effort be structured so that the effort can be
tracked (progress)? i.e. mechanisms to both organize the material in need of
being tested and track development progress.
|
I think this is a reasonable desire to have all tests referencing to
requirements. At the very least there should be at least one test
checking whether or not a requirement is implemented correctly. I am
not sure about unit tests. I would by no mean try covering system
level requirements with developer tests because if requirements are
written in a good way and describing the business logic and unit tests
are aimed at testing paths through the code I see a disconnect here.
So, I would recommend covering system level requirements with system
level tests and designing requirements with unit tests.
Now to the point. The best way of tracking development of a complex
system is breaking it down into several parts. Each iteration shall
include full scale of testing, starting at unit level, then advancing
to integration testing and finally system testing. This approach
implies that you know when and which requirements will be ready for
testing. Some additional time shall be allocated at the backend of the
project to fit in integration and system testing for the complete
system.
| Quote: |
Assume Doors as a requirement management tool. Set up any sort of tracing
mechanism that you want.
|
Easy. Requirements have numbers. Test names have the requirement
number encoded, or if you have a system like TestDirector it's even
simpler, just import your reqs into TestDirector and reference your
tests to achieve the coverage goals.
| Quote: |
Subquestion - when you are done do you have verification of validation of
the requirements?
|
Verification of validation? Not quite get it. But if you mean whether
or not we look at the final results of requirements coverage with test
then yes, we do it.
-----
Best Wishes,
Vladimir |
|
| Back to top |
|
|
| |
Ads |
Advertising
Sponsor
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
191 Attacks blocked
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
