SoftScience Group, Inc.

Problem Domain Expertise

White Papers

Problem Domain Expertise

By David Neal
President, SoftScience Group, Inc.

Problem domain expertise is just as important as technical platform expertise

The development of critical business software systems involves a variety of skills and knowledge on the part of the software developers. Obviously, technical competencies in software engineering, including analysis, design, programming, and database design are all important. But technical competencies and software platform knowledge are just the beginning of a successful software project. Problem domain expertise resident within the technical team is also necessary.

Many software development professionals focus on developing their skill sets from a programming/implementation platform perspective. Many do not fully recognize the need to intimately understand the background of the problem domain that a software project is designed to model.

Many developers view problem domain knowledge acquisition on a 'need to know' basis. "Tell me what to do and as a technology expert, I will build what you tell me to" is often the mindset. They are often willing to rely on non-technical liaison personnel to provide the bulk of the problem domain knowledge necessary to successfully develop the software.

Unfortunately, while there are formal analysis and design processes for obtaining the problem domain knowledge necessary to design a software project, our experience over the years has shown that most of the non-technical problem domain experts provided for a project are not well enough versed in software architecture to efficiently guide the project from conception, through implementation and testing, to successful production delivery.

Ideally, problem domain knowledge will be an integral part of the skill sets of the technical team members as well.

Non-technical problem domain resource people do not generally understand the technical sphere well enough to know which aspects of the problem solution space need to be considered at the earliest stages versus the decisions that could be made later in the process.

Similarly, technical personnel making design decisions about the software without fully understanding what it is that the software really needs to ultimately do, risk making major architectural choices that limit the software and may require major rewrites as a result of those limitations. Graceful evolution of the software system as needs change may also be impossible if the development team did not appropriately design the declarative knowledge aspects of the system.

This is particularly true within the design of the database structures used to store the declarative knowledge for the problem domain. Breaking information groupings into an appropriately rich granularity, properly normalized, based on the natural structure of declarative knowledge, is a process we term Information Naturalization. Designing the set of database tables which underlie a software system are decisions that must be made correctly from the beginning. Any programming code that accomplishes tasks with that declarative knowledge would potentially have to be rewritten if the database structure underlying the programming is later significantly changed.

Software Development and Environmental Problems

It is our view that the high rate of software development failure often reported in the literature and in the software development trade press can be attributed to two main weaknesses in software engineering practices.

The first weakness is that current analysis models (and their programming incarnations), including object-oriented approaches, are incomplete and do not adequately address the "natural" structure of the real world problem space. This problem renders current approaches less adaptable than are required in view of the compounding second weakness which recognizes the inability of software development lifecycle models to realistically handle common business / economic decision making and human thinking patterns.

These intransigent human thinking patterns generally cause software to be developed "piecemeal" (i.e. largely disorganized incremental development focusing first on the closest obstacles then moving to the next level obstacles, etc. rather than initially understanding the overall problem space before attempting to solve the most immediate sub-problems).

Example of piecemeal development, building analogy - favela

Building software systems in a piecemeal 'as needed' way without an overarching design vision often results in an unsafe, logistically difficult, and ineffective information storage system much like piecemeal housing solutions often result in unsafe, logistically difficult, and inconvenient housing solutions for the people living there, no matter how pretty the facades look. A naturalized declarative knowledge data layer provides the right architectural foundation upon which to build effective information systems.

(Credit ‐ SoftScience Web Media adaptation via Shutterstock)

Premature Economic Judgment

Among the most serious of these problems is premature economic judgment, or in other words, the need to estimate project costs and completion schedules before the project has really been thoroughly analyzed. Neither the client nor the developer can afford to invest significant time doing detailed analysis and design before the basic economic viability of the project can be determined.

Premature economic judgement analogy

Estimating the development cost of software systems before the analysis/design work is performed in order to figure out what needs to be included in the software project box is the 'premature economic judgment' paradox of software engineering. The high failure rate of software development in terms of time and budget can often be blamed on the rush to determine a cost/price for the project before enough work has been done to understand what needs to be built.

(Credit ‐ photo illustration by SoftScience Web Media)

In the client's case, the need for which the project was conceived in the first place is usually a pressing one, meaning the client organization is probably required to devote most of its available time in order to just complete the affected tasks with the old, inadequate system.

In the developer's case, a developer cannot commit significant resources to a project until a budget has been approved. Thus for many projects, the developer must commit to a budget and schedule based on an incomplete understanding of the task to be undertaken.

Limited Horizon Specification

Compounding this problem is the problem of "limited horizon" specification. Limited horizon is a problem arising from basic human nature.

Human thinking generally follows a progression from the general to the specific, from the simple to the more complex, and from the immediate to the longer term. These perspectives cause serious distortions in the early preliminary attempts to define and quantify a software project.

One manifestation of the limited horizon problem is that users, when thinking about the problems to be solved, tend to focus on describing the most immediate problem or obstacles standing in their way. Only when that immediate obstacle is removed, do the secondary obstacles become visible and demand attention. Don't allow users (or developers) to say, "Build this much first, then we'll worry about the rest."

Problem - Elephant Limited Horizon Specification

Software systems are often designed by focusing primarily on the biggest obstacles blocking the way to a business goal. The problem with this "limited focal horizon" human tendency is that once the immediate obstacle is removed and a new piece of software is deployed, the remaining hidden obstacles are revealed without the underlying design considerations having been made that would also remediate those business problems. Information Naturalization studies help to identify the hidden problems and discover potential solutions in the "adjacent possible" of the declarative knowledge asset of an organization.

(Credit ‐ animation by SoftScience Web Media)

Similarly, human thinking patterns support a realization that from a user’s perspective, explaining a process involves explaining how the process works in general. After the software is written, he/she will test it for the exceptions to the general rule.

Conversely, from a developer's perspective, the exceptions to the general rule are almost always more complex than the general rule, meaning that solutions for the exceptions must be factored into software's fundamental design from the very beginning.

Problem domain information engineering

If the software development team members all possess a combination of software engineering and problem domain understanding, the software solutions produced will inevitably be superior and less costly to those produced by a team which has a knowledge chasm in between the technical platform skills and the problem domain understanding capabilities of the team members.

The author believes that in order for a software developer to effectively do his/her job and produce software that appropriately solves a problem in the real world, that developer should be able to actually sit down and actually perform the process himself – without software – that he is trying to replicate in software form. While such a developer may not be able to perform the tasks as quickly as the worker who performs those functions on a routine basis, if the software developer does not have sufficient understanding to actually perform the job, then how can he/she be expected to produce a product that performs the job correctly in all instances?

SoftScience Group, Inc. has always considered problem domain expertise of its technical staff to be as important as their technical expertise. Finding individuals who understand this importance is a challenge for the software engineering community. We believe that information engineering as it applies to software problem domains are integral parts of the software engineering discipline.

We further believe that by using an Information Naturalization approach to understanding the problem domain and preserving the natural structure of declarative knowledge within our database implementations, our software developers quickly begin to learn the nuances of a problem domain simply through their exposure to a naturalized enterprise data layer used for software applications within that problem space.