|Table of Contents
MN GIS/LIS Consortium
2006 Conference Wrap-up
From the Chair
Scholarship Competition Results
Conference Auction Results
What is GISP?
GIS Rules of Conduct
MN GIS Speakers Bureau
North Carolinian Attends Conference
2006 Air Photos
Geospatial Services Survey
Gopher State One Call
MetroGIS DataFinder Improved
Housing/Jobs Mapping Site
Metro Land Conversion
GIS for Surveyors
2010 Census Address Program
2010 Census Options
National Land Cover Database
St. Mary's Update CEU's
GITA Announces Scholarship
Benefits of Parcel WMS
Get Ready for Real Time: Up-to-the Minute Data Exchange in Urban Information Systems
By Greg Sanders, Web Projects Manager, Chicago Metropolitan Agency for Planning, Reprinted from URISA News September/October 2006 issue. With permission.
Major sections of this article:
How it works
Why web services make sense for IT staffs
Why web services make for better government
What is XML?
Service-Oriented Architecture (SOA) and Enterprise Architecture (EA)
A Chicago web services network
Web service networks at the federal level
Web service networks at the state level
Bridging the gap between federal, state and local agencies
Barriers to web service implementation
A note on GML and PMML
The Chicago metropolitan area is a big place, especially if you measure it parcel by parcel. The six-county Chicago region is home to more than 8.3 million people on some 2.7 million parcels; Cook County alone has 1.6 million parcels. Every parcel has dozens of attributes of interest to planning and development agencies like ours (the Chicago Metropolitan Agency for Planning, or CMAP). Factor in the frequency of change to those parcel attributes—changes in land use, ownership, residency, structural alterations, employment, zoning and transportation access, to name a few—and you have a challenging data management project. This falls under the Stephen Wright axiom: “small world, but I wouldn’t want to paint it.” If you insist on parcel-level data, you commit to millions of records and daily updates.
I always wanted to have access to the most up-to-date, accurate, detailed data for every parcel in the region whenever I needed it, without making phone calls or sifting through paper archives. We got that, in a sense, when city and county agencies began posting parcel data on the web. Go to the website, type in a street address and up pops a list of data attributes, or a list of building permits or property sales.
But the thrill of those early web-accessible databases wore off when I needed to integrate parcel-level data into my own web systems. I displayed links so users could jump to county and city web sites, but the users clamored for a one-stop information site that would pull together data from all relevant sources in one place. Pulling it together was easy if you had good partners who frequently sent you their data, but some of the information was outdated as soon as you posted it. And for some reason our partners were unwilling to send data updates every 45 minutes.
Fortunately, public agencies can now provide up-to-the minute data to their partners without even working hard. City, county, regional, state and federal agencies can make data available via web services that can be called by other servers at any time. This means that all partners can incorporate the most current data available into their own data systems. The data can be fetched as needed from the most authoritative source, then displayed on a web form, pulled into a predictive model or used to calculate aggregate statistics.
Web services deliver the goods rapidly, typically employing XML (extensible markup language) for information requests and responses. Web services are not difficult to create, perhaps a bit more difficult to consume. They can be secured in a number of ways, including tiered access levels. The primary barriers to widespread implementation of web services across government divisions are organizational—the need for agreement on data definitions, concerns about security and the reluctance of some agencies to share information. The same factors that have hampered data sharing in the past can restrict the deployment of web services. But today, the technology for real-time data exchange is at a point where the benefits of data sharing far outweigh the costs.
How it works
Web services are fairly simple in theory and in practice, although they get more complex as the need to protect confidentiality increases. The core function of web services is to provide something—often data—to another server that has submitted a request. In urban information systems the typical scenario is an application that requires data about some parcel, census block or other geographic location. The requesting server could call a local database within its own internal network, assuming a copy of the data has been stored locally. That’s how it was done in the past. But instead, the data can now be fetched from a remote server via the internet. This is not quite as fast as a local data fetch across an internal network, but it can be surprisingly fast with sufficient bandwidth, server capacity and well-designed services. A large data fetch across a good web connection can be executed in well under a second. The remote data is embedded directly into the web page, a process called “transclusion”--inclusion of part of one document into another document.
CMAP’s “Full Circle” parcel information system is a good example of how this works in the real world. Users of our data system conduct property surveys for a variety of zoning, economic development and other purposes. They walk up to the property with a smart phone, open a browser, select a property address from the list and click “Go”. This sends a data request to CMAP’s web server, which retrieves some internally-stored data from CMAP databases. But our web server is smart enough to look elsewhere for the most detailed, up-to-date data about the property’s assessed value, ownership, building permits and so on. Our web server issues consecutive data requests to various county and city services, renders the resulting data into HTML and sends the whole batch down to the user’s browser. All within less than a second. Now our user is looking at the most current, authoritative data available for the specified property, without knowing or caring where the data was retrieved from, or how it was called.
Why web services make sense for IT staffs
Web services can increase efficiency, interoperability, transparency and data quality. I’ve witnessed many over-hyped technologies over the years that turned out to be more headache than help, but web service architecture is a no-brainer. It’s a fairly simple, reliable path to more accurate data, less duplication of effort and fewer hours spent importing and exporting data. And it can deliver what some people still believe is a pipe dream: government agencies that share information in real time. Government systems that talk to each other. Here are some reasons why government IT staffs might want to adopt web services.
- Reduce data discrepancies: Web services can eliminate the need for multiple organizations to maintain separate copies of the same data. “Any time you have redundant copies of the data, you introduce the possibility of inaccuracy and discrepancies,” says Eric Sweden of the National Association of State Chief Information Officers (NASCIO). As soon as we import any dynamic dataset from a partner, it is obsolete. Without web services, we have to choose between posting outdated data in my applications or leaving the data out altogether.
- Reduce duplication of effort: Tally up all the hours you have ever spent importing other people’s data, or exporting your data to others. Is this a productive use of your time? If you could write a few lines of code to export or import the data in small, precise batches just at the moment it is needed, wouldn’t that make your life easier? After you have imported the data, you spend hours maintaining it. Wouldn’t you rather let the database owner maintain the data, and send you pieces of it as needed? (Trust is a key factor here—data consumers need to trust not only that the data is accurate, but that the provider’s web service will always be up and running.) If nothing else, think about the FOIA requests you handle on a case-by-case basis. You could be referring all FOIA requests to a web service portal.
- Rationalize data access: Traditional data sharing arrangements often depend on personal relationships between data providers and data consumers. If you know someone who works at the county clerk’s office, lucky you! But when your friend leaves to pursue a lifelong dream of becoming a master chef, you have to get acquainted with a new provider. Each time such changes occur, the data flow stops until you take the time and energy to get it going again. With web services, the servers keep on humming through reorganizations and early retirements.
- Speed up the flow of data: Traditional import processes are too slow to be useful in any rapidly-changing situation. Emergencies are a case in point. At a time of crisis, are IT managers likely to stop their work to send out updated disks or post updated files on FTP servers? With web services, all partners can operate from the exact same data at the same moment. Governments have been criticized for lacking systems that “talk to each other.” Web services do exactly that—they communicate information, at the system level, without any costly human intervention. Even non-emergency situations sometimes call for more rapid data dissemination than traditional methods allow. In Chicago, neighborhood gentrification (a steep rise in property values and rapid conversion of apartments to condominiums) sometimes happens in a matter of months, not years. Quarterly downloads of building permits, property transaction and land use data from city and county sources might be too infrequent to allow a timely response to such changes.
- Targeted cost recovery and improved customer service: Public agencies often charge a fee for providing data, as a way of recovering costs associated with their data systems. Web services can allow agencies to provide targeted data feeds (say, tailored to a user’s own neighborhood) for a much lower cost than they might need to charge for exporting an entire data set. This is a win-win situation, since the customer can now get regular updates from the web service, and the provider can now market to smaller customers who can afford the limited service. Each user’s login ID can determine what data, and how much data, the user is permitted to extract.
Why web services make for better government
- Efficient government: Information is an under-utilized asset in government, according to Eric Sweden of the National Association of State CIOs. “Information that is not shared can’t be used by anyone other than the holder of that information. Information is an enterprise asset. If it’s not shared, are we getting the value out of it? We have a cost to maintain it, archive it, and protect it, but if we’re not sharing it, the enterprise is not benefiting from its full value. If information is not shared across the enterprise, then we find ourselves in the common scenario of maintaining multiple instances of the same information. Without proper administrative controls to keep those instances in sync, we face the potential risk of inaccurate or inconsistent information which can significantly impact the effectiveness and efficiency of business decision making. It’s more expensive to maintain because those who need it are maintaining it redundantly.”
- Better service delivery: Web services can also enhance delivery of government services, Sweden says. “We’ve got a generation of people growing up here who ask ‘why not?’ They are saying to individual agencies, ‘Why do I have to tell you the same information again? I just gave the same information to this other agency. Aren’t you connected?’ They’re going to be challenging government, saying ‘I don’t want to go through this again.’ They’re looking for convenience, they’re looking for access to government via the web because they don’t want to make a physical trip to an office…. They’re computer-literate, they’re comfortable with computers, they do almost everything on a computer and they’re saying ‘Why can’t I deal with government on my computer?’”
- Leveraging citizen participation: Phil Windley, former Chief Information Officer for the state of Utah, sees value in going beyond government-to-government data exchange, by making some web services accessible to the public. “Putting data out—not just building e-government applications, but putting data out—helps you leverage an entire set of developers, product managers and other people who are interested in building applications with that data, in a very similar way to the way open source works. Call it ‘open data’… The same reason why open source is successful in leveraging other people is a good reason to suspect that we can leverage other people if we make data available.”
- Breaking down silos: Silo-smashing is another fundamental benefit that web services can bring to government. Brand Niemann of the U.S. Environmental Protection Agency (EPA) says that data silos are “a systematic problem in the government. Congress funds IT systems by program within each agency…. We have 32 major systems that the states collect data for. We have legislation mandating these data silos. You’re not going to change that, but you can use web services to pull data together.” Both Niemann and Windley see data silos as inevitable in a decentralized government structure like ours in the United States. Both see the decentralization as generally positive, but the resulting data silos as problematic. Windley notes that “Web services allow us to keep the decentralized nature of government, and maintain the benefits that we see from that in terms of governance, and at the same time break down the barriers to providing good service to citizens. What you’d like is an application that takes data from at least three different agencies to build a single application…. With web services we can build that application because the data is available. Everybody gets to keep their own data, everybody keeps their own business processes, and yet the application can still be built and managed by some group. We keep the decentralization and the benefits that it has, and we can mitigate some of the disadvantages, particularly in the area of IT.”
- Better decision-making: Better decision-making is also at stake in the campaign for web-based information sharing. “If decision-makers have the information in front of them, they can make better decisions about where to direct resources,” Sweden says. “If that information is held back, they’re going to make decisions based on just what they know. If they don’t know everything, then clearly their decisions are going to be less effective. That’s where we get down to the value of information. If it is not shared, then it’s not impacting decision-making in all the circumstances where it could or should be.” Officials with an “executive dashboard” of data provided by various services, Sweden adds, can base their actions on the best information available.
- Government transparency: Windley sees value in the transparency that public web services can provide: “It does increase government transparency and accountability. Some people don’t like that but I think it’s probably what is required for good government.” This principle is echoed by District of Columbia city administrator Robert Bobb on DCStat web site: “The guiding principle for streaming city agency information to the web is to enable residents to better understand our government’s activities, thereby offering more opportunities to participate in improving the quality of life and promoting economic development in the District.”
- Emergency response: Lack of interoperability has been frequently cited as a contributing factor that hampered government responses to the 9/11 and Katrina disasters. Web services can deliver interoperability. No other current technology has a realistic chance of linking many disparate units of government in real time.
What is XML?
Extensible markup language is “a markup language for documents containing structured information.” (www.xml.com) XML has many uses, but since this article is about data we can focus on the fact that XML is a way of structuring data within a text document that travels easily over the internet via HTTP. Virtually any computer can request data from an XML web service and read in the data contained in that document. The data may then be pulled into an application, displayed on the screen or stored for future use on a network hard drive.
They call XML extensible because it is not limited, the way HTML is, to a small set of standard tags like <TABLE>. XML tags can be devised to fit the data they represent. For example, address data can be sectioned into logical components (address number, street, and so on). Moreover, XML supports hierarchical data—data in which a single attribute may have one or more “child” attributes. For example, a property may (or may not) have one or more building permits, code violations or ownership transactions. These can be embedded in the XML, with the child data clearly shown at a more detailed logical level than the property itself.
Below is an example of XML text that might be returned from a hypothetical web service called GetBuildingPermits. (Note to FGDC sticklers and all other sticklers: this is a simplified example to convey a sense of what XML looks like.)
<?xml version="1.0" encoding="ISO-8859-1"?>
<worktype>ALTERATION - SAME USE</worktype>
<buildingtype>Single Family Dwelling</buildingtype>
Note that this same data could have been arranged with <buildingpermits> as the root node, and <address> as a child element of each building permit record. I chose this layout because to me it makes more sense to see <address> as an element of a property, not a permit. XML’s greatest strength—its flexibility—is also the reason why it demands a great deal of agreement among users of a particular XML service.
Service-Oriented Architecture (SOA) and Enterprise Architecture (EA)
Web services are not just for data exchange with external partners; they also work well for cross-departmental sharing. A city government wishing to provide a one-page parcel profile for use by city employees might create a web form that pulls together property ownership, permits, physical characteristics, business licenses, court records, crime data, any public financing or subsidies, historic value of the structure, building condition and many other attributes. If a central IT department has control over all this data, great! You may not need web services. But typically these bits of information are housed and owned by various data stewards across several departments. A series of lightweight web services could be deployed as interfaces between departments.
But while web services work very well as add-ons to legacy systems, they are much more powerful when integrated into the core of an enterprise architecture. SOA, or service-oriented architecture, enjoys a great deal of support from IT professionals because of its loose coupling, code reuse, flexibility and emphasis on business process documentation. In these respects SOA is a specific form of Enterprise Architecture (EA), which is now strongly endorsed by many IT standards bodies—most notably within the federal government. The Federal Enterprise Architecture initiative led by the Office of Management and Budget (OMB) is changing the way federal agencies operate. OMB defines EA as “The explicit description and documentation of the current and desired relationships among business and management processes and information technology.”
Continued on page two.