Day: January 15, 2009

Aggregated Live Feeds in ArcGIS

/ Matt Artz

I recently spoke with ESRI’s Derrick Burke and Paul Dodd about a methodology they’ve developed to aggregate live data feeds in ArcGIS. 

dburke_pdodd

Derrick Burke (l) and Paul Dodd (r)

Derrick Burke is the Technology Team Lead in ESRI’s Technical Marketing department.  He holds a BA in Geography from SUNY Geneseo, a Masters in Geography, Urban Planning, and Sustainable Development from UNC Charlotte, and an MBA in Finance from the University of Redlands.  Derrick has worked at ESRI for more than eight years, first within professional services as a developer and then in technical marketing focusing on creating prototypes on new technology and presentations.

Paul Dodd is the GIS Systems Administration Team Lead in the Technical Marketing Department. A Computer Science Major at California State University, Paul has more than 25 years of experience in the computer industry working with Mainframe, Mini, and Micro computer systems and software. For more than 11 years at ESRI, Paul has working with ArcSDE in conjunction with various Oracle and Microsoft database products.

You’ve developed a technique you’re calling “Aggregated Live Feeds”…can you tell me more about it?

We have developed a methodology on how to aggregate Internet accessible data (eg. USGS/NOAA xml, GeoRSS, and so on) as services through ArcGIS Server. We load the aggregated data into layers within ArcSDE, expose them as Spatial Views, and serve through ArcGIS server in near real time. Any client that can consume ArcGIS Server services can leverage these services (ArcGIS Desktop, ArcGIS Server, ArcGIS Explorer, etc). This technique was developed in response to the desire to show near real-time data (such as weather data) and analysis within all of our ESRI products.

feedload A running load script.

What are the benefits of using this methodology?

Depending on the feed requirements, data is processed and loaded on the server side every few minutes to every few hours, taking the load off clients. Continuously needing to poll for fresh data can carry a heavy penalty, especially in browser-based applications. Processing feed data on the server side allows the client to poll for data only when needed, via standard ArcGIS Server protocols. Managing these feeds centrally can ease the demand on network resources by using a few systems to download feed content from the Internet, rather than potentially a large numbers of users. Clients can then access these local services as operational layers or fuse them with other base maps and operational content.

How was this built?

Our methodology uses simple batch scripting with a handful of public domain command line utilities to download and pre-process the feed data. The scripts then use ArcSDE command line functions to push this data to the database. The scripts also incorporate logic to track the process, making sure they run as expected. If a load should fail, an alert email is sent to an administrator. Scripts run every 5 minutes, 30 minutes, once an hour, or once a day depending on the appropriate need. The current methodology can handle shapefiles, raw ASCII (like CSV and some custom formats), and XML (like RSS, GeoRSS, and CAP). There are even utilities that allow the scripts to handle de-compressing files. Once the data is pushed into ArcSDE, ArcGIS Server services are authored and served, monitoring/notification on the availability of these services are provide through the service monitor available on arcscripts.esri.com.

noaa1Weather data feed.

Can you give the readers of this blog some examples of how could this methodology be used for scientific applications?

Sure, these aggregated live feeds have been used in applications ranging from homeland security to environmental analysis. For example, in one of our latest demonstrations a light weight browser application will call an analysis ArcGIS Server geoprocessing service to perform plume modeling (eg. a contaminant leak) based upon an aggregated ArcGIS Server service containing the latest wind velocity and direction. The analysis produces a plume which can then be chained to other ArcGIS Server analysis such as identifying the demographics of the area.

Do you plan on sharing this with the ESRI user community?

We plan to post this methodology on ESRI’s ArcScripts, blogs, and the new resource center. We’ve had many clients interested in doing this and in fact some clients have created their own tools that perform similar functions.

Any possibility this will lead to new functionality in ArcGIS?

This methodology was designed to work with previous, current, and future releases of our software because it’s more of a methodology than customizing our core tools. Because this idea is becoming more popular, the development team is thinking about supplying aggregator tools as part of the core software, but that project is still in design.

Transforming our Economy with Science and Technology

/ Matt Artz

Just released, a new summary of the current draft of the American Recovery and Reinvestment Bill, popularly known as the “Stimulus Package.” I’m pasting below the entire section titled “Transforming our Economy with Science and Technology.” 

We need to put scientists to work looking for the next great discovery, creating jobs in cutting-edge technologies and making smart investments that will help businesses in every community succeed in a global economy.

Broadband to Give Every Community Access to the Global Economy
• Wireless and Broadband Grants: $6 billion for broadband and wireless services in underserved areas to strengthen the economy and provide business and job opportunities in every section of America with benefits to e-commerce, education, and healthcare. For every dollar invested in broadband the economy sees a ten-fold return on that investment.

Scientific Research
• National Science Foundation: $3 billion, including $2 billion for expanding employment opportunities in fundamental science and engineering to meet environmental challenges and to improve global economic competitiveness, $400 million to build major research facilities that perform cutting edge science, $300 million for major research equipment shared by institutions of higher education and other scientists, $200 million to repair and modernize science and engineering research facilities at the nation’s institutions of higher education and other science labs, and $100 million is also included to improve instruction in science, math and engineering.
• National Institutes of Health Biomedical Research: $2 billion, including $1.5 billion for expanding good jobs in biomedical research to study diseases such as Alzheimer’s, Parkinson’s, cancer, and heart disease – NIH is currently able to fund less than 20% of approved applications – and $500 million to implement the repair and improvement strategic plan developed by the NIH for its campuses.
• University Research Facilities: $1.5 billion for NIH to renovate university research facilities and help them compete for biomedical research grants. The National Science Foundation estimates a maintenance backlog of $3.9 billion in biological science research space. Funds are awarded competitively.
• Centers for Disease Control and Prevention: $462 million to enable CDC to complete its Buildings and Facilities Master Plan, as well as renovations and construction needs of the National Institute for Occupational Safety and Health.
• Department of Energy: $1.9 billion for basic research into the physical sciences including high-energy physics, nuclear physics, and fusion energy sciences and improvements to DOE laboratories and scientific facilities. $400 million is for the Advanced Research Project Agency – Energy to support high-risk, high-payoff research into energy sources and energy efficiency.
• NASA: $600 million, including $400 million to put more scientists to work doing climate change research, including Earth science research recommended by the National Academies, satellite sensors that measure solar radiation critical to understanding climate change, and a thermal infrared sensor to the Landsat Continuing Mapper necessary for water management, particularly in the western states; $150 million for research, development, and demonstration to improve aviation safety and Next Generation air traffic control (NextGen); and $50 million to repair NASA centers damaged by hurricanes and floods last year.
• Biomedical Advanced Research and Development, Pandemic Flu, and Cyber Security: $900 million to prepare for a pandemic influenza, support advanced development of medical countermeasures for chemical, biological, radiological, and nuclear threats, and for cyber security protections at HHS.
• National Oceanic and Atmospheric Administration Satellites and Sensors: $600 million for satellite development and acquisitions, including climate sensors and climate modeling.
• National Institute of Standards and Technology: $300 million for competitive construction grants for research science buildings at colleges, universities, and other research organizations and $100 million to coordinate research efforts of laboratories and national research facilities by setting interoperability standards for manufacturing.
• Agricultural Research Service: $209 million for agricultural research facilities across the country. ARS has a list of deferred maintenance work at facilities of roughly $315 million.
• U.S. Geological Survey: $200 million to repair and modernize U.S.G.S. science facilities and equipment, including improvements to laboratories, earthquake monitoring systems, and computing capacity.