Northrop Grumman Corporation (TRW): VLDB Project

From Devwiki

Jump to: navigation, search

Northrop Grumman Corporation Mission Systems, formerly TRW, built a high throughput intelligence processing application for a government agency which must not be named. The customer is based in Langley, VA, but there are deployments in at least five locations to date. The system is deployed in the National Counter Terrorism Center (NCTC) in Rosslyn, VA (see External Links below).

Northrop Grumman Corporation is a $30 billion global defense and technology company whose 122,000 employees provide innovative systems, products, and solutions in information and services, electronics, aerospace and shipbuilding to government and commercial customers worldwide.

Contents

[hide]

[edit] Objectivity Case History

[edit] Important

This information is an archive, so any use of the present sense in the text should be taken in the historical context, generally determinable from the Status section below.

[edit] Customer Information

[edit] Status

  • First Contact: 1990
  • Lead came from: TRW contacted us as a result of seeing a Press Release about CERN's use of Objectivity/DB.
  • Evaluation Start Date: 1997
  • Evaluation Finish Date: 1997
  • First Purchase Date: 1997
  • Deployment Date: 2000
  • Current Status: Development is ongoing. The project was officially in a trial mode until around 2004, but it was so successful that it was allowed to process operational data and feed operational projects.
  • Can we talk about this customer and the product/project? Only to customers with adequate clearances and the permission of NGC and the customer.
  • Referenceable?: Carefully, without reference to the application, customer or location.

[edit] Environment

  • Hardware: Multiple Solaris multi-processor servers (initially E1000 series).
  • Operating System: Solaris 9.
  • Precision: 32-bit.
  • Development language: C++ in the servers and Java on the desktop.
  • Compiler: Sun C++ and Java.
  • Third Party vendor tools: STL initially, but it is no longer used.
  • Open Source tools: None.

[edit] The Project/Product

[edit] Project Background

TRW approached us after seeing a Press Release about Objectivity/DB and CERN. They had already ruled out Oracle on performance and scalability grounds. They wrote a high ingest benchmark that had to ingest over 1 Billion objects in a six hour period and ran it on Sun hardware using Versant and Objectivity/DB. Versant failed to load all of the data. Objectivity/DB achieved the benchmark several times over a single weekend.

The application was very highly guarded for many years, even within the customer organization. However, its use was eventually cautiously spread to three or four other locations and Northrop Grumman and the customer decided to give it much higher, though still strictly classified, exposure in 2006. The contract for the application was relet in 2002 and 2006, with Northrop Grumman retaining it.

[edit] Project/Product Description

The actual purpose of the application must remain classified. However, it can be characterized as a typical intelligence ingest, correlation, analysis and dissemination system. A wide variety of different data types are involved, including data obtained periodically from scientific databases. It is believed to be an extremely large database, ingesting many Terabytes of data per day. It quickly reached many Petabytes. Many thousands of threads ingest, correlate and query the information. Small subsets (tens to hundreds of Gigabytes) are copied to other sites every night after new batches of data have been ingested. There were initially about 50 users of the system, but this number is believed to have grown substantially since 9/11/2001 and is now estimated to be around 1,000 concurrent users.

The application uses a cluster of Sun servers attached to an EMC mass storage system. It does not AMS or data replication for performance reasons. The TRW developers are probably the most experienced users of Objectivity/DB in a High Performance Computing, VLDB environment.

[edit] Buying Criteria

[edit] Business Priorities

Flexible pricing was important as TRW could not disclose exactly how many sites or end users might be involved. The final licensing conditions settled on a server/processor based license. TRW buys developer and some end-user licenses. The government customer buys the deployment licenses via a specialized contract vehicle.

[edit] Technical Priorities

  • Speed and Performance of the Competitive Benchmark, which tests:
    • Load performance
    • Sizing behavior (how many bytes does it occupy)
    • Analysis performance (queries and retrieval of data)
    • Performance of STL (which was eventually phased out after TRW built their own collection classes).
    • Stability and reliability.

TRW said in a joint Press Release in 1992 that - "Objectivity was selected based on its performance relative to three different criteria:

  • The first was the ability to ingest data in one-writer mode, to avoid the performance issues of lock contention, at a rate of 2000 items per second.
  • The second was that access had to be interactive for 50 simultaneous users. As a practical matter, this meant that the system had to be able to provide interactive analytical solutions to all concurrent users.
  • The third requirement was that the persistent model be efficient enough to store billions of items within an economical storage system.

The system also needed to be capable of scaling to much higher volumes in the future without losing its interactive character."

[edit] Competitors/Alternatives

TRW evaluated both RDBMSs and ODBMSs:

  • Oracle was ruled out because of its inability to support the processing load with so many relationships.
  • ObjectStore was ruled out because of its known lack of scalability. TRW contacted CERN and SLAC for verification of this decision.
  • The benchmark was run with Versant and Objectivity/DB.

Objectivity/DB won convincingly against Versant, aided by:

  • Its more scalable architecture.
  • Page server versus object server performance.
  • Faster navigation of relationships between objects.
  • Better object clustering.
  • Smaller database size.
  • Better utilization of the hardware.

[edit] Why They Chose Objectivity

  • "Brilliant" technical evaluators and visionaries.
  • We won the benchmark.
  • Great references from CERN and SLAC.
  • Close attention to their requirements and their confidence in our ability to support them.
  • Perceived long term corporate viability because of our focus on long term contracts.

[edit] Partners

  • Sun lent technical assistance to help tune the operating system and storage configuration, but they were neutral in the Sales process.

[edit] Collateral

  1. Press Releases: 9/24/2002
  2. Flyers: None.
  3. White Papers: Building a high throughput data repository with high query performance
  4. Case Study: You're reading it.
  5. Other:
    1. A talk at WorldView 2005
    2. Dave Capka's presentation in the Government webinar series Conquering large and complex data sets in security and intelligence applications

[edit] Contact Information

[edit] Related Pages

[edit] External Links

  • NCTC - How A Little-Known Spy Agency Helped Track Down Osama Bin Laden

[edit] Categories

Retrieved from "http://devwiki.objy.com/objywiki/index.php/Northrop_Grumman_Corporation_(TRW):_VLDB_Project"