The Daytona^TM data management system is used by AT&T to solve a wide spectrum of data management problems. For example, Daytona is managing over 312 terabytes of data in a 7×24 production data warehouse whose largest table contains over 743 billion rows as of Sept 2005. Indeed, for this database, Daytona is managing over 1.924 trillion rows; it could easily manage more but we ran out of data.

Daytona’s architecture is based on translating its high-level query language Cymbal^TM (which includes SQL as a subset) completely into C and then compiling that C into object code. The system resulting from this architecture is fast, powerful, easy to use and administer, reliable and open to UNIX tools. In particular, two forms of data compression plus robust horizontal partitioning and effective SPMD parallelization enable Daytona to handle terabytes with ease. Fast, large-scale in-memory operations are supported by in-memory tables and scalar and tuple-valued multi-dimensional associative arrays.

Daytona offers all the essentials of data management including a high-level query language, data dictionary, B-tree indexing, locking, transactions, logging, and recovery. Users are pleased with Daytona’s speed, its powerful query language, its ability to easily manage large amounts of data in minimal space, its simplicity, its ease of administration, and its openness to other tools. In particular, Daytona supports SQL, Perl DBI, and JDBC.

1. Daytona In Use
2. Compared To Awk & Perl For Flat File Processing
3. Instructional Queries
4. Recently Added Features

[…]

http://www.research.att.com/projects/daytona/

Astonishing. None of this is ‘secret’ as such; they say their largest single table has 743 Billion rows. What they fail to disclose is what is in those tables. They leave it to you to ask (or not) just what the hell are you keeping in those tables?

And who prompted the NSA that this data was being kept and could be used as a ‘security’ asset? Did AT&T hint that they would be willing to sell access to this treasure trove to the NSA?

Have they made a mysqlhotcopy of this database for other agencies to sift through?
What other telecoms companies have this type of facility?

This entry was posted on Monday, March 6th, 2006 at 12:31 pm and is filed under Geekn.