Monday, 18 January 2010

Telecoms, Compliance & DQ Management

This blog post was inspired by a post that I saw today from Ken O'Connor: Achieving Regulatory Compliance - the devil is in the data.

I had actually planned to blog about some of my experience with Compliance in the Telecommunications Industry, and the role that Data Quality Management played in ensuring compliance, so thanks to Ken I started thinking about it a lot on the train home and have come up with my first post on the subject.

The European Union's Data Retention Directive

Or more formally known as "Directive 2006/24/EC of the European Parliament and of the Council of 15 March 2006 on the retention of data generated or processed in connection with the provision of publicly available electronic communications services or of public communications networks and amending Directive 2002/58/EC"

What does it mean to you?

Well, if you live in the European Union it means that telecommunication companies are, by law, required to retain certain data relating to customer usage for a set period of time. From my understanding, this varies depending on which country you are living in - Denmark has adopted further data retention policies than required, and Polish law, which came into effect in January 2010 states that Polish telecommunications companies must store data relating to customer activity for 2 years.

This Directive has come under some criticism from Privacy groups and in security circles, however, I am not a lawyer so I will talk about my experiences with the directive and how DQ Management techniques were implemented and adhered to in order to ensure full compliance.

Home Office code of practice

Since approval, and coming into force on 1st October 2007 the British Home Office code of practice recommends the following (simplified) data retention guidelines:

  • Subscriber Information should be retained for 12 months (Name, DOB, Address, Telephone Number, IMEI etc.)
  • Telephony Data should be retained for 12 months (Call from, Call To, Date, Duration, Location data etc.)
  • SMS/MMS Data should be retained for 6 months (From, To, Date, Location Data etc.)
  • Web Activity Data should be retained for 4 days (IP Address, URLs visited etc.)

How Data Quality Management was utilised

In order to be compliant, data must be stored accurately and timely from the initial point of capture in source systems, to subsequent transfer and manipulation between systems, through to any reporting that is undertaken upon the data.

A risk - a potential event or uncontrolled intervention in the data flow - which may affect the completeness, accuracy or timeliness of the data must be mitigated by ensuring correct controls are in place.

  1. Data Quality Profiling is undertaken to ensure completeness and consistency of data values and formats, as well as integrity of relationships between linked fields/records. Null values are checked, dates are subject to input masks and any incomplete/irregular data is sent to 'Suspense' tables where manual review procedures should be in place.

  2. Reconciliations between systems are undertaken (for instance, between Mediation & Interconnect). Within a telecoms organisation this may be the job of specialist Revenue Assurance teams, but support should be given by data quality specialists to ensure consistency, and communication.

    (In Telecoms 'Mediation' is the platform used where Call Display Records (CDRs) are collected from the Telephone Switch. Mediation then distributes this information the relevant downstream systems, such as Billing or Interconnect, which holds data relating to the interconnection and exchange of traffic between different telecommunications networks)

  3. Information Chain Mapping should be documented to ensure that information flow between systems, from source to target, is fully understand, including any modifications, changes in datatype formats, and if relevant, any further rules applied to data.

  4. Security & Access control policies are defined and adhered to (Data Management teams in Liaison with Risk/Security teams). Due to the nature of the systems involved, access should be limited to the service team responsible for operations of the systems, and only to development teams in emergency. this will ensure that risk of manual data loss/manipulation is minimised.

  5. If Data files are moved from system to system by FTP, or disk, File Checksum comparisons should be undertaken to ensure file integrity and completeness.

  6. Finally, in order to allow us to measure effectiveness, we should monitor performance of quality checks. If records were sent to a 'suspense' table - what was the reason? How were they resolved? These type of issues should be logged, resolved, and reported in a similar way to how you would manage other data quality issues within your organisation.

In Conclusion

Helping an organisation to be compliant is an area in which Data Quality professionals and teams can add huge value. By applying Data Quality Management Principles to Data Retention Compliance, you can raise the profile of both your team and of the need to think about Data Quality, and perhaps use this as a springboard to gain sponsorship for further initiatives within your organisation.


Dylan Jones said...

Excellent post Phil, I was sitting in a meeting at a large telco in the UK when the data retention "problem" arose, everyone was panicking about the technicalities but I remember thinking this creates a great opportunity to force the organisation to start taking data quality far more seriously.

As you say, there are so many techniques that an organisation must learn to cope with compliance and these can all be shared and re-invested back into the business across other areas.

Daragh O Brien said...


Excellent post. Having worked on both CRM and compliance side in a telco, I found that the Information Quality lense was often the best one to put tricky issues into focus.

Phil Wright said...

Dylan, Daragh,

Thank you both for your comments.

CRM - another great area in telco where some DQ quick wins and relationships can be built.

Post a Comment