VTL 1.0 – Validation and Transformation Language

VTL is a standard language for defining validation and transformation rules (set of operators, their syntax and semantics) for any kind of statistical data. VTL builds on the Transformation section of the SDMX information model, by taking the common parts of GSIM, SDMX and DDI as for the representation of concepts and data.

The assumption is that such a logical formalization of validation and transformation rules will provide a “technology-neutral” expression at business level of the processing taking place, against which various implementations and specific programming languages can be mapped.

The specifications for exchanging VTL validation rules in SDMX messages, for storing rules and for requesting validation rules from web services will be provided in a specific update to the SDMX Technical Standards on which the SDMX Technical Working Group is working on.

The VTL 1.0 package includes:

  1. Part 1, highlighting the main characteristics of VTL, its core assumptions and the information model the language is based on;
  2. Part 2, containing the full library of operators ordered by category, including examples; this first version can support validation and basic compilation needs. Future versions, whose design is already in progress, will include more features related to transformation of data;
  3. EBNF notation (Extended Backus-Naur Form) which is the technical notation to be used as a test bed for all the examples throughout the document.

Download the complete VTL 1.0 package (part 1, part 2, EBNF)

Please send your comments to the SDMX Technical Working Group (TWG) at twg@sdmx.org. Each email message should contain only a limited set of numbered – and possibly homogeneous – comments and should contain the elements indicated in the attached comment log.

Posted on 3 March 2015

Public Review Announcement for the new version of the SDMX RESTful API Specification

The SDMX Technical Working Group (TWG) has prepared a version v1.1.0 of the SDMX RESTful web services API specification for public review. The review period extends until 30 June 2015.  Feedback is welcome.

Version v1.1.0 contains two new features, two improvements to existing features and one bug fix. Furthermore, the documentation has been improved with the addition of a tutorial, a cheat sheet and new sections about best practices for data providers and data consumers.

All changes can be viewed on Github at the following location: https://github.com/sdmx-twg/sdmx-rest. The use of Github significantly eases the work of comparing the changes with the previous releases of the specification.

– Complete list of releases:
https://github.com/sdmx-twg/sdmx-rest/releases

– Complete list of changes since the previous release v1.0.2:
https://github.com/sdmx-twg/sdmx-rest/compare/v1.0.2…master

– Overview of new features, improvements and bug fixes in the new version:
https://github.com/sdmx-twg/sdmx-rest/issues?q=is%3Aissue+milestone%3Av1.1.0+is%3Aclosed

Please send your comments to the SDMX Technical Working Group (SDMX-TWG) at twg@sdmx.org, using the attached comment log. Alternatively, changes can be proposed by forking the Github repository and sending pull requests.

Posted on 26 February 2015

SDMX-NA and BOP: new DSD packages published

The Ownership Group for SDMX implementation in Balance of Payments (BOP), Foreign Direct Investment (FDI) and National Accounts (NA) is pleased to inform you that updated DSDs have been made available for NA and BOP. FDI was not updated at this time.

The new National Accounts release adds support for SNA93, Euro Area 19 extension, additional coding for ESA-GFS tables and other minor additions after comments received from data providers and international organisations. Along with the new DSDs, updated Excel templates have been published, which support the new DSD versions. Apart from some ESA-GFS tables, all templates are fully backwards compatible with earlier versions of the DSDs.

For more details, release notes and downloads please consult the SDMX-NA maintenance page or SDMX-BOP/FDI maintenance page.

Public Review Announcement for the new VTL language, allowing the definition and exchange of validation and transformation rules

A dedicated task-force of the SDMX Technical Working Group (TWG) submits the new specification of a Validation and Transformation Language (VTL) for public review. The review period extends until 8th November 2014 in order to allow time for consolidation of a final version at the end of 2014.  Feedback is welcome.

At the end of 2012, the SDMX Secretariat requested the TWG to start a new work package on “Expressions and Calculations” aimed at allowing the definition and the exchange of validation and transformation rules, based on the generic framework for defining mathematical expressions already existing in the SDMX information model. To make such framework fully operational, a standard language for defining validation and transformation rules (set of operators, their syntax and semantics) should be adopted, while appropriate IT formats for exchanging such rules – and web services to store and retrieve them – should be designed.

The intention is to provide a language which is usable by statisticians to express logical validation rules and transformations on data, whether described as a dimensional table or as unit-record data. Although a work-product of SDMX, the VTL language is based on the GSIM data model and should be usable with other standards as well. The present draft for public review also provides (in its part 1) a formal description of data validation at business level against which information models can be mapped.

VTL – part 1: Validation and Transformation Language (General Description)

VTL – part 2: Validation and Transformation Language (Library of Operators)

Please send your comments to the SDMX Technical Working Group (SDMX-TWG) at twg@sdmx.org. Each email message should contain only a limited set of numbered – and possibly homogeneous – comments and should contain the elements indicated in the attached comment log.

Posted on 21 September 2014