* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Slide 1
Data Protection Act, 2012 wikipedia , lookup
Data center wikipedia , lookup
Clusterpoint wikipedia , lookup
Data analysis wikipedia , lookup
Forecasting wikipedia , lookup
Semantic Web wikipedia , lookup
Web analytics wikipedia , lookup
3D optical data storage wikipedia , lookup
Data vault modeling wikipedia , lookup
Database model wikipedia , lookup
Exchange of trade data using SDMX COMTRADE joint UNSD-OECD project 9 January 2007 Comtrade - overview Provide a coherent source of merchandise trade statistics data Eliminate duplication in provision and processing of data Develop common data processing standards and tools Share development costs Nearly identical integrated systems for data collection and processing at both agencies OECD processes member country data for OECD UNSD processes data for all other countries Content of complete databases in each Organisation is synchronised via the Data Exchange Suite (DES) DES = web service & synchronisation program (ComtradeTools) DECISIONS !!! Use SDMX as data exchange standard for UNSD OECD data transfers !!! Use SDMX as data dissemination format Comtrade - screens Comtrade - content • • • • • • • • • UN Comtrade is a main disseminator of annual commodity trade statistics - data since 1962 over 150 reporters 1.3billion records Over 150 reporters per year (>95% of the World Trade) On average, more than 2 billion records downloaded every month Newly published data synchronized with other international organizations such as World Bank, ECLAC, OECD, ITC, UNCTAD, etc via Web Services Imports, Exports, Re-exports, Reimports Commodity by Trading Partner Annual data Value in US dollars Net weight and secondary quantities Comtrade – key elements • Classifications and Commodity Codes HS (Harmonized System) and SITC (Standard International Trade Classification) • • • • • • Reporter country (ISO-3166-1 3 digit numeric) Partner country (ISO-3166-1 3 digit numeric) Trade Flow (Export, Import, Re-export, Re-import) Time (Year) Unit of measurements (Weight, Currency) Observations (Net weight, Net value) • • Comtrade uses SDMX cross-sectional format v1.0. Each commodity classification has a specific Key Family. Therefore, there are 6 key families: HS2002, HS1996, HS1992, SITC Rev.3, SITC Rev.2, and SITC Rev.1 Explanatory notes are integrated with Comtrade SDMX. This is a beta version. The SDMX structure can be modified. Comtrade Web Services are open Site License users. For further information please go to http://comtrade.un.org/kb/article.aspx?id=10050 • • • • Comtrade – key families •Data series defined in SDMX Key Families element structure:Dimension Comtrade – data exchange • Server: Comtrade web services – Internet-based, HTTP protocol, REST architecture • Client: – Can access Comtrade web service via web browser – and/or use ComtradeTools – a command-line program for data transfer/conversion applications • SDMX cross-sectional format, schemas specific to each classification - see UN site for more info • Web service can compress data on the fly (70-80%) • Access security based on IP address Comtrade – data transfer Trade Data Transfer Architecture United Nations Comtrade Tariff Line Total Trade Comtrade Web Services Web Server Database Server SDMX-ML Element based XML Organization #1 Http protocol Internet PC XML Files Organization #2 Comtrade Tools Other Tools PC Organization #3 Http protocol Comtrade Tools Text Files XML Files Other Tools Database Server Comtrade – SDMX downloads Data can be downloaded in SDMX format directly, e.g.: – Classification: H2 = Harmonized System 2002 – Reporter: 36 = Australia; Partner 251 = France – Years: 2003 and 2004; Commodity: Total trade http://unstats.un.org/unsd/comtradedist/getSdmxV1.aspx?px=H2&r =36&y=2004,2003&cc=TOTAL&p=251&comp=false – Classification : S2 = SITC2 – Reporter: 36 = Australia; Partner 0 = World – Year : 2004; Commodity: Total trade http://unstats.un.org/unsd/comtradedist/getSdmxV1.aspx?px Comtrade - tools • Requires Win2000 or higher • Standalone command-line program for Comtrade data download; requires additional software to synchronize SQL database • Downloads data in SDMX-ML format, can transform into CSV (or import into SQL Server) • Available from http://unstats.un.org/unsd/comtradedist/comtradeTool ssetup.zip Comtrade – tools usage • ComtradeTools /px:s2 /r:36 /p:0 /cc:TOTAL /y:2004 • ComtradeTools /px:H2 /r:251 /cc:10* /y:2004 • ComtradeTools /px:H2 /r:251 /cc:10* /y:2004 /action:DownloadAndConvertToCSV /outputDirectory:c:\temp\ • ComtradeTools /help • ComtradeTools /year:2003 /action:DownloadDataSetList • ComtradeTools /r:360 /y:2003 /px:H1 /source:Comtrade /format:SDMX /action:DownloadAndImportToSQLDB Comtrade – conclusions, future • • • • • • • • • A production application, based on SDMX data exchange format and web services SDMX cross-sectional format – standardised V1.0 Rapid - exchange can be compressed by web service Can convert to CSV format, and upload/synchronise with SQL database - no specialised SDMX knowledge necessary for end-users SDMX is becoming “just a part of the plumbing” UN Comtrade SDMX v.2.0 is in progress Data collection via SDMX: would need to be simple, via a generic format to avoid multiple schemas User-friendly interface for ComtradeTools Aligning with other trade SDMX trade projects such as CDS (Common Data Set) from WTO