Scribd
Upload
12 views41 pages

Overview of THREDDS Data Server

The THREDDS Data Server (TDS) is a web server designed for serving scientific data, written entirely in Java, and capable of handling various dataset formats like netCDF and HDF. It provides services for data access, collection, and metadata management, including OPeNDAP and OGC WMS/WCS. The server is open-source and packaged as a single file, facilitating easy deployment and use in scientific data applications.

Uploaded by

jessicaborgesp
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views41 pages

Overview of THREDDS Data Server

The THREDDS Data Server (TDS) is a web server designed for serving scientific data, written entirely in Java, and capable of handling various dataset formats like netCDF and HDF. It provides services for data access, collection, and metadata management, including OPeNDAP and OGC WMS/WCS. The server is open-source and packaged as a single file, facilitating easy deployment and use in scientific data applications.

Uploaded by

jessicaborgesp
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

THREDDS

 and  the    
THREDDS  Data  Server  (TDS)  

Don  Murray  –  CU/CIRES  


 Slides  by  Ethan  Davis  -­‐  Unidata  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


THREDDS  Data  Server  (TDS)  
• Web  server  for  scienMfic  data  (wriPen  in  100%  Java)  
• Can  serve  any  dataset  the  netCDF-­‐Java  library  can  read  
– E.g.,  netCDF-­‐3,  netCDF-­‐4,  HDF-­‐4,  HDF-­‐5,  HDF-­‐EOS,  GRIB-­‐1,  GRIB-­‐2  
• Catalogs  adverMse  available  datasets  and  services  
• Data  access  (subset)  services:  
– OPeNDAP  
– OGC  WMS  and  WCS  
– NCSS  
• Data  collecMon  services  
– AggregaMon  
– Point/staMon  collecMon  
• Metadata  services  
– THREDDS  
– ncISO:  ISO,  UDDC,  NcML  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


THREDDS  Data  Server  (cont.)  
• 100%  Java    
• Java  1.6    
– JDK  1.7  is  coming  “early  2011”  :  new  File  API  
• Tomcat  6.0    
– Reports  that  JePy  works  
• Spring  2.5    
– Internal  configuraMon  
– Spring  MVC  
• Packaged  as  a  single  file:  [Link]  
• Open-­‐source  license  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


THREDDS  Data  Server  
Servlet  Container  
[Link]  

•WCS   Remote  Access  


THREDDS  Server   •OPeNDAP   Client  
•HTTPServer  
•WMS  

NetCDF-­‐Java  
library  

confi[Link]  
IDD  Data  
Datasets  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  
[Link]  
Common  Data  Model  (CDM)  
• Implemented  in  the  netCDF-­‐Java  library  (3  layers)  
• NetCDF  Data  Access:  
– Access  to  array-­‐based  scienMfic  data  (netCDF  data  model)  
– Read  various  file  formats,  map  into  data  model  (IOSP)  
• NetcdfDataset:  
– GeospaMal  coordinate  systems  (convenMons)  
– NcML:  Modify  exisMng  dataset  
– AggregaMon  of  datasets  
• ScienMfic  Feature  Types,  e.g.:  
– Gridded  Data;  Radial  Data;  Swath  Data  
– Discrete  Sampling  Features:  
• Point  Data;  StaMon  Data;  Profile  Data  
• Trajectory  (e.g.,  aircrah  track)  Data  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


ApplicaMon  
ScienMfic  Feature  Types  

Datatype  Adapter  
NetCDF-­‐Java/  
NetcdfDataset  
CDM  architecture  
CoordSystem  Builder  

NetcdfFile  
THREDDS  
I/O  service  provider  

[Link]   OPeNDAP   NetCDF-­‐3   NIDS  

NetCDF-­‐4   GRIB  

NcML   HDF5   GINI  


Nexrad  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011   DMSP  
…  
Coordinate  Systems  
• Used  to  geolocate  data  
• Dataset  must  describe  the  coordinate  systems  used  
• Use  convenMons  to  describe  coordinate  systems  
– CF  ConvenMon  (recommended)  
– Other  convenMons  are  recognized  by  netCDF-­‐Java  library  
• What  you  get  if  your  data  follows  convenMons  
– Standard  visualizaMon,  debugging,  and  data  manipulaMon  
tools    
– Standard  servers  to  make  your  data  remotely  accessible  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


Coordinate  Systems    

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


ScienMfic  Feature  Types  
• Gridded  Data  
• Radial  Data  
• Swath  Data  
• Discrete  Sampling  Features  
– Point  data  
– StaMon  data  
– Profile  data  
– Trajectory  (i.e.,  aircrah  track)  data  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


Gridded  Data  
•  Cartesian  coordinates  
•  Data  is  2,3,4D  
•  All  dimensions  have  1D  coordinate  
variables  (separable)  
float  gridData(t,z,y,x);  
   float  t(t);  
   float  y(y);  
   float  x(x);  
   float  z(z);  

•  netCDF:  coordinate  variables  


•  OPeNDAP:  grid  map  variables  
•  HDF:  dimension  scales  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  
Swath  
•  two  dimensional  
•  track  and  cross-­‐track  
•  not  separate  Mme  dimension  
•  aka  curvilinear  coordinates  

float  swathData(  track,  xtrack)  


   float  lat(track,  xtrack)  
   float  lon(track,  xtrack)      
   float  alt(track,  xtrack)  
   float  Mme(track)  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


Point  ObservaMon  Data  
•  Set  of  measurements  at  the  same  
point  in  space  and  Mme  =  obs  
•  CollecMon  of  obs  =  dataset  
•  Sample  dimension  not  connected  

float  obs1(sample);  
float  obs2(sample);  
   float  lat(sample);  
   float  lon(sample);  
   float  z(sample);  
   float  Mme(sample);  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


THREDDS  Data  Server  
Servlet  Container  
[Link]  

•WCS   Remote  Access  


THREDDS  Server   •OPeNDAP   Client  
•HTTPServer  
•WMS  

NetCDF-­‐Java  
library  

confi[Link]  
IDD  Data  
Datasets  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  
[Link]  
THREDDS  Catalogs  
• Catalogs  collect,  organize,  and  describe  datasets  
– Hierarchical  collecMons  of  datasets  (virtual  
directories)  
– Each  dataset  may  have  mulMple  access  methods  
– DocumentaMon  (metadata)  may  be  added  
• To  the  catalog  itself  
• To  individual  datasets  (inheritance  opMonal)  
• Servers  can  use  catalogs  to  adverMse  the  
datasets  they  serve  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  
THREDDS  Catalogs  
<?xml version='1.0' encoding='UTF-8'?>
<catalog xmlns=“[Link]
xmlns:xlink="[Link] version="1.0.2">
<service name="odap" serviceType="OPENDAP" base="/thredds/dodsC/" />
<dataset name="TDS Tutorial: example 2">
<metadata inherited="true">
<serviceName>odap</serviceName>
</metadata>
<dataset name="TDS Tutorial: example data 1" urlPath="test/[Link]" />
<dataset name="TDS Tutorial: example data 2" urlPath="test/[Link]" />
<dataset name="TDS Tutorial: example data 3" urlPath="test/[Link]" />
<catalogRef xlink:title="My Other Catalog“
xlink:href="[Link]" />
<catalogRef xlink:title="Far Away Univ catalog“
xlink:href="[Link] />
</dataset>
</catalog>

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


THREDDS  Catalogs  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


THREDDS  Catalogs  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


THREDDS  Catalogs  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


THREDDS  Data  Server  
Servlet  Container  
[Link]  

•WCS   Remote  Access  


THREDDS  Server   •OPeNDAP   Client  
•HTTPServer  
•WMS  

NetCDF-­‐Java  
library  

confi[Link]  
IDD  Data  
Datasets  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  
[Link]  
Data  Access  Services  
• Bulk  File  Transfer  
– HTTP  Server  (any  file)  
• Remote  access,  subsemng  CDM  files  
– OPeNDAP  (any*  CDM  file)  
– Web  Coverage  Service  (grids)  
– Web  Map  Service  (grids)  
– Others  
• NetCDF  Subset  Service  (grids)  
• Radar  CollecMon  Subset  Service  
• CDM  Remote  Service  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


Data  Access  Services  
Clients  
• OPeNDAP  (lots,  including  IDV,  NCL,  IDL,  MatLAB)  
• WCS  (sMll  minimal,  but  expect  to  grow)  
– GDAL,  CADCorp,  George  Mason,  OWSLib  
• WMS  (speak  WMS;  consume  GIF,  JPEG,  and  PNG)  
– Browsers,  Portals,  ARC  GIS,  Google  Earth,  IDV,  etc  
• NetCDF  SubsePer  
– HTML  forms  interface  
– Web  service  (CUAHSI)  
– IDV,  ESRI  ArcGIS  (manual  download  of  netcdf-­‐CF)  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


OPeNDAP  Service  
• Protocol  for  remote,  subset  access  to  CDM  files  
• Current  version  2.0;  NASA  ESE  standard  
– Working  on  new  4.0  protocol  spec  
• TDS  can  serve  any*  CDM  file  
– Subsemng  by  variable,  index  range  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


OGC  Web  Services  
• OGC  (Open  GeospaMal  ConsorMum)  
• Web  Services:    
– Web  Feature  Service  (vector,  GIS,  GML)    
– Web  Map  Service  (raster,  GIS,  2D,  jpeg)    
– Web  Coverage  Service  (4D,  data  formats)  
– Web  Processing  Service  Remote  GIS  operaMons    
– Web  Catalog  Service  (CSW)  (SQL-­‐like  query  service)  
• Based  on  ISO/TC  211  Family  of  Standards  
– ISO/TC  211  =  Geographic  InformaMon/GeomaMcs  
• Google  donated  KML  spec  to  OGC  
• MoMvated  by  a  huge  drive  towards  standardizaMon  
• Standards  sMll  evolving  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


OGC  Web  Coverage  Service  (WCS)  
• TDS  supports  WCS  1.0  
• Can  subset  the  dataset  by:  
– Lat/lon  or  projecMon  bounding  box  
– Mme  and  verMcal  coordinate  range  
– list  of  Variables  
• Return  formats  
– GeoTIFF  floaMng  point,  grayscale  
– NetCDF/CF-­‐1.0  
• No  reprojecMons,  resamplings  
• Restricted  to  Gridded  Datasets  the  CDM  recognizes  
– AddiMonal  WCS  restricMon:  
• Grid  must  be  evenly  spaced  in  x,y  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


Web  Map  Service  (WMS)  
• GetCapabili3es  -­‐>  metadata  
• GetMap  -­‐>  map  image,  based  on  client-­‐selected  parameters,  inc:  
– Image  width/height  
– Image  coordinate  reference  system  
– Geographic  bounding  box  
– Depth/height  and  Mme  
– Style  
• GetFeatureInfo  -­‐>  informaMon  about  a  parMcular  map  pixel  
– Format  not  standardized  
• SpecificaMon  supports  4D  data  (although  tools  ohen  do  not)  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


OGC  Web  Map  Service  (WMS)  
• TDS  WMS:  
– Based  on  ncWMS  (Jon  Blower,  U  of  Reading)  
– Supports  1.3.0  and  1.1.1  
• ncWMS  notable  features  
– Fast  generaMon  of  images  
– Ability  to  read  CDM  datasets  on  variety  of  projecMons  
– Can  reproject  images  into  large  number  of  coordinate  systems  
• Uses  geotoolkit’s  extensive  projecMon  library  
• WMS  specificaMon  supports  4D  data  (tools  ohen  do  not)  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


TDS  WMS  (cont)  
• Godiva2  web  client  (ncWMS)  
– Uses  OpenLayers  (JavaScript)  library  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


WMS  Interoperability  
NASA  World  Wind   3rd-party
clients can’t
use the
Cadcorp  SIS   custom WMS
extensions
Google  Earth  

Godiva  2  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


NetCDF  Subset  Service  
• Experiment  with  REST  style  web  service  
• Allow  to  subset  the  dataset  by:  
– Lat/lon  bounding  box  
– Mme  and  verMcal  coordinate  range  
– list  of  Variables  
• Output:  NetCDF,  XML,  CSV  (spreadsheet)  
• Gridded  Data  
– Output  is  a  CF  /  netCDF  file  
– VariaMon  of  WCS  (simplified  request  protocol)  
• Grid  as  Point  Datasets  
– Extract  verMcal  profile,  Mme  series  from  one  point  in  model  data  
– StaMon  Data:  metars  (7  day  rolling  archive)  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


Using  WMS  and  NCSS  

* From PacIOOS site, developed by John Maurer, U of HI


Using  WMS  and  NCSS  

* From PacIOOS site, developed by John Maurer, U of HI


ncISO  
• Developed  by  NOAA/NGDC  
• Supports  three  metadata  services  
– Provides  an  NcML  representaMon  of  a  dataset  
– Provides  an  ISO  19115-­‐2  metadata  representaMon  
of  a  dataset  
– Provides  an  evaluaMon  of  how  well  the  metadata  
contained  in  the  dataset  conforms  to  the  
NetCDF  APribute  ConvenMon  for  Data  Discovery  
(NACDD)  
• HarvesMng  tools:  GI-­‐Cat  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  
THREDDS  Data  Server  
Servlet  Container  
[Link]  

•WCS   Remote  Access  


THREDDS  Server   •OPeNDAP   Client  
•HTTPServer  
•WMS  

NetCDF-­‐Java  
library  

confi[Link]  
IDD  Data  
Datasets  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  
[Link]  
ConfiguraMon  Catalogs  
• Administrator  tells  the  TDS  which  /  how  
datasets  are  to  be  served  
• IdenMfy  the  files  on  disk  
• Name,  organize,  sort,  filter  into  virtual  
collecMons  
• Add  discovery  metadata  
• Integrated  with  NcML  
– Create  virtual  datasets  
Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  
TDS  ConfiguraMon  Catalogs  
…
<datasetScan name="Ocean Satellite Data“
path="/data/ocean/sat/"
location= "/data/tds/netcdf/">

<metadata inherited="true">
<serviceName>odap</serviceName>
<dataType>Grid</dataType>
<dataFormat>NetCDF</dataFormat>
<documentation xlink:href="[Link]
xlink:title="NCEP Model documentation" />
<documentation type="processing_level">
Transmitted through Unidata Internet Data Distribution.
</documentation>
</metadata>
</datasetScan>
…

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


TDS  and  NcML  
• NcML:  an  XML  representaMon  of  netCDF.    
– Can  “fix”  metadata  problems  
– Can  augment  metadata  
– Can  aggregate  datasets  
• General  AggregaMons:  joinNew,  joinExisMng,  Union  
• Specialized  aggregaMons:  
– Forecast  Model  Run  CollecMon  (FMRC)  
– Point  Feature  CollecMons  

• Embed  NcML  into  the  TDS  configuraMon  catalog  


• Server  serves  a  virtual  dataset  defined  by  NcML  
– NcML  hidden  from  the  client  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


TDS  /  NcML    
Modify  and  serve  through  TDS    
<dataset name=“Polar Orbiter Data" urlPath =“idd/sat/PolarData“ >

<netcdf location="/data/sat/[Link]”>
<attribute name="Conventions" value="CF-1.4"/>
<variable name="Reflectivity" orgName=“R34768”>
<attribute name="units" value=“dBZ" />
<attribute name=“coordinates" value=“time lat lon" />
</variable>
</netcdf>

</dataset>

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


TDS  /  NcML    
Modify  all  files  in  datasetScan    
<datasetScan name=“Polar Orbiter" path="/data/sat/"
location= "/data/hdf/polar/">

<netcdf>
<attribute name="Conventions" value="CF-1.4"/>
<variable name="Reflectivity" orgName=“R34768”>
<attribute name="units" value=“dBZ" />
<attribute name=“coordinates" value=“time lat lon" />
</variable>
</netcdf>

</datasetScan>

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


TDS  /  NcML  
AggregaMon  
<dataset name="WEST-CONUS_4km Aggregation“
urlPath="satellite/3.9/WEST-CONUS_4km">

<netcdf>
<aggregation dimName="time" type="joinExisting">
<scan location="/data/satellite/WEST-CONUS_4km/" suffix=".gini" />
</aggregation>
</netcdf>

</dataset>

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


TDS  FMRC  AggregaMon  
…
<featureCollection name="NCEP-NAM-Polar_90km" featureType="FMRC“
path="fmrc/NCEP/NAM/Polar_90km">
<collection spec=“/data/NAM_Polar_90km_#yyyyMMdd_HHmm#.grib2$“
recheckAfter="15 min" olderThan="5 min"/>
<update startup="true" rescan="0 5 3 * * ? *" />
<protoDataset choice="Penultimate" change="0 2 3 * * ? *" />
<fmrcConfig regularize="true“ datasetTypes="TwoD Best Files Runs" />
</featureCollection>
…

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  


For  More  InformaMon  
• TDS  Home  page  
– hPp://[Link]/projects/THREDDS/tech/[Link]  
• TDS  Workshop/Tutorial  
– hPp://[Link]/projects/THREDDS/tech/tds4.2/
tutorial/  
• Unidata  THREDDS  Server  (motherlode)  
– hPp://[Link]/thredds  

Joint  AAP/ICTP  Workshop  -­‐  July  27,  2011  

You might also like