Download Slide 1

XRootD Release 4
And Beyond
GSI Seminar
Stanford University/SLAC
July15, 2015
Andrew Hanushevsky, SLAC
http://xrootd.org
User Visible Changes I
New client library: libXrdCl.so

First available in 3.3.0 now is default

Old deprecated client library is libXrdClient.so
Copy command: xrdcopy
xrdcopy renamed to xrdcp
 Xrdcopy is now a symlink to xrdcp
 Original xrdcp renamed toxrdcp-old


July 15, 2014
All have same command line interface!
GSI Seminar
‹#›
User Visible Changes II
New meta command: xrdfs

Logical replacement to xrd command
Different command line interface
 More user friendly and intuitive

When installing Release 4 RPM

You must de-install previous XRootD release

July 15, 2014
They are incompatible
GSI Seminar
‹#›
Root Support
New client is the default in ROOT

Since 5.34.14 (Dec 14, 2013)


Via TNetXNGFile plug-in
Setting envar XNet.UseOldClient to 1

Loads TNetXFile instead (old client)
Root 6 does not support new client
Due to conflicts with PROOF
 Expected to be resolved in release 6.1

July 15, 2014
GSI Seminar
‹#›
Admin Visible Changes
IPv6 support
Public Private Networking
Disk Caching Proxy
HTTP Plug-in
Remote Debugging
Optional Client Plug-ins
Miscellaneous
July 15, 2014
GSI Seminar
‹#›
IPv6 Support
The new client supports IPv6

It prefers IPv6 whenever possible


This corresponds to HEPIX request
Connects using mapped IPv4 addr if need be
The old client is deprecated

It will not support IPv6
July 15, 2014
GSI Seminar
‹#›
IPv6 Configuration
Frankly none needed

Clients detect available network stacks

May force it to use various IP modes


Via configuration file or envars
Server detects interfaces and DNS settings
It’s important that DNS is setup correctly
 May force server to use IPv4 only


July 15, 2014
Via command line option (-I v4)
GSI Seminar
‹#›
IPv6 Considerations I
While servers and redirectors can accept
both IPv6 and IPv4 clients. . .

IPv6 clients are assumed dual-stack

May be redirected to either IPv6 or IPv4 node


This works if all the servers are dual-stack


Client error recovery will resolve this if unworkable
Generally the preferred migration path
Since redirection is via hostname…

July 15, 2014
DNS entry must have no un-routable entries
GSI Seminar
‹#›
IPv6 Considerations II
Client dual-stack assumption will be lifted
in release 4.1 (3Q14)

Clients will be redirected to compatible nodes


If none, the client receives an error
Largely driven by public-private networks

July 15, 2014
And interactions with IPv6/4
GSI Seminar
‹#›
Public-Private Network Support
Redirectors are now network cognizant

Servers inform redirectors of usable i/f’s
Clients always compatibly redirected

Private to private and public to public

Subject to configured network topology


July 15, 2014
Via new xrd.network directive option
Applies to servers and redirectors
GSI Seminar
‹#›
Public-Private Net Topologies
xrd.network routes type [use if1[,if2]]

type: local | common | split

local (default)


common



Private incoming -> private (preferred) or public
Public incoming -> only public
split


No address differentiation (i.e. pre R4 mode)
Incoming address must match outgoing address
If use unspecified addresses come from DNS!
July 15, 2014
GSI Seminar
‹#›
Private Addresses & DNS
Private addresses should never be in DNS

Unless
It’s a site local DNS server or
 The address is zone registered (i.e. only locally available)

This is the assumption used by XRootD

To avoid DNS reverse translation timeouts
Hence, the use option might be needed

If a server connects using a private address
July 15, 2014
GSI Seminar
‹#›
Public-Private Considerations
The available server interfaces
Must be uniform within a cluster
 Must be compatible with configured topology

Restrictions relaxed in R 4.1

Redirectors will match clients & server i/f’s


May lead to inaccessible files if no match exists
Still working through external access issues

May require a separate redirector for external access

July 15, 2014
Due to IPv6/4 and public-private network interactions
GSI Seminar
‹#›
Networking Is Complex!
There are many combinations now

Client and server capabilities must now match
There are 4 basic combinations
Public
Private
IPv6
IPv4
But things are not that simple!
Dual stack clients add another 4 combinations
July 15, 2014
GSI Seminar
‹#›
Disk Caching Proxy
New proxy server mode

Configured via pss.cachelib directive
Caches whole files or file segments

Mode is configurable
Cached content available for future access

Until LRU purged (configurable)
Many use cases to increase access speed
July 15, 2014
GSI Seminar
‹#›
Typical Disk Caching Proxy Uses
Speed up
Remote XRootD
Remote Access
Clusters
Caching
Proxy
Local
Clients
July 15, 2014
Caching
Proxy
XRootD
Server
HDFS
Speed up
Random Access
Caching
Proxy
XRootD
Cluster
FS
Speed up
HD Access
SSD
GSI Seminar
‹#›
HTTP Plug-in
Basic http, https, WebDav access
Suitable for browsers, curl, wget, & davix
 Provides another mode of well-known access


http is neither low latency nor high-performance


Google & Microsoft have proposed improvements
• Changes submitted to W3C and IETF as http2
• Improvements are considered incremental
• They only address the most vexing problems
Configured via xrd.protocol directive

July 15, 2014
And specialized http.xxx directives
GSI Seminar
‹#›
XRootD Multi-Protocol Support
Always supported multiple protocols
Improved architecture makes it much easier
 New protocols can leverage XRootD features


Security, monitoring, file system plug-ins, etc
XRootD
Server
Clients
XRootD
Bridge
Memory Based
Protocol Conversion
Loadable
Protocol
July 15, 2014
GSI Seminar
‹#›
Remote Debugging via DigFS
XRootD provided pseudo file system
Provides restricted selectable R/O access to

Configuration file
 Core files
Log files
/proc/self (Linux only)


Has authentication & authorization options

Including access control restrictions
View is standardized regardless of location
 Configured via xrootd.diglib directive

July 15, 2014
GSI Seminar
‹#›
The DigFS View
/=/
Virtual
exported
path
July 15, 2014
conf
conf/etc (site specific)
core/cmsd
core/xrootd
logs/cmsd
logs/xrootd
proc/cmsd
proc/xrootd
GSI Seminar
‹#›
DigFS Authorization
DigFS consults authorization file

Created by the site and specified in config file

xrootd.diglib * authfile
+
all
[-]conf
[-]core allow
[-]logs
[-]proc
July 15, 2014
gsi
host
krb5
pwd
sss
unix
GSI Seminar
g=group
h=host
n=name
o=org
r=role
+



 


‹#›
XRootD Client Relationships
External Package
XRootD - add on
XRootD - core
XRootD – plug-ins
Athena
FTS3
xrdcopy
CopyProcess
XrdClFilesystem
xrdfs
Dirac
Gaudi
PyXRootD
root
XrdClFile
CMSSW
PROOF
XrdClPostMaster
Plug-ins replace original implementation

All calls may be replaced

July 15, 2014
All layers above benefit with any code changes
GSI Seminar
‹#›
Client Plug-ins
Plug-ins are loaded at run-time

Client looks for plug-in configuration files
1st Locally: ~/.xrootd/client.plugins.d/
 2nd Globally: /etc/xrootd/client.plugins.d/


Both locations can be over-ridden via envar

XRD_PLUGINCONFDIR
Plug-ins are strictly version checked

Allows for independent development
July 15, 2014
GSI Seminar
‹#›
Miscellaneous I
Readv proxy pass-through

Automatic & greatly improves performance
Manual log file rotation (e.g. logrotate)

Via extended –k command line option
High precision log file timestamps

New –z command line option

July 15, 2014
Log timestamp appears in microsecond format
GSI Seminar
‹#›
Miscellaneous II
Special stat() plug-in for odd file systems

Configured via the oss.statlib directive

Includes plug-in for GPFS with tape backend
Cancellable third party copy

Used by transfer tools
Fast directory listings

Stat info can now be included w/ dir entry

July 15, 2014
xrdfs uses this to speed up long listings
GSI Seminar
‹#›
Miscellaneous III
New query config options

Use the xrdfs command to display
Query cms
 Query role
 Query sitename
 Query version

- shows cmsd status
- shows server’s cluster role
- shows the site’s name
- shows server’s version
Cluster node blacklisting

Via cms.blacklist directive & blacklist file

July 15, 2014
Useful in federated environments
GSI Seminar
‹#›
Miscellaneous IV
New monitoring information

User login record now also includes
Name of the client’s executable
 Contents of client’s XRD_MONINFO envar



July 15, 2014
Useful to tie external information to actual data usage
E.g. the Panda jobid to cross-reference I/O usage
GSI Seminar
‹#›
Looking Beyond Release 4
Cross Protocol Redirection
Meta-links
I/O Throttling Plug-in
July 15, 2014
GSI Seminar
‹#›
Cross-Protocol Redirections I
XRootD protocol is capable of redirecting
to a protocol other than xroot

On file open the server may tell the client that
it’s more efficient to try something else

E.g. read the file locally from disk

July 15, 2014
Open(xroot://host/filename) -> redirect file://filename
GSI Seminar
‹#›
Cross-Protocol Redirections II
New client already capable of processing
out-of protocol redirections
Server needs some development to do so
Root’s TFile framework needs updating to
handle a change in protocols

Changes already in development

July 15, 2014
Target for root 6.1 or 6.2
GSI Seminar
‹#›
Meta-Link Files I
XML file that describes one or more data
files available for access

Meta-link file identified by dot suffix


metalink (v3) or meta4 (v4 & incompatible w/ v3)
File is read and parsed by the client

Client picks one based on certain criteria


July 15, 2014
E.G. priority, location, etc
If access fails, client can pick another one
GSI Seminar
‹#›
Metalink Files II
XRootD client will support meta-link files

But the road is not straight-forward
Incompatible meta-link formats, which one?
 Need to avoid encumbering installation


I.e. pre-reqs for sites that don’t care about meta-links
Targeting availability in 4Q14
July 15, 2014
GSI Seminar
‹#›
I/O Throttling Plug-in
Allows site to limit I/O access to disk

Useful for throttling external access


Used in federated environments via proxy server
Configured via xrootd.fslib directive

And specific throttling directives
Plug-in is in code review phase

Already used by CMS in production
Targeting 3Q14
July 15, 2014
GSI Seminar
‹#›
Acknowledgements
Current Software Contributors
ATLAS: Doug Benjamin, Patrick McGuigan, Ilija Vukotic
CERN: Lukasz Janyst, Andreas Peters, Justin Salmon
Fermi: Tony Johnson
Root: Gerri Ganis, Bertrand Bellenot
SLAC: Andrew Hanushevsky, Wilko Kroeger, Daniel Wang, Wei Yang
UCSD: Matevz Tadel
UNL: Brian Bockelman
WLCG: Mattias Ellert, Fabrizio Furano, David Smith








US Department of Energy

Contract DE-AC02-76SF00515 with Stanford University
July 15, 2014
GSI Seminar
‹#›