ATS for Systems Administrators. Apache Traffic Server. Basic configuration. For Systems Administrators. Notes

Transcription

1 ATS for Systems Administrators Apache Traffic Server For Systems Administrators Basic configuration records.config storage.config remap.config (for reverse proxies) ATS has lots of configuration files, and it s pretty daunting when you first start. These are the files you ll want to look at to begin with, and you don t need to worry about the rest. Other files and what they do: ae_ua.config - Accept-Encoding/User-Agent filtering Used for fixing Accept-Encoding for bad clients cache.config - Lets you change cache parameters for certain pages This is where you fix badly behaved origin servers that send no-cache headers or otherwise need to override caching parameters. cluster.config - cluster configuration file - not user editable congestion.config - Configure throttling when origin servers are congested hosting.config - Lets you specify which volumes are cached to by site icp.config - Inter Cache Protocol config ICP is broken in the current ATS release - TS 32 ip_allow.config - Allow/disallow requests and methods based on IP log_hosts.config - Lets you split up logs by hostname logs_xml.config - Lets you set up custom logs parent.config - Allows you to set nested/parent proxies by domain Similar to remap.config, but for multiple levels of proxies plugin.config - Specify which plugins to load records.config - General settings remap.config - Specify origin servers by domain This actually rewrites URLs before passing the request on socks.config - Configure a SOCKS proxy splitdns.config - Lets you specify alternate DNS servers per domain ssl_multicert.config - Lets you specify SSL certificates for each IP Note that for SSL you need one IP per host (ignoring SNI for now), so the mapping is per IP that ATS listens on. storage.config - Disk cache config

2 update.config - Lets you schedule updating content from origin servers vaddrs.config - VIP configuration volume.config - Lets you define volumes for partitioning the cache Use with hosting.config to limit the cache used for certain domains records.config CONFIG proxy.config.http.server_ports STRING 80 CONFIG proxy.config.cache.ram_cache.size INT 10G CONFIG proxy.config.log.logfile_dir STRING /var/log/trafficserver CONFIG proxy.config.proxy_name STRING servername.example.com # The folllowing are for reverse proxies CONFIG proxy.config.url_remap.remap_required INT 1 CONFIG proxy.config.http.cache.ignore_client_cc_max_age INT 1 CONFIG proxy.config.http.normalize_ae_gzip INT 1 proxy.config.http.server_port - 80 for reverse proxy, 8080 or 3128 or whatever you want for a forward proxy. Ram cache size: As big as possible. You don t want to leave this at the default of disk cache size/10. Allow space for ATS itself and the OS, and give the rest to Ram cache. Measure memory usage with ATS and a tiny RAM cache to see what the base level would be. And don t worry about OS disk cache. Logfile dir - where you want logs to go Proxy name - defaults to the name of the build machine, or FIXME if you install via RPM. It shows up in the Via: header and it s especially useful if you load balance across different proxy servers. remap_required - This is on by default, and should stay this way for a reverse proxy. For a forward proxy, set it to 0. It specifies whether a valid entry in remap.config is required to allow a request. In other words, whether to allow proxying to any site or just ones you specify. proxy.config.http.normalize_ae_gzip - prevents caching multiple copies of compressed objects if clients pass different Accept-Encoding headers for compressed content. proxy.config.http.cache.ignore_client_cc_max_age ignores Cache Control: max-age headers from the client. You want this for reverse proxies to prevent clients forcing a hit to the origin server. records.config proxy.config.url_remap.pristine_host_hdr Passes the original Host: header to the origin server instead of changing it to be the origin server name. Very useful when throwing a reverse proxy in front of an existing site quickly. A situation that comes up often is quickly turning caching on for a site. Some sites have name based virtual hosting turned on, and you need to provide the right Host: header for it to respond correctly. This setting lets you put the proxy in place without any changes (you still have issues with what can/can t be cached of course)

3 You don t need reverse_map directives if this is turned on (see later) storage.config Directory: /path/to/cache 20G Filesystem device or entire disk: /dev/sde Size is optional when specifying an entire disk - ATS can work it out. Make sure your disk size is bigger than RAM size. If you don t do this, it will limit the amount of RAM cache used to that of the disk cache size. remap.config Specify which origin server to use for a given URL going through the proxy: map Required for reverse proxy, but can be used for forward proxies also if you want to redirect requests for a certain domain. Reverse mapping deals with redirects from the server: reverse_map reverse_map - translates redirect responses (301/302) from the server to point to traffic server instead of the origin server. reverse_map isn t needed if you have pristine host headers (the redirect location doesn t need changing) It s common to have multiple URLs go to the same origin server (e.g. example.com and You should only add one reverse_map in place, for whichever URL you consider to be canonical. Processes traffic_cop - watchdog process traffic_manager - allows live configuration of the server traffic_server - the proxy process itself Each of the above starts the process below it

4 You can start traffic_manager or traffic_server directly if you don t need the features of the process before it. E.g. On solaris, you just start traffic_manager and let SMF deal with restarting processes. Traffic Line Read variables/stats: traffic_line -r varname Change a config option: traffic_line -s varname -v value Reload traffic server: traffic_line -x Lots more options, but these are most useful. traffic_line -r can be used to read the same variables as provided with the stats module. Any variable mentioned elsewhere in this presentation can be read with this command. E.g.: traffic_line -r proxy.process.net.connections_currently_open See also the stats_over_http plugin for a better way to view multiple stats. traffic_line -s also edits records.config when a variable changes, so it can be useful for making quick fixes on the fly. In our experience traffic_line -x doesn t always work and a full restart is required. YMMV. Other traffic line commands restart various processes or clear statistics. Logs Trafficserver has 3 general types of logs: System Logs - Error messages, debug output, diagnostic logging Event Logs - Access logs of every transaction Error Logs - Logs of transactions with errors System logs Controlling where diagnostics get sent: proxy.config.diags.output.* STRING [OESL] Not in records.config by default, you need to add them in at the end. O = Stdout, E = Stderr, S = Syslog, L = diags.log Syslog events are logged to the daemon facility By default, most types of entries are Syslog, Stderr or both

5 Default values: proxy.config.diags.output.diag E proxy.config.diags.output.debug E proxy.config.diags.output.status S proxy.config.diags.output.note S proxy.config.diags.output.warning S proxy.config.diags.output.error SE proxy.config.diags.output.fatal SE proxy.config.diags.output.alert SE proxy.config.diags.output.emergency SE Other notes: On Centos, E means that it ends up in traffic.out, and not traffic_server.stderr. On Solaris, look in SMF logs to see stdout/stderr. You can change the syslog facility with the proxy.config.syslog_facility option in records.config. Debug logs CONFIG proxy.config.diags.debug.enabled INT [0 1] CONFIG proxy.config.diags.debug.tags STRING http.* dns.* Set to 1 to enable debug logs Filter what gets logged with the tags option (regex) Debug logs are a pretty good way of finding out why something isn t working as expected. What to put for tags: look in source code for Debug() calls. Taking a guess can work too (e.g. you re looking for issues with logging, then log.* would be a good place to start). Quick and dirty command to extract debug tags from the source (assumes GNU grep): grep -orh 'Debug("[^"]*' * sort uniq sed 's/debug("//' Event/Access logs Logs every request/transaction CONFIG proxy.config.log.logging_enabled INT 3 Built in log formats (squid, common, extended, extended2)

6 CONFIG proxy.config.log.*_log_enabled INT 0 CONFIG proxy.config.log.*_log_is_ascii INT 1 CONFIG proxy.config.log.*_log_name STRING * proxy.config.log.logging_enabled - this is on by default for both error/transaction logging (3). You usualy want to leave it alone. Built in log formats: foo_log_enabled - enable this log format foo_log_is_ascii - ASCII or binary logging Binary logging is for speed of writing and processing It can save space, but not always: see for more information on this. foo_log_name - the name of the log file (without the.log or.blog extension). This defaults to the name of the file format. Event log formats squid - Squid-like format common - Netscape/Apache common log format extended - Netscape extended log format extended2 - Netscape extended2 log format The squid log format is close, but not exactly the same as the squid format. The codes used for cache hit/miss and so on may be slightly different. Squid log analysis tools may work, but will probably need a little modification. The common log format is very similar to the common log format often used with apache, but it doesn t contain information on cache hits/misses. Other logging options Log file dir: CONFIG proxy.config.log.logfile_dir STRING /var/log/trafficserver Limit space for log files: CONFIG proxy.config.log.max_space_mb_for_logs INT CONFIG proxy.config.log.max_space_mb_headroom INT 1000 If you set proxy.config.log.auto_delete_rolled_files, then ATS will delete logs when it starts to run out of space (space used for logs is at max_space_mb_for_logs - max_space_mb_headroom).

7 If you haven t set auto_delete_rolled_files, then ATS will log an error and refuse to log any further if space is exhausted. Via header ATS Passes a Via: header with each request or response Quick way to tell if a request was cached CONFIG proxy.config.http.insert_request_via_str INT 2 CONFIG proxy.config.http.insert_response_via_str INT 2 Looks like this: Via: http/1.1 foo-proxy (ApacheTrafficServer/3.0.4 [c s f ]) Decoder tool at: request_via_header includes the header in the request to the origin server response_via_header included it in the response to the client Verbosity can be increased/decreased: (1 3) Very useful in debugging. The information leakage may/may not be acceptable to you though. Info on decoding the header: to decode a header Via Header Decoding quick reference: [cxsxfx] c - Cache lookup, s - Server information, f - Cache fill c: M=miss, H=hit, S=stale, A=cached version not acceptable (miss) s: S=served, N=not-modified, E=error f: W=written to cache, D=deleted from cache, U=updated cached copy Short version of Via status code with just c,s,f See for information on interpreting the verbose via header. C = Results of the cache lookup Blank == no cache lookup made S = Response from origin server (200, 204, error) Blank == no server connection made F = What happened to the entry in the cache

8 Blank == no cache write made Purging the cache To remove an item from the cache, run: curl -X PURGE -x cacheserver:80 -v Response codes: The item wasn t cached The item was cached, and has been removed Access denied Here is a script to do this, and loop through multiple servers: #!/bin/bash SERVERS=" " [[ -z $1 ]] && { echo "Usage: $0 [URL]" exit 1 } for s in $SERVERS; do echo -n "Sending purge request to $s... " r=$(curl -X PURGE -x $s:80 -v $1 2>&1 grep '< HTTP/1.1' \ awk '{print $3}') case $r in 404) echo "Not cached" ;; 200) echo "Removed from cache" ;; 403) echo "Access denied" ;; *) echo "Unknown response code: $r" ;; esac done Limiting PURGE requests ATS before records.config: # 1026 == 0x402 == allow purge CONFIG proxy.config.http.quick_filter.mask INT 1026 remap.config:

9 # Block purge requests from anywhere except block_purge This doesn t work in ATS and above - the quick_filter_mask option has been removed. See next slide for how to do it in and above. (TS 1140 and 32.html for details) Limiting PURGE requests ATS and later - ip_allow.config # Allow purge requests from src_ip= action=ip_allow method=purge ip_allow.config is first match, so put the allow line before any lines that deny purge requests. You can put method=all if you like (as in the default line that allows localhost). Forward proxy Set proxy.config.url_remap.remap_required to 0 You can leave proxy.config.reverse_proxy.enabled set if you wish. This enables the use of remap.config. Troubleshooting Performance/bandwidth less than expected ATS might be throttling new connections. Check connection count proxy.process.net.connections_currently_open. Increase proxy.config.net.connections_throttle if needed. Remember each request has 2 connections - client and server. Trafficserver has a connection limit beyond which it will throttle connections. It s quite possible to hit this limit on a busy site, and still be well below the capabilities of a modern server. Troubleshooting/fixing issues: Each request has 2 connections - client to proxy, proxy to origin, so you should double this value for your expected maximum amount of concurrent connections. Stats to look at: proxy.process.http.current_client_connections

10 proxy.process.http.current_server_connections proxy.process.net.connections_currently_open (combined) Troubleshooting Items not getting cached Check Cache-control/Expires headers being returned Fix at origin server if possible Check proxy.config.http.cache.required_headers Override caching in cache.config if not possible Ideally you d set the cache control headers at the origin server appropriately, but if this isn t possible, here are a couple of things you can modify: proxy.config.http.cache.required_headers cache.config: 0 - Don t require any headers to cache a document 1 - Last-Modified:, Expires:, or Cache-Control: max-age required 2 - explicit lifetime required, Expires: or Cache-Control: max-age lets you override cache behavior on a per-domain or per-url basis. Troubleshooting Items not getting cached Example cache.config: dest_domain= ttl-in-cache=300s dest_domain= suffix=.asp action=ignore-no-cache Plugins regex_remap - More advanced regular expression remap rules conf_remap - Override records.config entries per remap rule header_filter - Strip/add headers to from/to requests stats_over_http - json output of ATS stats cacheurl - Use the same cache entry for multiple URLs stale_while_revalidate - Serve stale items from the cache while revalidating. Plugins - regex_remap

11 The built in regex_map only works on hostname. Regex_remap lets you match on the full URL. Example: map You would put your regular expression remaps in maps-example-com.reg: ^/site/(.*)/ The first pparams contains the filename for the other remap rules The default path is /usr/local/etc/regex_remap. Currently this is hardcoded in the plugin, so you probably want to use a full path here. There are other option you can provide to change the behavior of regex_remap. For example, will make the regex map on the entire URL (e.g. By default only the path and query string are provided. See for full details. Plugins - conf_remap Lets you override records.config entries per remap rule. This is useful when you reverse proxy for multiple sites in a single ATS instance. Example: map Entries in the config file you provide are the same as in records.conf. Not all records.conf entries can be overridden in this manner. A list of supported options (and further docs) are at Plugins - header_filter Lets you add/remove headers for requests/responses. Useful if you need to strip cookies or other headers. Example: map You can also add this to plugins.config:

12 header_filter.so /etc/trafficserver/hdr_filters.conf Plugin docs: Plugins - header_filter Example hdr_filter.conf file: [READ_REQUEST_HDR] Cookie [READ_RESPONSE_HDR] Cookie [SEND_RESPONSE_HDR] X-My-Hdr +Some Value+ You can use the following sections in a config file: READ_REQUEST_HDR - as ATS reads the request from the client READ_RESPONSE_HDR - as ATS reads the response from the origin SEND_REQUEST_HDR - as ATS sends the request to the origin SEND_RESPONSE_HDR - as ATS sends the response to the client By default headers mentioned in the file are stripped If you add a value afterwards, surrounded by +, then the header is added. If you add a value afterwards, surrounded by =, then the header is modified (any existing header is replaced). Plugins - stats_over_http Exposes statistics via json Add stats_over_http.so to plugins.config Go to to view stats This is compatible with the circonus JSON module Premium support customers get a free circonus account with 2 free hosts to try this out. Plugins - cacheurl

13 Lets you rewrite the cache key used for a URL. Useful if you have multiple origin servers/mirrors with the same content. Example: Add cacheurl.so to plugins.config Create a cacheurl.config file in the plugins directory It s good to add a.tsinternal to the end of the rewritten cache key to ensure you never clash with a real URL. The first parameter is the URL to match on, the second is the URL to rewrite it to, but only for caching purposes. The content is still loaded from the original URL. More documentation Google searches tend to link to the v2 docs, which may not be up to date. You can try changing v2 to admin in the URL, but it doesn t always work (the page name has changed), but you can usually find the right page by going to the main admin guide for trunk and looking at the links on the right hand side. Questions?