November 20th, 2013
In the last few months, I’ve been building Hoot, a private video messaging app for Android, with my partner Christina. Part of a video messaging app is hosting and serving – you guessed it – videos. Hoot’s videos are served off the filesystem by nginx on a shared media server, separate from our application servers that process Hoot’s API requests.
Accessing yourwebser.ver/videos/bill/to_jane/cute_puppies.mp4 will load whatever’s in /srv/videos/bill/to_jane/cute_puppies.mp4. Great! Unfortunately, this setup doesn’t work for media that’s meant to be private, as any video is accessible to anyone.
One solution is just to make the media request within your application server (Rails, Django, Flask, etc) and proxy the data back to nginx, but don’t do that. It hogs a bunch of application resources, and, in my experience, is a 3-4x slowdown when compared to serving files from nginx. What we really want is for the application to decide whether a request should be allowed to access a file and leave it up to nginx to handle serving it.
Fortunately, nginx allows this through the X-Accel-Redirect header, documented in the X-Sendfile feature. If a proxy server sets this header, nginx rewrites the request and returns the result of the location listed in that header. The media location could be either on the same machine on which nginx is hosted (e.g. /srv/videos/…) or on some machine that nginx can access (192.168.1.100/media/videos/…). That’s a mouthful, so let’s look at some code. This example uses Flask, but the concepts are easily applied to other web frameworks.
Let’s assume your nginx conf has video files accessible either on the local file system or on some other server in your network. As I mentioned above, in Hoot’s case, our video files live on an external media server.
And your upstream Flask file is:
In the case where host videos using the same nginx instance as where our app server is running, nginx parses X-Accel-Redirect from the app, sees that it’s of the form /raw_videos/local/some/great/path.mp4, and so matches it with this location block:
Nginx will then look for a local file: /srv/videos/some/great/path.mp4.
In the case where we want to return some media that’s hosted elsewhere, the value of X-Accel-Redirect is something like /raw_videos/url/http://somegreaturl.com/whee/media.mp4, which nginx matches with this location block:
Since we used a regular expression to match the location and indicated a group using parentheses, the URL indicated by our app is available to our location block as the variable $1. We want nginx to serve the media at this URL, so we simply use this variable as the value for the proxy_pass directive.
Neat trick, huh? There are a couple of gotchas that are worth mentioning.
The first gotcha is that in processing an X-Accel-Redirect, nginx treats the redirected URL similar to how it processes the rewrite directive. If you do anything fancy with your appserver access logging or add particular headers to your app’s response, please note that nginx processes the redirect this as a separate request, so you’ll need to specify those directives again in your media location blocks.
Second, since the media redirect is processed as a separate request, in the case where you proxy to an external media server, the upstream headers (
$upstream_http_HEADER) come from the external media server, not from the initial request to the app! That is, if you set a special header like
$upstream_http_x_myapp_username) and use that value in your access logs, you’ll need to save the headers from the initial request to your app before you proxy_pass to your media server.
And finally, please note that nginx has its own DNS implementation that doesn’t look at /etc/hosts when it makes a proxied request to your external media server (or any proxied request, for that matter). That is, if you’ve set up /etc/hosts within your production network with short hostnames that you reference elsewhere in the app (e.g. web-1, web-2, media-1, db-1), you have to use something like dnsmasq as your DNS server on each machine and use the resolver directive to point at your local DNS server. For example, in your nginx.conf:
Have fun keeping your media safe! If you’d like to see this in action, make sure to check out Hoot.